From d00d04f8900f1292a1701db7eb8ef87cef74290b Mon Sep 17 00:00:00 2001 From: Branislav Jansik <branislav.jansik@vsb.cz> Date: Mon, 31 Jul 2023 14:57:03 +0200 Subject: [PATCH] Update shell-and-data-access.md --- docs.it4i/general/shell-and-data-access.md | 32 ++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/docs.it4i/general/shell-and-data-access.md b/docs.it4i/general/shell-and-data-access.md index 3ff4e7732..6a0bcdcfa 100644 --- a/docs.it4i/general/shell-and-data-access.md +++ b/docs.it4i/general/shell-and-data-access.md @@ -149,6 +149,38 @@ local $ man scp local $ man sshfs ``` +The rsync client uses ssh to establish connection. + +```console +local $ rsync my-local-file +``` + +```console +local $ rsync -r my-local-dir username@cluster-name.it4i.cz:directory +``` + +!!! note + The data transfer speed is limited by the single-core ssh encryption speed to about **150 MB/s** + Use **parallel rsync** for unlimited transfers + +The parallel rsync transfers multiple files at the time, utilizing multiple cores to accelerate encryption. +First, set up ssh-agent single sign on: + +``` +console +local $ eval `ssh-agent` +local $ ssh-add +Enter passphrase for /home/user/.ssh/id_rsa: +``` +Then run the rsync in parallel: + +```console +ls my-local-dir | xargs -I% -n 2 -P 4 rsync % username@cluster-name.it4i.cz:directory/. +``` + +The -n XXX argument determines how many files will be transfered per single rsync process. The -P YYY argument +determines number of parallel rsync processes. Set to number of cores on your local machine. + On Windows, use the [WinSCP client][c] to transfer data. The [win-sshfs client][d] provides a way to mount the cluster filesystems directly as an external disc. ## Connection Restrictions -- GitLab