diff --git a/docs.it4i/general/shell-and-data-access.md b/docs.it4i/general/shell-and-data-access.md index 35867df6d131833cad2aa5acc62d6e051dc6d283..0ae879456088e25225bec3795ebf80b18fbc4fef 100644 --- a/docs.it4i/general/shell-and-data-access.md +++ b/docs.it4i/general/shell-and-data-access.md @@ -161,9 +161,9 @@ local $ rsync -r my-local-dir username@cluster-name.it4i.cz:directory !!! note The data transfer speed is limited by the single-core ssh encryption speed to about **150 MB/s** - Use **parallel rsync** for unlimited transfers + Run **multiple rsync** instances for unlimited transfers -The parallel rsync transfers multiple files at the time, utilizing multiple rsync processes and cores to accelerate encryption. +Parallel execution of multiple rsync processes utilizes multiple cores to accelerate encryption and multiple tcp streams for enhanced bandwidth. First, set up ssh-agent single sign on: ```console @@ -172,14 +172,14 @@ local $ ssh-add Enter passphrase for /home/user/.ssh/id_rsa: ``` -Then run the rsync in parallel: +Then run multiple rsync instances in parallel, f.x.: ```console local $ cd my-local-dir -local $ ls | xargs -I% -n 1 -P 4 rsync % username@cluster-name.it4i.cz:my-remote-dir/. +local $ ls | xargs -n 2 -P 4 /bin/bash -c 'rsync "$@" username@cluster-name.it4i.cz:mydir' sh ``` -The -P YYY argument determines number of parallel rsync processes. Set to number of cores on your local machine. +The -n XXX argument detemines the number of files to transfer in one rsync call. Set according to file size and count. The -P YYY argument determines number of parallel rsync processes. Set to number of cores on your local machine. On Windows, use the [WinSCP client][c] to transfer data. The [win-sshfs client][d] provides a way to mount the cluster filesystems directly as an external disc.