diff --git a/docs.it4i/general/shell-and-data-access.md b/docs.it4i/general/shell-and-data-access.md index 3ff4e773281bb773f152ab74c32e5aea7c080034..6a0bcdcfa1685d75c9fbcf30c8c39b764cc849ae 100644 --- a/docs.it4i/general/shell-and-data-access.md +++ b/docs.it4i/general/shell-and-data-access.md @@ -149,6 +149,38 @@ local $ man scp local $ man sshfs ``` +The rsync client uses ssh to establish connection. + +```console +local $ rsync my-local-file +``` + +```console +local $ rsync -r my-local-dir username@cluster-name.it4i.cz:directory +``` + +!!! note + The data transfer speed is limited by the single-core ssh encryption speed to about **150 MB/s** + Use **parallel rsync** for unlimited transfers + +The parallel rsync transfers multiple files at the time, utilizing multiple cores to accelerate encryption. +First, set up ssh-agent single sign on: + +``` +console +local $ eval `ssh-agent` +local $ ssh-add +Enter passphrase for /home/user/.ssh/id_rsa: +``` +Then run the rsync in parallel: + +```console +ls my-local-dir | xargs -I% -n 2 -P 4 rsync % username@cluster-name.it4i.cz:directory/. +``` + +The -n XXX argument determines how many files will be transfered per single rsync process. The -P YYY argument +determines number of parallel rsync processes. Set to number of cores on your local machine. + On Windows, use the [WinSCP client][c] to transfer data. The [win-sshfs client][d] provides a way to mount the cluster filesystems directly as an external disc. ## Connection Restrictions