@@ -172,8 +172,10 @@ local $ rsync -r my-local-dir username@cluster-name.it4i.cz:directory
...
@@ -172,8 +172,10 @@ local $ rsync -r my-local-dir username@cluster-name.it4i.cz:directory
### Parallel Transfer
### Parallel Transfer
!!! note
!!! note
The data transfer speed is limited by the single-core ssh encryption speed to about **250 MB/s** (750 MB/s in case of aes256-gcm@openssh.com cipher)
The data transfer speed is limited by the single TCP stream and single-core ssh encryption speed to about **250 MB/s** (750 MB/s in case of aes256-gcm@openssh.com cipher)
Run **multiple** instances for unlimited transfers
Run **multiple** streams for unlimited transfers
#### Many Files
Parallel execution of multiple rsync processes utilizes multiple cores to accelerate encryption and multiple tcp streams for enhanced bandwidth.
Parallel execution of multiple rsync processes utilizes multiple cores to accelerate encryption and multiple tcp streams for enhanced bandwidth.
First, set up ssh-agent single sign on:
First, set up ssh-agent single sign on:
...
@@ -194,6 +196,20 @@ local $ ls | xargs -n 2 -P 4 /bin/bash -c 'rsync "$@" username@cluster-name.it4i
...
@@ -194,6 +196,20 @@ local $ ls | xargs -n 2 -P 4 /bin/bash -c 'rsync "$@" username@cluster-name.it4i
The **-n** argument detemines the number of files to transfer in one rsync call. Set according to file size and count (large for many small files).
The **-n** argument detemines the number of files to transfer in one rsync call. Set according to file size and count (large for many small files).
The **-P** argument determines number of parallel rsync processes. Set to number of cores on your local machine.
The **-P** argument determines number of parallel rsync processes. Set to number of cores on your local machine.
Alternatively, use HyperQueue. First get HyperQueue binary, then run:
```console
local $hq server start &
local $hq worker start &
local $find my-local-dir -type f | xargs -n 2 > jobfile
local $hq submit --log=/dev/null --progress--array--each-line jobfile \
On Windows, use the [WinSCP client][c] to transfer data. The [win-sshfs client][d] provides a way to mount the cluster filesystems directly as an external disc.
On Windows, use the [WinSCP client][c] to transfer data. The [win-sshfs client][d] provides a way to mount the cluster filesystems directly as an external disc.