Skip to content
Snippets Groups Projects
Commit 12952039 authored by Branislav Jansik's avatar Branislav Jansik
Browse files

Update shell-and-data-access.md

parent 8420fe0f
No related branches found
No related tags found
1 merge request!468Master
Pipeline #37800 failed
...@@ -172,8 +172,10 @@ local $ rsync -r my-local-dir username@cluster-name.it4i.cz:directory ...@@ -172,8 +172,10 @@ local $ rsync -r my-local-dir username@cluster-name.it4i.cz:directory
### Parallel Transfer ### Parallel Transfer
!!! note !!! note
The data transfer speed is limited by the single-core ssh encryption speed to about **250 MB/s** (750 MB/s in case of aes256-gcm@openssh.com cipher) The data transfer speed is limited by the single TCP stream and single-core ssh encryption speed to about **250 MB/s** (750 MB/s in case of aes256-gcm@openssh.com cipher)
Run **multiple** instances for unlimited transfers Run **multiple** streams for unlimited transfers
#### Many Files
Parallel execution of multiple rsync processes utilizes multiple cores to accelerate encryption and multiple tcp streams for enhanced bandwidth. Parallel execution of multiple rsync processes utilizes multiple cores to accelerate encryption and multiple tcp streams for enhanced bandwidth.
First, set up ssh-agent single sign on: First, set up ssh-agent single sign on:
...@@ -194,6 +196,20 @@ local $ ls | xargs -n 2 -P 4 /bin/bash -c 'rsync "$@" username@cluster-name.it4i ...@@ -194,6 +196,20 @@ local $ ls | xargs -n 2 -P 4 /bin/bash -c 'rsync "$@" username@cluster-name.it4i
The **-n** argument detemines the number of files to transfer in one rsync call. Set according to file size and count (large for many small files). The **-n** argument detemines the number of files to transfer in one rsync call. Set according to file size and count (large for many small files).
The **-P** argument determines number of parallel rsync processes. Set to number of cores on your local machine. The **-P** argument determines number of parallel rsync processes. Set to number of cores on your local machine.
Alternatively, use HyperQueue. First get HyperQueue binary, then run:
```console
local $ hq server start &
local $ hq worker start &
local $ find my-local-dir -type f | xargs -n 2 > jobfile
local $ hq submit --log=/dev/null --progress --array --each-line jobfile \
bash -c 'rsync -R $HQ_ENTRY username@cluster-name.it4i.cz:mydir'
```
#### Single Very Large File
### Data Transfer From Windows Clients
On Windows, use the [WinSCP client][c] to transfer data. The [win-sshfs client][d] provides a way to mount the cluster filesystems directly as an external disc. On Windows, use the [WinSCP client][c] to transfer data. The [win-sshfs client][d] provides a way to mount the cluster filesystems directly as an external disc.
## Connection Restrictions ## Connection Restrictions
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment