Skip to content
Snippets Groups Projects
Commit 00651640 authored by Branislav Jansik's avatar Branislav Jansik
Browse files

Update shell-and-data-access.md

parent bf196aa6
No related branches found
No related tags found
No related merge requests found
Pipeline #33334 passed with warnings
......@@ -161,9 +161,9 @@ local $ rsync -r my-local-dir username@cluster-name.it4i.cz:directory
!!! note
The data transfer speed is limited by the single-core ssh encryption speed to about **150 MB/s**
Use **parallel rsync** for unlimited transfers
Run **multiple rsync** instances for unlimited transfers
The parallel rsync transfers multiple files at the time, utilizing multiple rsync processes and cores to accelerate encryption.
Parallel execution of multiple rsync processes utilizes multiple cores to accelerate encryption and multiple tcp streams for enhanced bandwidth.
First, set up ssh-agent single sign on:
```console
......@@ -172,14 +172,14 @@ local $ ssh-add
Enter passphrase for /home/user/.ssh/id_rsa:
```
Then run the rsync in parallel:
Then run multiple rsync instances in parallel, f.x.:
```console
local $ cd my-local-dir
local $ ls | xargs -I% -n 1 -P 4 rsync % username@cluster-name.it4i.cz:my-remote-dir/.
local $ ls | xargs -n 2 -P 4 /bin/bash -c 'rsync "$@" username@cluster-name.it4i.cz:mydir' sh
```
The -P YYY argument determines number of parallel rsync processes. Set to number of cores on your local machine.
The -n XXX argument detemines the number of files to transfer in one rsync call. Set according to file size and count. The -P YYY argument determines number of parallel rsync processes. Set to number of cores on your local machine.
On Windows, use the [WinSCP client][c] to transfer data. The [win-sshfs client][d] provides a way to mount the cluster filesystems directly as an external disc.
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment