Skip to content
Snippets Groups Projects
Commit d00d04f8 authored by Branislav Jansik's avatar Branislav Jansik
Browse files

Update shell-and-data-access.md

parent 666015b2
No related branches found
No related tags found
No related merge requests found
Pipeline #33329 passed with warnings
...@@ -149,6 +149,38 @@ local $ man scp ...@@ -149,6 +149,38 @@ local $ man scp
local $ man sshfs local $ man sshfs
``` ```
The rsync client uses ssh to establish connection.
```console
local $ rsync my-local-file
```
```console
local $ rsync -r my-local-dir username@cluster-name.it4i.cz:directory
```
!!! note
The data transfer speed is limited by the single-core ssh encryption speed to about **150 MB/s**
Use **parallel rsync** for unlimited transfers
The parallel rsync transfers multiple files at the time, utilizing multiple cores to accelerate encryption.
First, set up ssh-agent single sign on:
```
console
local $ eval `ssh-agent`
local $ ssh-add
Enter passphrase for /home/user/.ssh/id_rsa:
```
Then run the rsync in parallel:
```console
ls my-local-dir | xargs -I% -n 2 -P 4 rsync % username@cluster-name.it4i.cz:directory/.
```
The -n XXX argument determines how many files will be transfered per single rsync process. The -P YYY argument
determines number of parallel rsync processes. Set to number of cores on your local machine.
On Windows, use the [WinSCP client][c] to transfer data. The [win-sshfs client][d] provides a way to mount the cluster filesystems directly as an external disc. On Windows, use the [WinSCP client][c] to transfer data. The [win-sshfs client][d] provides a way to mount the cluster filesystems directly as an external disc.
## Connection Restrictions ## Connection Restrictions
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment