Skip to content
Snippets Groups Projects
Commit a936737c authored by Branislav Jansik's avatar Branislav Jansik
Browse files

Update shell-and-data-access.md

parent 29c9e3ca
No related branches found
No related tags found
1 merge request!468Master
Pipeline #37818 failed
...@@ -196,7 +196,7 @@ local $ ls | xargs -n 2 -P 4 /bin/bash -c 'rsync "$@" username@cluster-name.it4i ...@@ -196,7 +196,7 @@ local $ ls | xargs -n 2 -P 4 /bin/bash -c 'rsync "$@" username@cluster-name.it4i
The **-n** argument detemines the number of files to transfer in one rsync call. Set according to file size and count (large for many small files). The **-n** argument detemines the number of files to transfer in one rsync call. Set according to file size and count (large for many small files).
The **-P** argument determines number of parallel rsync processes. Set to number of cores on your local machine. The **-P** argument determines number of parallel rsync processes. Set to number of cores on your local machine.
Alternatively, use HyperQueue. First get HyperQueue binary, then run: Alternatively, use [HyperQueue][11]. First get [HyperQueue binary][e], then run:
```console ```console
local $ hq server start & local $ hq server start &
...@@ -213,7 +213,7 @@ Again, the **-n** argument detemines the number of files to transfer in one rsyn ...@@ -213,7 +213,7 @@ Again, the **-n** argument detemines the number of files to transfer in one rsyn
To transfer single very large file efficienty, we need to transfer many blocks of the file in parallel, utilizing multiple cores to accelerate ssh encryption and multiple tcp streams for enhanced bandwidth. To transfer single very large file efficienty, we need to transfer many blocks of the file in parallel, utilizing multiple cores to accelerate ssh encryption and multiple tcp streams for enhanced bandwidth.
First, set up ssh-agent single sign on as [described above][10]. First, set up ssh-agent single sign on as [described above][10].
Second, start the HyperQueue server and HyperQueue worker: Second, start the [HyperQueue server and HyperQueue worker][f]:
```console ```console
local $ hq server start & local $ hq server start &
...@@ -356,8 +356,11 @@ Now, configure the applications proxy settings to `localhost:6000`. Use port for ...@@ -356,8 +356,11 @@ Now, configure the applications proxy settings to `localhost:6000`. Use port for
[7]: ../general/accessing-the-clusters/graphical-user-interface/vnc.md [7]: ../general/accessing-the-clusters/graphical-user-interface/vnc.md
[8]: ../general/accessing-the-clusters/vpn-access.md [8]: ../general/accessing-the-clusters/vpn-access.md
[9]: #port-forwarding-from-compute-nodes [9]: #port-forwarding-from-compute-nodes
[10]: [10]: #many-files
[11]: ../general/hyperqueue.md
[b]: http://linux.die.net/man/1/sshfs [b]: http://linux.die.net/man/1/sshfs
[c]: http://winscp.net/eng/download.php [c]: http://winscp.net/eng/download.php
[d]: http://code.google.com/p/win-sshfs/ [d]: http://code.google.com/p/win-sshfs/
[e]: https://github.com/It4innovations/hyperqueue/releases/latest
[f]: https://it4innovations.github.io/hyperqueue/stable/cheatsheet/
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment