From 8420fe0fa8efab34d805a74b86c70566c18cc2af Mon Sep 17 00:00:00 2001 From: Branislav Jansik <branislav.jansik@vsb.cz> Date: Fri, 3 May 2024 14:14:43 +0200 Subject: [PATCH] Update shell-and-data-access.md --- docs.it4i/general/shell-and-data-access.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/docs.it4i/general/shell-and-data-access.md b/docs.it4i/general/shell-and-data-access.md index 15510007d..aa7531dad 100644 --- a/docs.it4i/general/shell-and-data-access.md +++ b/docs.it4i/general/shell-and-data-access.md @@ -108,6 +108,8 @@ After logging in, you will see the command prompt with the name of the cluster a ## Data Transfer +### Serial Transfer + Data in and out of the system may be transferred by SCP and SFTP protocols. | Cluster | Port | Protocol | @@ -133,6 +135,14 @@ or local $ sftp -o IdentityFile=/path/to/id_rsa username@cluster-name.it4i.cz ``` +You may request the **aes256-gcm@openssh.com cipher** for more efficient ssh based transfer: + +```console +local $ scp -c aes256-gcm@openssh.com -i /path/to/id_rsa -r my-local-dir username@cluster-name.it4i.cz:directory +``` + +The -c argument may be used with ssh, scp and sftp, and is also applicable to sshfs and rsync below. + A very convenient way to transfer files in and out of the cluster is via the fuse filesystem [SSHFS][b]. ```console @@ -159,9 +169,11 @@ local $ rsync my-local-file local $ rsync -r my-local-dir username@cluster-name.it4i.cz:directory ``` +### Parallel Transfer + !!! note - The data transfer speed is limited by the single-core ssh encryption speed to about **150 MB/s** - Run **multiple rsync** instances for unlimited transfers + The data transfer speed is limited by the single-core ssh encryption speed to about **250 MB/s** (750 MB/s in case of aes256-gcm@openssh.com cipher) + Run **multiple** instances for unlimited transfers Parallel execution of multiple rsync processes utilizes multiple cores to accelerate encryption and multiple tcp streams for enhanced bandwidth. First, set up ssh-agent single sign on: -- GitLab