From d3043001ecded627275e1ba324021c9f350bcdcf Mon Sep 17 00:00:00 2001
From: Jan Siwiec <jan.siwiec@vsb.cz>
Date: Tue, 23 Mar 2021 14:06:43 +0100
Subject: [PATCH 1/5] added cesnet storage file
---
docs.it4i/storage/cesnet-storage.md | 101 ++++++++++++++++++++++++++++
1 file changed, 101 insertions(+)
create mode 100644 docs.it4i/storage/cesnet-storage.md
diff --git a/docs.it4i/storage/cesnet-storage.md b/docs.it4i/storage/cesnet-storage.md
new file mode 100644
index 000000000..8bb1a9025
--- /dev/null
+++ b/docs.it4i/storage/cesnet-storage.md
@@ -0,0 +1,101 @@
+# CESNET Data Storage
+
+IT4Innovations' shared file systems are not intended as a backup for large amount of data or for long-term archiving purposes.
+Academic staff and students of research institutions in the Czech Republic should use [CESNET Storage service][a].
+
+CESNET data storage can be used by any organization or an individual person who is in the current employment relationship (employees) or the current study relationship (students) to a legal entity (organization) that meets the “Principles for access to CESNET Large infrastructure (Access Policy)”.
+
+The user may only use the CESNET data storage for data transfer and storage associated with activities in science, research, development, spread of education, culture, and prosperity. For details, see “Acceptable Use Policy CESNET Large Infrastructure (Acceptable Use Policy, AUP)”.
+
+The service is documented [here][b]. For special requirements contact directly CESNET Storage Department via e-mail [du-support(at)cesnet.cz][c].
+
+The procedure to obtain the CESNET access is quick and simple.
+
+## CESNET Storage Access
+
+### Understanding CESNET Storage
+
+!!! note
+ It is very important to understand the CESNET storage before uploading data. [Read][d] first.
+
+Once registered for CESNET Storage, you may [access the storage][e] in number of ways. We recommend the SSHFS and RSYNC methods.
+
+### SSHFS Access
+
+!!! note
+ SSHFS: The storage will be mounted like a local hard drive
+
+The SSHFS provides a very convenient way to access the CESNET Storage. The storage will be mounted onto a local directory, exposing the vast CESNET Storage as if it was a local removable hard drive. Files can be than copied in and out in a usual fashion.
+
+First, create the mount point:
+
+```console
+$ mkdir cesnet
+```
+
+Mount the storage. Note that you can choose among ssh.du4.cesnet.cz (Ostrava) and ssh.du5.cesnet.cz (Jihlava).
+Mount tier1_home **(only 5120M !)**:
+
+```console
+$ sshfs username@ssh.du4.cesnet.cz:. cesnet/
+```
+
+For easy future access, install your public key:
+
+```console
+$ cp .ssh/id_rsa.pub cesnet/.ssh/authorized_keys
+```
+
+Mount tier1_cache_tape for the Storage VO:
+
+```console
+$ sshfs username@ssh.du4.cesnet.cz:/cache_tape/VO_storage/home/username cesnet/
+```
+
+View the archive, copy the files and directories in and out:
+
+```console
+$ ls cesnet/
+$ cp -a mydir cesnet/.
+$ cp cesnet/myfile .
+```
+
+Once done, remember to unmount the storage:
+
+```console
+$ fusermount -u cesnet
+```
+
+### Rsync Access
+
+!!! note
+ Rsync provides delta transfer for best performance and can resume interrupted transfers.
+
+Rsync is a fast and extraordinarily versatile file copying tool. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
+
+Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated.
+
+More about Rsync [here][f].
+
+Transfer large files to/from CESNET storage, assuming membership in the Storage VO:
+
+```console
+$ rsync --progress datafile username@ssh.du4.cesnet.cz:VO_storage-cache_tape/.
+$ rsync --progress username@ssh.du4.cesnet.cz:VO_storage-cache_tape/datafile .
+```
+
+Transfer large directories to/from CESNET storage, assuming membership in the Storage VO:
+
+```console
+$ rsync --progress -av datafolder username@ssh.du4.cesnet.cz:VO_storage-cache_tape/.
+$ rsync --progress -av username@ssh.du4.cesnet.cz:VO_storage-cache_tape/datafolder .
+```
+
+Transfer rates of about 28MB/s can be expected.
+
+[a]: https://du.cesnet.cz/
+[b]: https://du.cesnet.cz/en/start
+[c]: mailto:du-support@cesnet.cz
+[d]: https://du.cesnet.cz/en/navody/home-migrace-plzen/start
+[e]: https://du.cesnet.cz/en/navody/faq/start)
+[f]: https://du.cesnet.cz/en/navody/rsync/start#pro_bezne_uzivatele
--
GitLab
From 7327af3b7e18ed64b0f1cee9a256020cb811e966 Mon Sep 17 00:00:00 2001
From: Jan Siwiec <jan.siwiec@vsb.cz>
Date: Tue, 23 Mar 2021 14:07:58 +0100
Subject: [PATCH 2/5] added cesnet storage to TOC
---
mkdocs.yml | 1 +
1 file changed, 1 insertion(+)
diff --git a/mkdocs.yml b/mkdocs.yml
index 1cb1cd5af..a6186b2aa 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -71,6 +71,7 @@ nav:
- Withdrawal from service: anselm-salomon-shutdown.md
- PROJECT Storage Availability: project-storage-availability.md
- Storage:
+ - Cesnet Storage: storage/cesnet-storage.md
- PROJECT: storage/project-storage.md
- Clusters:
- Barbora:
--
GitLab
From 54ddefa4ae235e5406c5ee1cb3bbbbee0250c1f1 Mon Sep 17 00:00:00 2001
From: Jan Siwiec <jan.siwiec@vsb.cz>
Date: Tue, 23 Mar 2021 14:11:30 +0100
Subject: [PATCH 3/5] removed cesnet section
---
docs.it4i/barbora/storage.md | 107 +----------------------------------
1 file changed, 1 insertion(+), 106 deletions(-)
diff --git a/docs.it4i/barbora/storage.md b/docs.it4i/barbora/storage.md
index d591683c4..6cce2e8a5 100644
--- a/docs.it4i/barbora/storage.md
+++ b/docs.it4i/barbora/storage.md
@@ -251,107 +251,9 @@ Each node is equipped with RAMDISK storage accessible at /tmp, /lscratch and /ra
| /scratch | scratch temoporary | Lustre | 310TB | 5GB/s, 30GB/s burst buffer | Quota 10TB | Compute and login nodes |files older 90 days autoremoved |
| /lscratch | local scratch ramdisk | tmpfs | 180GB | 130GB/s | none | Node local | auto purged after job end |
-## CESNET Data Storage
-
-Do not use shared filesystems at IT4Innovations as a backup for large amount of data or long-term archiving purposes.
-
-!!! note
- IT4Innovations does not provide storage capacity for data archiving. Academic staff and students of research institutions in the Czech Republic can use [CESNET Storage service][f].
-
-The CESNET Storage service can be used for research purposes, mainly by academic staff and students of research institutions in the Czech Republic.
-
-Users of data storage CESNET (DU) association can become organizations or individual persons who are in the current employment relationship (employees) or the current study relationship (students) to a legal entity (organization) that meets the “Principles for access to CESNET Large infrastructure (Access Policy)”.
-
-User may only use data storage CESNET for data transfer and storage associated with activities in science, research, development, spread of education, culture, and prosperity. For details, see “Acceptable Use Policy CESNET Large Infrastructure (Acceptable Use Policy, AUP)”.
-
-The service is documented [here][g]. For special requirements contact directly CESNET Storage Department via e-mail [du-support(at)cesnet.cz][h].
-
-The procedure to obtain the CESNET access is quick and simple.
-
-## CESNET Storage Access
-
-### Understanding CESNET Storage
-
-!!! note
- It is very important to understand the CESNET storage before uploading data. [Read][i] first.
-
-Once registered for CESNET Storage, you may [access the storage][j] in number of ways. We recommend the SSHFS and RSYNC methods.
-
-### SSHFS Access
-
-!!! note
- SSHFS: The storage will be mounted like a local hard drive
-
-The SSHFS provides a very convenient way to access the CESNET Storage. The storage will be mounted onto a local directory, exposing the vast CESNET Storage as if it was a local removable hard drive. Files can be than copied in and out in the usual fashion.
-
-First, create the mount point:
-
-```console
-$ mkdir cesnet
-```
-
-Mount the storage. Note that you can choose among `ssh.du4.cesnet.cz` (Ostrava) and `ssh.du5.cesnet.cz` (Jihlava). Mount tier1_home **(only 5120MB!)**:
-
-```console
-$ sshfs username@ssh.du4.cesnet.cz:. cesnet/
-```
-
-For convenient future access from Barbora, install your public key:
-
-```console
-$ cp .ssh/id_rsa.pub cesnet/.ssh/authorized_keys
-```
-
-Mount tier1_cache_tape for the Storage VO:
-
-```console
-$ sshfs username@ssh.du4.cesnet.cz:/cache_tape/VO_storage/home/username cesnet/
-```
-
-View the archive, copy the files and directories in and out:
-
-```console
-$ ls cesnet/
-$ cp -a mydir cesnet/.
-$ cp cesnet/myfile .
-```
-
-Once done, remember to unmount the storage:
-
-```console
-$ fusermount -u cesnet
-```
-
-### RSYNC Access
-
-!!! info
- RSYNC provides delta transfer for best performance and can resume interrupted transfers.
-
-RSYNC is a fast and extraordinarily versatile file copying tool. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. RSYNC is widely used for backups and mirroring and as an improved copy command for everyday use.
-
-RSYNC finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated.
-
-[More about RSYNC][k].
-
-Transfer large files to/from CESNET storage, assuming membership in the Storage VO:
-
-```console
-$ rsync --progress datafile username@ssh.du4.cesnet.cz:VO_storage-cache_tape/.
-$ rsync --progress username@ssh.du4.cesnet.cz:VO_storage-cache_tape/datafile .
-```
-
-Transfer large directories to/from CESNET storage, assuming membership in the Storage VO:
-
-```console
-$ rsync --progress -av datafolder username@ssh.du4.cesnet.cz:VO_storage-cache_tape/.
-$ rsync --progress -av username@ssh.du4.cesnet.cz:VO_storage-cache_tape/datafolder .
-```
-
-Transfer rates of about 28 MB/s can be expected.
-
[1]: #home-file-system
[2]: #scratch-file-system
-[3]: #cesnet-data-storage
+[3]: ../storage/cesnet-storage.md
[4]: ../general/obtaining-login-credentials/obtaining-login-credentials.md
[5]: #project-file-system
[6]: ../storage/project-storage.md
@@ -361,10 +263,3 @@ Transfer rates of about 28 MB/s can be expected.
[c]: http://doc.lustre.org/lustre_manual.xhtml#managingstripingfreespace
[d]: https://support.it4i.cz/rt
[e]: http://man7.org/linux/man-pages/man1/nfs4_setfacl.1.html
-[f]: https://du.cesnet.cz/
-[g]: https://du.cesnet.cz/en/start
-[h]: mailto:du-support@cesnet.cz
-[i]: https://du.cesnet.cz/en/navody/home-migrace-plzen/start
-[j]: https://du.cesnet.cz/en/navody/faq/start
-[k]: https://du.cesnet.cz/en/navody/rsync/start#pro_bezne_uzivatele
-[l]: http://man7.org/linux/man-pages/man1/nfs4_getfacl.1.html
--
GitLab
From dc39ec095e7cdaee2d852283d0a997b719e78567 Mon Sep 17 00:00:00 2001
From: Jan Siwiec <jan.siwiec@vsb.cz>
Date: Tue, 23 Mar 2021 14:13:26 +0100
Subject: [PATCH 4/5] removed cesnet storage section
---
docs.it4i/salomon/storage.md | 107 +----------------------------------
1 file changed, 1 insertion(+), 106 deletions(-)
diff --git a/docs.it4i/salomon/storage.md b/docs.it4i/salomon/storage.md
index 640844b2e..ad024e000 100644
--- a/docs.it4i/salomon/storage.md
+++ b/docs.it4i/salomon/storage.md
@@ -398,111 +398,12 @@ N = number of compute nodes in the job.
N = number of compute nodes in the job.
-## CESNET Data Storage
-
-Do not use shared file systems at IT4Innovations as a backup for large amount of data or long-term archiving purposes.
-
-!!! note
- IT4Innovations does not provide storage capacity for data archiving. Academic staff and students of research institutions in the Czech Republic can use [CESNET Storage service][e].
-
-The CESNET Storage service can be used for research purposes, mainly by academic staff and students of research institutions in the Czech Republic.
-
-The user of data storage CESNET (DU) association can be an organization or an individual person who is in the current employment relationship (employees) or the current study relationship (students) to a legal entity (organization) that meets the “Principles for access to CESNET Large infrastructure (Access Policy)”.
-
-The user may only use the CESNET data storage for data transfer and storage associated with activities in science, research, development, spread of education, culture, and prosperity. For details, see “Acceptable Use Policy CESNET Large Infrastructure (Acceptable Use Policy, AUP)”.
-
-The service is documented [here][f]. For special requirements contact directly CESNET Storage Department via e-mail [du-support(at)cesnet.cz][g].
-
-The procedure to obtain the CESNET access is quick and simple.
-
-## CESNET Storage Access
-
-### Understanding CESNET Storage
-
-!!! note
- It is very important to understand the CESNET storage before uploading data. [Read][h] first.
-
-Once registered for CESNET Storage, you may [access the storage][i] in number of ways. We recommend the SSHFS and RSYNC methods.
-
-### SSHFS Access
-
-!!! note
- SSHFS: The storage will be mounted like a local hard drive
-
-The SSHFS provides a very convenient way to access the CESNET Storage. The storage will be mounted onto a local directory, exposing the vast CESNET Storage as if it was a local removable hard drive. Files can be than copied in and out in a usual fashion.
-
-First, create the mount point:
-
-```console
-$ mkdir cesnet
-```
-
-Mount the storage. Note that you can choose among ssh.du4.cesnet.cz (Ostrava) and ssh.du5.cesnet.cz (Jihlava).
-Mount tier1_home **(only 5120M !)**:
-
-```console
-$ sshfs username@ssh.du4.cesnet.cz:. cesnet/
-```
-
-For easy future access, install your public key:
-
-```console
-$ cp .ssh/id_rsa.pub cesnet/.ssh/authorized_keys
-```
-
-Mount tier1_cache_tape for the Storage VO:
-
-```console
-$ sshfs username@ssh.du4.cesnet.cz:/cache_tape/VO_storage/home/username cesnet/
-```
-
-View the archive, copy the files and directories in and out:
-
-```console
-$ ls cesnet/
-$ cp -a mydir cesnet/.
-$ cp cesnet/myfile .
-```
-
-Once done, remember to unmount the storage:
-
-```console
-$ fusermount -u cesnet
-```
-
-### Rsync Access
-
-!!! note
- Rsync provides delta transfer for best performance and can resume interrupted transfers.
-
-Rsync is a fast and extraordinarily versatile file copying tool. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
-
-Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated.
-
-More about Rsync [here][j].
-
-Transfer large files to/from CESNET storage, assuming membership in the Storage VO:
-
-```console
-$ rsync --progress datafile username@ssh.du4.cesnet.cz:VO_storage-cache_tape/.
-$ rsync --progress username@ssh.du4.cesnet.cz:VO_storage-cache_tape/datafile .
-```
-
-Transfer large directories to/from CESNET storage, assuming membership in the Storage VO:
-
-```console
-$ rsync --progress -av datafolder username@ssh.du4.cesnet.cz:VO_storage-cache_tape/.
-$ rsync --progress -av username@ssh.du4.cesnet.cz:VO_storage-cache_tape/datafolder .
-```
-
-Transfer rates of about 28MB/s can be expected.
-
[1]: #home
[2]: #shared-filesystems
[3]: #work
[4]: #temp
[5]: #archiving
-[6]: #cesnet-data-storage
+[6]: ../storage/cesnet-storage.md
[7]: #home-filesystem
[8]: #scratch-filesystem
[9]: #shared-workspaces
@@ -510,9 +411,3 @@ Transfer rates of about 28MB/s can be expected.
[c]: https://access.redhat.com/documentation/en-US/Red_Hat_Storage/2.0/html/Administration_Guide/ch09s05.html
[d]: https://support.it4i.cz/rt
-[e]: https://du.cesnet.cz/
-[f]: https://du.cesnet.cz/en/start
-[g]: mailto:du-support@cesnet.cz
-[h]: https://du.cesnet.cz/en/navody/home-migrace-plzen/start
-[i]: https://du.cesnet.cz/en/navody/faq/start)
-[j]: https://du.cesnet.cz/en/navody/rsync/start#pro_bezne_uzivatele
--
GitLab
From eb222c1550af89ad4cb1c12db16578d6fa39cf31 Mon Sep 17 00:00:00 2001
From: Jan Siwiec <jan.siwiec@vsb.cz>
Date: Tue, 23 Mar 2021 14:15:26 +0100
Subject: [PATCH 5/5] Update mkdocs.yml
---
mkdocs.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/mkdocs.yml b/mkdocs.yml
index a6186b2aa..5a2c1af6f 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -71,7 +71,7 @@ nav:
- Withdrawal from service: anselm-salomon-shutdown.md
- PROJECT Storage Availability: project-storage-availability.md
- Storage:
- - Cesnet Storage: storage/cesnet-storage.md
+ - CESNET: storage/cesnet-storage.md
- PROJECT: storage/project-storage.md
- Clusters:
- Barbora:
--
GitLab