diff --git a/docs.it4i/anselm-cluster-documentation/resource-allocation-and-job-execution/capacity-computing.md b/docs.it4i/anselm-cluster-documentation/capacity-computing.md similarity index 100% rename from docs.it4i/anselm-cluster-documentation/resource-allocation-and-job-execution/capacity-computing.md rename to docs.it4i/anselm-cluster-documentation/capacity-computing.md diff --git a/docs.it4i/anselm-cluster-documentation/resource-allocation-and-job-execution/capacity.zip b/docs.it4i/anselm-cluster-documentation/capacity.zip similarity index 100% rename from docs.it4i/anselm-cluster-documentation/resource-allocation-and-job-execution/capacity.zip rename to docs.it4i/anselm-cluster-documentation/capacity.zip diff --git a/docs.it4i/anselm-cluster-documentation/compute-nodes.md b/docs.it4i/anselm-cluster-documentation/compute-nodes.md index 85cf3c05bae514cc1a06b79127196affde8f1575..d6ddb121afa3a0a1e938fec3c9b6434ca7004c28 100644 --- a/docs.it4i/anselm-cluster-documentation/compute-nodes.md +++ b/docs.it4i/anselm-cluster-documentation/compute-nodes.md @@ -54,7 +54,7 @@ Anselm is cluster of x86-64 Intel based nodes built on Bull Extreme Computing bu ### Compute Nodes Summary - |Node type|Count|Range|Memory|Cores|[Access](resource-allocation-and-job-execution/resources-allocation-policy/)| + |Node type|Count|Range|Memory|Cores|[Access](resources-allocation-policy/)| |---|---|---|---|---|---| |Nodes without accelerator|180|cn[1-180]|64GB|16 @ 2.4Ghz|qexp, qprod, qlong, qfree| |Nodes with GPU accelerator|23|cn[181-203]|96GB|16 @ 2.3Ghz|qgpu, qprod| diff --git a/docs.it4i/anselm-cluster-documentation/hardware-overview.md b/docs.it4i/anselm-cluster-documentation/hardware-overview.md index f435c50bb71f5cc968ddaf3f0326e43bac700231..5aac8f98ee189a6754ec7cf0da6497b61b60dc0c 100644 --- a/docs.it4i/anselm-cluster-documentation/hardware-overview.md +++ b/docs.it4i/anselm-cluster-documentation/hardware-overview.md @@ -20,14 +20,14 @@ There are four types of compute nodes: [More about Compute nodes](compute-nodes/). -GPU and accelerated nodes are available upon request, see the [Resources Allocation Policy](resource-allocation-and-job-execution/resources-allocation-policy/). +GPU and accelerated nodes are available upon request, see the [Resources Allocation Policy](resources-allocation-policy/). All these nodes are interconnected by fast InfiniBand network and Ethernet network. [More about the Network](network/). Every chassis provides Infiniband switch, marked **isw**, connecting all nodes in the chassis, as well as connecting the chassis to the upper level switches. -All nodes share 360TB /home disk storage to store user files. The 146TB shared /scratch storage is available for the scratch data. These file systems are provided by Lustre parallel file system. There is also local disk storage available on all compute nodes /lscratch. [More about Storage](storage/storage/). +All nodes share 360TB /home disk storage to store user files. The 146TB shared /scratch storage is available for the scratch data. These file systems are provided by Lustre parallel file system. There is also local disk storage available on all compute nodes /lscratch. [More about Storage](storage/). -The user access to the Anselm cluster is provided by two login nodes login1, login2, and data mover node dm1. [More about accessing cluster.](accessing-the-cluster/shell-and-data-access/) +The user access to the Anselm cluster is provided by two login nodes login1, login2, and data mover node dm1. [More about accessing cluster.](shell-and-data-access/) The parameters are summarized in the following tables: @@ -58,4 +58,4 @@ The parameters are summarized in the following tables: |MIC accelerated|2x Intel Sandy Bridge E5-2470, 2.3GHz|96GB|Intel Xeon Phi P5110| |Fat compute node|2x Intel Sandy Bridge E5-2665, 2.4GHz|512GB|-| -For more details please refer to the [Compute nodes](compute-nodes/), [Storage](storage/storage/), and [Network](network/). +For more details please refer to the [Compute nodes](compute-nodes/), [Storage](storage/), and [Network](network/). diff --git a/docs.it4i/anselm-cluster-documentation/introduction.md b/docs.it4i/anselm-cluster-documentation/introduction.md index 6db81fd060f288f14e5f238fa19bd493dcb290f9..fb46ed5451253fb547770bb1a847b5c57f636c23 100644 --- a/docs.it4i/anselm-cluster-documentation/introduction.md +++ b/docs.it4i/anselm-cluster-documentation/introduction.md @@ -7,7 +7,7 @@ The cluster runs bullx Linux ([bull](http://www.bull.com/bullx-logiciels/systeme User data shared file-system (HOME, 320TB) and job data shared file-system (SCRATCH, 146TB) are available to users. -The PBS Professional workload manager provides [computing resources allocations and job execution](resource-allocation-and-job-execution/introduction/). +The PBS Professional workload manager provides [computing resources allocations and job execution](resource-allocation-and-job-execution/). -Read more on how to [apply for resources](../get-started-with-it4innovations/applying-for-resources/), [obtain login credentials,](../get-started-with-it4innovations/obtaining-login-credentials/obtaining-login-credentials/) and [access the cluster](accessing-the-cluster/shell-and-data-access/). +Read more on how to [apply for resources](../get-started-with-it4innovations/applying-for-resources/), [obtain login credentials,](../get-started-with-it4innovations/obtaining-login-credentials/obtaining-login-credentials/) and [access the cluster](/shell-and-data-access/). diff --git a/docs.it4i/anselm-cluster-documentation/resource-allocation-and-job-execution/job-priority.md b/docs.it4i/anselm-cluster-documentation/job-priority.md similarity index 100% rename from docs.it4i/anselm-cluster-documentation/resource-allocation-and-job-execution/job-priority.md rename to docs.it4i/anselm-cluster-documentation/job-priority.md diff --git a/docs.it4i/anselm-cluster-documentation/resource-allocation-and-job-execution/job-submission-and-execution.md b/docs.it4i/anselm-cluster-documentation/job-submission-and-execution.md similarity index 99% rename from docs.it4i/anselm-cluster-documentation/resource-allocation-and-job-execution/job-submission-and-execution.md rename to docs.it4i/anselm-cluster-documentation/job-submission-and-execution.md index 4f91a18ea914db14f5398bad2b478d365d072d8c..3d3bb34e62130c48205c35956ac487c80aee8055 100644 --- a/docs.it4i/anselm-cluster-documentation/resource-allocation-and-job-execution/job-submission-and-execution.md +++ b/docs.it4i/anselm-cluster-documentation/job-submission-and-execution.md @@ -342,7 +342,7 @@ exit In this example, some directory on the /home holds the input file input and executable mympiprog.x . We create a directory myjob on the /scratch filesystem, copy input and executable files from the /home directory where the qsub was invoked ($PBS_O_WORKDIR) to /scratch, execute the MPI programm mympiprog.x and copy the output file back to the /home directory. The mympiprog.x is executed as one process per node, on all allocated nodes. !!! Note "Note" - Consider preloading inputs and executables onto [shared scratch](../storage/storage/) before the calculation starts. + Consider preloading inputs and executables onto [shared scratch](storage/) before the calculation starts. In some cases, it may be impractical to copy the inputs to scratch and outputs to home. This is especially true when very large input and output files are expected, or when the files should be reused by a subsequent calculation. In such a case, it is users responsibility to preload the input files on shared /scratch before the job submission and retrieve the outputs manually, after all calculations are finished. @@ -382,7 +382,7 @@ sections. !!! Note "Note" Local scratch directory is often useful for single node jobs. Local scratch will be deleted immediately after the job ends. -Example jobscript for single node calculation, using [local scratch](../storage/storage/) on the node: +Example jobscript for single node calculation, using [local scratch](storage/) on the node: ```bash #!/bin/bash diff --git a/docs.it4i/anselm-cluster-documentation/prace.md b/docs.it4i/anselm-cluster-documentation/prace.md index 8bf2f5c696785727b4fb0e6ba79971d3b2ebbdad..d8699bf6e7487f5ce48315b8226974ae0e2191a5 100644 --- a/docs.it4i/anselm-cluster-documentation/prace.md +++ b/docs.it4i/anselm-cluster-documentation/prace.md @@ -197,7 +197,7 @@ Generally both shared file systems are available through GridFTP: |/home|Lustre|Default HOME directories of users in format /home/prace/login/| |/scratch|Lustre|Shared SCRATCH mounted on the whole cluster| -More information about the shared file systems is available [here](storage/storage/). +More information about the shared file systems is available [here](storage/). Usage of the cluster -------------------- @@ -217,7 +217,7 @@ PRACE users can use the "prace" module to use the [PRACE Common Production Envir ### Resource Allocation and Job Execution -General information about the resource allocation, job queuing and job execution is in this [section of general documentation](resource-allocation-and-job-execution/introduction/). +General information about the resource allocation, job queuing and job execution is in this [section of general documentation](resource-allocation-and-job-execution/). For PRACE users, the default production run queue is "qprace". PRACE users can also use two other queues "qexp" and "qfree". diff --git a/docs.it4i/anselm-cluster-documentation/resource-allocation-and-job-execution/introduction.md b/docs.it4i/anselm-cluster-documentation/resource-allocation-and-job-execution.md similarity index 100% rename from docs.it4i/anselm-cluster-documentation/resource-allocation-and-job-execution/introduction.md rename to docs.it4i/anselm-cluster-documentation/resource-allocation-and-job-execution.md diff --git a/docs.it4i/anselm-cluster-documentation/resource-allocation-and-job-execution/resources-allocation-policy.md b/docs.it4i/anselm-cluster-documentation/resources-allocation-policy.md similarity index 100% rename from docs.it4i/anselm-cluster-documentation/resource-allocation-and-job-execution/resources-allocation-policy.md rename to docs.it4i/anselm-cluster-documentation/resources-allocation-policy.md diff --git a/docs.it4i/anselm-cluster-documentation/shell-and-data-access.md b/docs.it4i/anselm-cluster-documentation/shell-and-data-access.md index efa04a72cbcf3ffa1e0c3154baefdd15b59d5946..6831c121198b259875a8c59fb46777ae602eb0c8 100644 --- a/docs.it4i/anselm-cluster-documentation/shell-and-data-access.md +++ b/docs.it4i/anselm-cluster-documentation/shell-and-data-access.md @@ -11,7 +11,7 @@ The Anselm cluster is accessed by SSH protocol via login nodes login1 and login2 |login1.anselm.it4i.cz|22|ssh|login1| |login2.anselm.it4i.cz|22|ssh|login2| -The authentication is by the [private key](../../../get-started-with-it4innovations/accessing-the-clusters/shell-access-and-data-transfer/ssh-keys/) +The authentication is by the [private key](../get-started-with-it4innovations/accessing-the-clusters/shell-access-and-data-transfer/ssh-keys/) !!! Note "Note" Please verify SSH fingerprints during the first logon. They are identical on all login nodes: @@ -33,7 +33,7 @@ If you see warning message "UNPROTECTED PRIVATE KEY FILE!", use this command to local $ chmod 600 /path/to/id_rsa ``` -On **Windows**, use [PuTTY ssh client](../../../get-started-with-it4innovations/accessing-the-clusters/shell-access-and-data-transfer/putty/putty/). +On **Windows**, use [PuTTY ssh client](../get-started-with-it4innovations/accessing-the-clusters/shell-access-and-data-transfer/putty.md). After logging in, you will see the command prompt: @@ -54,9 +54,8 @@ Last login: Tue Jul 9 15:57:38 2013 from your-host.example.com Example to the cluster login: -<tty-player controls src=/src/anselm/login_anselm.ttyrec></tty-player> - ->The environment is **not** shared between login nodes, except for [shared filesystems](../storage/storage/#section-1). +!!! Note "Note" + The environment is **not** shared between login nodes, except for [shared filesystems](storage/#shared-filesystems). Data Transfer ------------- @@ -69,7 +68,7 @@ Data in and out of the system may be transferred by the [scp](http://en.wikipedi |login2.anselm.it4i.cz|22|scp, sftp| |dm1.anselm.it4i.cz|22|scp, sftp| -The authentication is by the [private key](../../../get-started-with-it4innovations/accessing-the-clusters/shell-access-and-data-transfer/ssh-keys/) +The authentication is by the [private key](../get-started-with-it4innovations/accessing-the-clusters/shell-access-and-data-transfer/ssh-keys/) !!! Note "Note" Data transfer rates up to **160MB/s** can be achieved with scp or sftp. @@ -86,8 +85,6 @@ On linux or Mac, use scp or sftp client to transfer the data to Anselm: ```bash local $ scp -i /path/to/id_rsa my-local-file username@anselm.it4i.cz:directory/file ``` -For example how to using scp command: -<tty-player controls src=/src/anselm/transfer_scp_anselm.ttyrec></tty-player> ```bash local $ scp -i /path/to/id_rsa -r my-local-dir username@anselm.it4i.cz:directory @@ -98,8 +95,6 @@ or ```bash local $ sftp -o IdentityFile=/path/to/id_rsa username@anselm.it4i.cz ``` -For example how to using sftp command: -<tty-player controls src=/src/anselm/transfer_sftp_anselm.ttyrec></tty-player> Very convenient way to transfer files in and out of the Anselm computer is via the fuse filesystem [sshfs](http://linux.die.net/man/1/sshfs) @@ -119,7 +114,7 @@ $ man sshfs On Windows, use [WinSCP client](http://winscp.net/eng/download.php) to transfer the data. The [win-sshfs client](http://code.google.com/p/win-sshfs/) provides a way to mount the Anselm filesystems directly as an external disc. -More information about the shared file systems is available [here](../../storage/storage/). +More information about the shared file systems is available [here](storage/). Connection restrictions @@ -170,8 +165,7 @@ Note: Port number 6000 is chosen as an example only. Pick any free port. Remote port forwarding from compute nodes allows applications running on the compute nodes to access hosts outside Anselm Cluster. -First, establish the remote port forwarding form the login node, as [described -above](outgoing-connections.html#port-forwarding-from-login-nodes). +First, establish the remote port forwarding form the login node, as [described above](#port-forwarding-from-login-nodes). Second, invoke port forwarding from the compute node to the login node. Insert following line into your jobscript or interactive shell @@ -196,10 +190,10 @@ local $ ssh -D 1080 localhost On Windows, install and run the free, open source [Sock Puppet](http://sockspuppet.com/) server. -Once the proxy server is running, establish ssh port forwarding from Anselm to the proxy server, port 1080, exactly as [described above](outgoing-connections/#port-forwarding-from-login-nodes). +Once the proxy server is running, establish ssh port forwarding from Anselm to the proxy server, port 1080, exactly as [described above](#port-forwarding-from-login-nodes). ```bash local $ ssh -R 6000:localhost:1080 anselm.it4i.cz ``` -Now, configure the applications proxy settings to **localhost:6000**. Use port forwarding to access the [proxy server from compute nodes](outgoing-connections/#port-forwarding-from-compute-nodes) as well . +Now, configure the applications proxy settings to **localhost:6000**. Use port forwarding to access the [proxy server from compute nodes](#port-forwarding-from-compute-nodes) as well . diff --git a/docs.it4i/anselm-cluster-documentation/storage/storage.md b/docs.it4i/anselm-cluster-documentation/storage.md similarity index 75% rename from docs.it4i/anselm-cluster-documentation/storage/storage.md rename to docs.it4i/anselm-cluster-documentation/storage.md index 72486eb6c463ddcb93cb1bf87a269282a3edbbd7..3e607eccce26fa5a06e4bfee0967522e6f8d246d 100644 --- a/docs.it4i/anselm-cluster-documentation/storage/storage.md +++ b/docs.it4i/anselm-cluster-documentation/storage.md @@ -1,17 +1,17 @@ Storage ======= -There are two main shared file systems on Anselm cluster, the [HOME](../storage/#home) and [SCRATCH](../storage/#scratch). All login and compute nodes may access same data on shared filesystems. Compute nodes are also equipped with local (non-shared) scratch, ramdisk and tmp filesystems. +There are two main shared file systems on Anselm cluster, the [HOME](#home) and [SCRATCH](#scratch). All login and compute nodes may access same data on shared filesystems. Compute nodes are also equipped with local (non-shared) scratch, ramdisk and tmp filesystems. Archiving --------- -Please don't use shared filesystems as a backup for large amount of data or long-term archiving mean. The academic staff and students of research institutions in the Czech Republic can use [CESNET storage service](cesnet-data-storage/), which is available via SSHFS. +Please don't use shared filesystems as a backup for large amount of data or long-term archiving mean. The academic staff and students of research institutions in the Czech Republic can use [CESNET storage service](#cesnet-data-storage), which is available via SSHFS. Shared Filesystems ------------------ -Anselm computer provides two main shared filesystems, the [HOME filesystem](../storage.html#home) and the [SCRATCH filesystem](../storage/#scratch). Both HOME and SCRATCH filesystems are realized as a parallel Lustre filesystem. Both shared file systems are accessible via the Infiniband network. Extended ACLs are provided on both Lustre filesystems for the purpose of sharing data with other users using fine-grained control. +Anselm computer provides two main shared filesystems, the [HOME filesystem](#home) and the [SCRATCH filesystem](#scratch). Both HOME and SCRATCH filesystems are realized as a parallel Lustre filesystem. Both shared file systems are accessible via the Infiniband network. Extended ACLs are provided on both Lustre filesystems for the purpose of sharing data with other users using fine-grained control. ### Understanding the Lustre Filesystems @@ -320,3 +320,103 @@ Summary |/lscratch|node local jobs' data|local|330 GB|100 MB/s|none|Compute nodes|purged after job ends| |/ramdisk|node local jobs' data|local|60, 90, 500 GB|5-50 GB/s|none|Compute nodes|purged after job ends| |/tmp|local temporary files|local|9.5 GB|100 MB/s|none|Compute and login nodes|auto| purged + +CESNET Data Storage +------------ +Do not use shared filesystems at IT4Innovations as a backup for large amount of data or long-term archiving purposes. + +!!! Note "Note" + The IT4Innovations does not provide storage capacity for data archiving. Academic staff and students of research institutions in the Czech Republic can use [CESNET Storage service](https://du.cesnet.cz/). + +The CESNET Storage service can be used for research purposes, mainly by academic staff and students of research institutions in the Czech Republic. + +User of data storage CESNET (DU) association can become organizations or an individual person who is either in the current employment relationship (employees) or the current study relationship (students) to a legal entity (organization) that meets the “Principles for access to CESNET Large infrastructure (Access Policy)”. + +User may only use data storage CESNET for data transfer and storage which are associated with activities in science, research, development, the spread of education, culture and prosperity. In detail see “Acceptable Use Policy CESNET Large Infrastructure (Acceptable Use Policy, AUP)”. + +The service is documented at <https://du.cesnet.cz/wiki/doku.php/en/start>. For special requirements please contact directly CESNET Storage Department via e-mail [du-support(at)cesnet.cz](mailto:du-support@cesnet.cz). + +The procedure to obtain the CESNET access is quick and trouble-free. + +(source [https://du.cesnet.cz/](https://du.cesnet.cz/wiki/doku.php/en/start "CESNET Data Storage")) + +CESNET storage access +------------ +### Understanding Cesnet storage + +!!! Note "Note" + It is very important to understand the Cesnet storage before uploading data. Please read <https://du.cesnet.cz/en/navody/home-migrace-plzen/start> first. + +Once registered for CESNET Storage, you may [access the storage](https://du.cesnet.cz/en/navody/faq/start) in number of ways. We recommend the SSHFS and RSYNC methods. + +### SSHFS Access + +!!! Note "Note" + SSHFS: The storage will be mounted like a local hard drive + +The SSHFS provides a very convenient way to access the CESNET Storage. The storage will be mounted onto a local directory, exposing the vast CESNET Storage as if it was a local removable harddrive. Files can be than copied in and out in a usual fashion. + +First, create the mountpoint + +```bash + $ mkdir cesnet +``` + +Mount the storage. Note that you can choose among the ssh.du1.cesnet.cz (Plzen), ssh.du2.cesnet.cz (Jihlava), ssh.du3.cesnet.cz (Brno) Mount tier1_home **(only 5120M !)**: + +```bash + $ sshfs username@ssh.du1.cesnet.cz:. cesnet/ +``` + +For easy future access from Anselm, install your public key + +```bash + $ cp .ssh/id_rsa.pub cesnet/.ssh/authorized_keys +``` + +Mount tier1_cache_tape for the Storage VO: + +```bash + $ sshfs username@ssh.du1.cesnet.cz:/cache_tape/VO_storage/home/username cesnet/ +``` + +View the archive, copy the files and directories in and out + +```bash + $ ls cesnet/ + $ cp -a mydir cesnet/. + $ cp cesnet/myfile . +``` + +Once done, please remember to unmount the storage + +```bash + $ fusermount -u cesnet +``` + +### Rsync access + +!!! Note "Note" + Rsync provides delta transfer for best performance, can resume interrupted transfers + +Rsync is a fast and extraordinarily versatile file copying tool. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use. + +Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated. + +More about Rsync at <https://du.cesnet.cz/en/navody/rsync/start#pro_bezne_uzivatele> + +Transfer large files to/from Cesnet storage, assuming membership in the Storage VO + +```bash + $ rsync --progress datafile username@ssh.du1.cesnet.cz:VO_storage-cache_tape/. + $ rsync --progress username@ssh.du1.cesnet.cz:VO_storage-cache_tape/datafile . +``` + +Transfer large directories to/from Cesnet storage, assuming membership in the Storage VO + +```bash + $ rsync --progress -av datafolder username@ssh.du1.cesnet.cz:VO_storage-cache_tape/. + $ rsync --progress -av username@ssh.du1.cesnet.cz:VO_storage-cache_tape/datafolder . +``` + +Transfer rates of about 28MB/s can be expected. diff --git a/docs.it4i/anselm-cluster-documentation/storage/cesnet-data-storage.md b/docs.it4i/anselm-cluster-documentation/storage/cesnet-data-storage.md deleted file mode 100644 index 35dd71ba778f5b1e45a2acc8c979323a5041da49..0000000000000000000000000000000000000000 --- a/docs.it4i/anselm-cluster-documentation/storage/cesnet-data-storage.md +++ /dev/null @@ -1,103 +0,0 @@ -CESNET Data Storage -=================== - -Introduction ------------- -Do not use shared filesystems at IT4Innovations as a backup for large amount of data or long-term archiving purposes. - -!!! Note "Note" - The IT4Innovations does not provide storage capacity for data archiving. Academic staff and students of research institutions in the Czech Republic can use [CESNET Storage service](https://du.cesnet.cz/). - -The CESNET Storage service can be used for research purposes, mainly by academic staff and students of research institutions in the Czech Republic. - -User of data storage CESNET (DU) association can become organizations or an individual person who is either in the current employment relationship (employees) or the current study relationship (students) to a legal entity (organization) that meets the “Principles for access to CESNET Large infrastructure (Access Policy)”. - -User may only use data storage CESNET for data transfer and storage which are associated with activities in science, research, development, the spread of education, culture and prosperity. In detail see “Acceptable Use Policy CESNET Large Infrastructure (Acceptable Use Policy, AUP)”. - -The service is documented at <https://du.cesnet.cz/wiki/doku.php/en/start>. For special requirements please contact directly CESNET Storage Department via e-mail [du-support(at)cesnet.cz](mailto:du-support@cesnet.cz). - -The procedure to obtain the CESNET access is quick and trouble-free. - -(source [https://du.cesnet.cz/](https://du.cesnet.cz/wiki/doku.php/en/start "CESNET Data Storage")) - -CESNET storage access ---------------------- - -### Understanding Cesnet storage - -!!! Note "Note" - It is very important to understand the Cesnet storage before uploading data. Please read <https://du.cesnet.cz/en/navody/home-migrace-plzen/start> first. - -Once registered for CESNET Storage, you may [access the storage](https://du.cesnet.cz/en/navody/faq/start) in number of ways. We recommend the SSHFS and RSYNC methods. - -### SSHFS Access - -!!! Note "Note" - SSHFS: The storage will be mounted like a local hard drive - -The SSHFS provides a very convenient way to access the CESNET Storage. The storage will be mounted onto a local directory, exposing the vast CESNET Storage as if it was a local removable harddrive. Files can be than copied in and out in a usual fashion. - -First, create the mountpoint - -```bash - $ mkdir cesnet -``` - -Mount the storage. Note that you can choose among the ssh.du1.cesnet.cz (Plzen), ssh.du2.cesnet.cz (Jihlava), ssh.du3.cesnet.cz (Brno) Mount tier1_home **(only 5120M !)**: - -```bash - $ sshfs username@ssh.du1.cesnet.cz:. cesnet/ -``` - -For easy future access from Anselm, install your public key - -```bash - $ cp .ssh/id_rsa.pub cesnet/.ssh/authorized_keys -``` - -Mount tier1_cache_tape for the Storage VO: - -```bash - $ sshfs username@ssh.du1.cesnet.cz:/cache_tape/VO_storage/home/username cesnet/ -``` - -View the archive, copy the files and directories in and out - -```bash - $ ls cesnet/ - $ cp -a mydir cesnet/. - $ cp cesnet/myfile . -``` - -Once done, please remember to unmount the storage - -```bash - $ fusermount -u cesnet -``` - -### Rsync access - -!!! Note "Note" - Rsync provides delta transfer for best performance, can resume interrupted transfers - -Rsync is a fast and extraordinarily versatile file copying tool. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use. - -Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated. - -More about Rsync at <https://du.cesnet.cz/en/navody/rsync/start#pro_bezne_uzivatele> - -Transfer large files to/from Cesnet storage, assuming membership in the Storage VO - -```bash - $ rsync --progress datafile username@ssh.du1.cesnet.cz:VO_storage-cache_tape/. - $ rsync --progress username@ssh.du1.cesnet.cz:VO_storage-cache_tape/datafile . -``` - -Transfer large directories to/from Cesnet storage, assuming membership in the Storage VO - -```bash - $ rsync --progress -av datafolder username@ssh.du1.cesnet.cz:VO_storage-cache_tape/. - $ rsync --progress -av username@ssh.du1.cesnet.cz:VO_storage-cache_tape/datafolder . -``` - -Transfer rates of about 28MB/s can be expected. diff --git a/docs.it4i/modules-salomon-uv.md b/docs.it4i/modules-salomon-uv.md new file mode 100644 index 0000000000000000000000000000000000000000..3553cf37ac250455380087e155174d5298647c15 --- /dev/null +++ b/docs.it4i/modules-salomon-uv.md @@ -0,0 +1,3 @@ +| Name | Version | +| ---- | ------: | +| modulename | 3.0.3 | diff --git a/docs.it4i/salomon/network/7d-enhanced-hypercube.md b/docs.it4i/salomon/7d-enhanced-hypercube.md similarity index 100% rename from docs.it4i/salomon/network/7d-enhanced-hypercube.md rename to docs.it4i/salomon/7d-enhanced-hypercube.md diff --git a/docs.it4i/salomon/resource-allocation-and-job-execution/capacity-computing.md b/docs.it4i/salomon/capacity-computing.md similarity index 100% rename from docs.it4i/salomon/resource-allocation-and-job-execution/capacity-computing.md rename to docs.it4i/salomon/capacity-computing.md diff --git a/docs.it4i/salomon/resource-allocation-and-job-execution/capacity.zip b/docs.it4i/salomon/capacity.zip similarity index 100% rename from docs.it4i/salomon/resource-allocation-and-job-execution/capacity.zip rename to docs.it4i/salomon/capacity.zip diff --git a/docs.it4i/salomon/hardware-overview.md b/docs.it4i/salomon/hardware-overview.md index ca6a26750b605c10a7ab9f8b3bd206e33386a714..3ab8ea22f052e450965c8560416e5de865793f4e 100644 --- a/docs.it4i/salomon/hardware-overview.md +++ b/docs.it4i/salomon/hardware-overview.md @@ -5,7 +5,7 @@ Introduction ------------ The Salomon cluster consists of 1008 computational nodes of which 576 are regular compute nodes and 432 accelerated nodes. Each node is a powerful x86-64 computer, equipped with 24 cores (two twelve-core Intel Xeon processors) and 128GB RAM. The nodes are interlinked by high speed InfiniBand and Ethernet networks. All nodes share 0.5PB /home NFS disk storage to store the user files. Users may use a DDN Lustre shared storage with capacity of 1.69 PB which is available for the scratch project data. The user access to the Salomon cluster is provided by four login nodes. -[More about schematic representation of the Salomon cluster compute nodes IB topology](../network/ib-single-plane-topology/). +[More about schematic representation of the Salomon cluster compute nodes IB topology](ib-single-plane-topology/).  @@ -19,7 +19,7 @@ General information |Primary purpose|High Performance Computing| |Architecture of compute nodes|x86-64| |Operating system|CentOS 6.7 Linux| -|[**Compute nodes**](../compute-nodes/)|| +|[**Compute nodes**](compute-nodes/)|| |Totally|1008| |Processor|2x Intel Xeon E5-2680v3, 2.5GHz, 12cores| |RAM|128GB, 5.3GB per core, DDR4@2133 MHz| @@ -39,7 +39,7 @@ Compute nodes |w/o accelerator|576|2x Intel Xeon E5-2680v3, 2.5GHz|24|128GB|-| |MIC accelerated|432|2x Intel Xeon E5-2680v3, 2.5GHz|24|128GB|2x Intel Xeon Phi 7120P, 61cores, 16GB RAM| -For more details please refer to the [Compute nodes](../compute-nodes/). +For more details please refer to the [Compute nodes](compute-nodes/). Remote visualization nodes -------------------------- diff --git a/docs.it4i/salomon/network/ib-single-plane-topology.md b/docs.it4i/salomon/ib-single-plane-topology.md similarity index 87% rename from docs.it4i/salomon/network/ib-single-plane-topology.md rename to docs.it4i/salomon/ib-single-plane-topology.md index e3896525898082a6034c9befe9a1fc2bb144d938..31175bb1cb7e4210e1dbf686588b054f72c56ec9 100644 --- a/docs.it4i/salomon/network/ib-single-plane-topology.md +++ b/docs.it4i/salomon/ib-single-plane-topology.md @@ -13,16 +13,16 @@ The SGI ICE X IB Premium Blade provides the first level of interconnection via d Each colour in each physical IRU represents one dual-switch ASIC switch. - + ### IB single-plane topology - Accelerated nodes Each of the 3 inter-connected D racks are equivalent to one half of Mcell rack. 18x D rack with MIC accelerated nodes [r21-r38] are equivalent to 3 Mcell racks as shown in a diagram [7D Enhanced Hypercube](7d-enhanced-hypercube/). -As shown in a diagram  +As shown in a diagram  - Racks 21, 22, 23, 24, 25, 26 are equivalent to one Mcell rack. - Racks 27, 28, 29, 30, 31, 32 are equivalent to one Mcell rack. - Racks 33, 34, 35, 36, 37, 38 are equivalent to one Mcell rack. - + diff --git a/docs.it4i/salomon/resource-allocation-and-job-execution/job-priority.md b/docs.it4i/salomon/job-priority.md similarity index 100% rename from docs.it4i/salomon/resource-allocation-and-job-execution/job-priority.md rename to docs.it4i/salomon/job-priority.md diff --git a/docs.it4i/salomon/resource-allocation-and-job-execution/job-submission-and-execution.md b/docs.it4i/salomon/job-submission-and-execution.md similarity index 99% rename from docs.it4i/salomon/resource-allocation-and-job-execution/job-submission-and-execution.md rename to docs.it4i/salomon/job-submission-and-execution.md index 4be50e35c11cd0f6ce1c1718ccdf8827ec186e61..9738861315dadced3de4f71a1901c3765ee67bbd 100644 --- a/docs.it4i/salomon/resource-allocation-and-job-execution/job-submission-and-execution.md +++ b/docs.it4i/salomon/job-submission-and-execution.md @@ -402,7 +402,7 @@ exit In this example, some directory on the /home holds the input file input and executable mympiprog.x . We create a directory myjob on the /scratch filesystem, copy input and executable files from the /home directory where the qsub was invoked ($PBS_O_WORKDIR) to /scratch, execute the MPI programm mympiprog.x and copy the output file back to the /home directory. The mympiprog.x is executed as one process per node, on all allocated nodes. !!! Note "Note" - Consider preloading inputs and executables onto [shared scratch](../storage/storage/) before the calculation starts. + Consider preloading inputs and executables onto [shared scratch](storage/) before the calculation starts. In some cases, it may be impractical to copy the inputs to scratch and outputs to home. This is especially true when very large input and output files are expected, or when the files should be reused by a subsequent calculation. In such a case, it is users responsibility to preload the input files on shared /scratch before the job submission and retrieve the outputs manually, after all calculations are finished. @@ -441,7 +441,7 @@ HTML commented section #2 (examples need to be reworked) !!! Note "Note" Local scratch directory is often useful for single node jobs. Local scratch will be deleted immediately after the job ends. Be very careful, use of RAM disk filesystem is at the expense of operational memory. -Example jobscript for single node calculation, using [local scratch](../storage/storage/) on the node: +Example jobscript for single node calculation, using [local scratch](storage/) on the node: ```bash #!/bin/bash diff --git a/docs.it4i/salomon/network/network.md b/docs.it4i/salomon/network.md similarity index 100% rename from docs.it4i/salomon/network/network.md rename to docs.it4i/salomon/network.md diff --git a/docs.it4i/salomon/outgoing-connections.md b/docs.it4i/salomon/outgoing-connections.md deleted file mode 100644 index c6cdc119fbd917b5fe8044953dd82ea8dc11811c..0000000000000000000000000000000000000000 --- a/docs.it4i/salomon/outgoing-connections.md +++ /dev/null @@ -1,83 +0,0 @@ -Outgoing connections -==================== - -Connection restrictions ------------------------ -Outgoing connections, from Salomon Cluster login nodes to the outside world, are restricted to following ports: - -|Port|Protocol| -|---|---| -|22|ssh| -|80|http| -|443|https| -|9418|git| - -!!! Note "Note" - Please use **ssh port forwarding** and proxy servers to connect from Salomon to all other remote ports. - -Outgoing connections, from Salomon Cluster compute nodes are restricted to the internal network. Direct connections form compute nodes to outside world are cut. - -Port forwarding ---------------- - -### Port forwarding from login nodes - -!!! Note "Note" - Port forwarding allows an application running on Salomon to connect to arbitrary remote host and port. - -It works by tunneling the connection from Salomon back to users workstation and forwarding from the workstation to the remote host. - -Pick some unused port on Salomon login node (for example 6000) and establish the port forwarding: - -```bash -local $ ssh -R 6000:remote.host.com:1234 salomon.it4i.cz -``` - -In this example, we establish port forwarding between port 6000 on Salomon and port 1234 on the remote.host.com. By accessing localhost:6000 on Salomon, an application will see response of remote.host.com:1234. The traffic will run via users local workstation. - -Port forwarding may be done **using PuTTY** as well. On the PuTTY Configuration screen, load your Salomon configuration first. Then go to Connection->SSH->Tunnels to set up the port forwarding. Click Remote radio button. Insert 6000 to Source port textbox. Insert remote.host.com:1234. Click Add button, then Open. - -Port forwarding may be established directly to the remote host. However, this requires that user has ssh access to remote.host.com - -```bash -$ ssh -L 6000:localhost:1234 remote.host.com -``` - -Note: Port number 6000 is chosen as an example only. Pick any free port. - -### Port forwarding from compute nodes - -Remote port forwarding from compute nodes allows applications running on the compute nodes to access hosts outside Salomon Cluster. - -First, establish the remote port forwarding form the login node, as [described above](outgoing-connections/#port-forwarding-from-login-nodes). - -Second, invoke port forwarding from the compute node to the login node. Insert following line into your jobscript or interactive shell - -```bash -$ ssh -TN -f -L 6000:localhost:6000 login1 -``` - -In this example, we assume that port forwarding from login1:6000 to remote.host.com:1234 has been established beforehand. By accessing localhost:6000, an application running on a compute node will see response of remote.host.com:1234 - -### Using proxy servers - -Port forwarding is static, each single port is mapped to a particular port on remote host. Connection to other remote host, requires new forward. - -!!! Note "Note" - Applications with inbuilt proxy support, experience unlimited access to remote hosts, via single proxy server. - -To establish local proxy server on your workstation, install and run SOCKS proxy server software. On Linux, sshd demon provides the functionality. To establish SOCKS proxy server listening on port 1080 run: - -```bash -local $ ssh -D 1080 localhost -``` - -On Windows, install and run the free, open source [Sock Puppet](http://sockspuppet.com/) server. - -Once the proxy server is running, establish ssh port forwarding from Salomon to the proxy server, port 1080, exactly as [described above](outgoing-connections/#port-forwarding-from-login-nodes). - -```bash -local $ ssh -R 6000:localhost:1080 salomon.it4i.cz -``` - -Now, configure the applications proxy settings to **localhost:6000**. Use port forwarding to access the [proxy server from compute nodes](outgoing-connections/#port-forwarding-from-compute-nodes) as well . diff --git a/docs.it4i/salomon/prace.md b/docs.it4i/salomon/prace.md index f8162731d527bae379e5bcd9bf050d098651ff60..f3b4d4f566b4a06303b7bb944a852404bd2393ff 100644 --- a/docs.it4i/salomon/prace.md +++ b/docs.it4i/salomon/prace.md @@ -119,11 +119,11 @@ If the user uses GSI SSH based access, then the procedure is similar to the SSH ### Access with SSH -After successful obtainment of login credentials for the local IT4Innovations account, the PRACE users can access the cluster as regular users using SSH. For more information please see the [section in general documentation](accessing-the-cluster/shell-and-data-access/shell-and-data-access/). +After successful obtainment of login credentials for the local IT4Innovations account, the PRACE users can access the cluster as regular users using SSH. For more information please see the [section in general documentation](shell-and-data-access/). File transfers ------------------ -PRACE users can use the same transfer mechanisms as regular users (if they've undergone the full registration procedure). For information about this, please see [the section in the general documentation](accessing-the-cluster/shell-and-data-access/shell-and-data-access/). +PRACE users can use the same transfer mechanisms as regular users (if they've undergone the full registration procedure). For information about this, please see [the section in the general documentation](shell-and-data-access/). Apart from the standard mechanisms, for PRACE users to transfer data to/from Salomon cluster, a GridFTP server running Globus Toolkit GridFTP service is available. The service is available from public Internet as well as from the internal PRACE network (accessible only from other PRACE partners). @@ -215,7 +215,7 @@ Usage of the cluster -------------------- There are some limitations for PRACE user when using the cluster. By default PRACE users aren't allowed to access special queues in the PBS Pro to have high priority or exclusive access to some special equipment like accelerated nodes and high memory (fat) nodes. There may be also restrictions obtaining a working license for the commercial software installed on the cluster, mostly because of the license agreement or because of insufficient amount of licenses. -For production runs always use scratch file systems. The available file systems are described [here](storage/storage/). +For production runs always use scratch file systems. The available file systems are described [here](storage/). ### Software, Modules and PRACE Common Production Environment @@ -229,7 +229,7 @@ PRACE users can use the "prace" module to use the [PRACE Common Production Envir ### Resource Allocation and Job Execution -General information about the resource allocation, job queuing and job execution is in this [section of general documentation](resource-allocation-and-job-execution/introduction/). +General information about the resource allocation, job queuing and job execution is in this [section of general documentation](resource-allocation-and-job-execution/). For PRACE users, the default production run queue is "qprace". PRACE users can also use two other queues "qexp" and "qfree". @@ -244,7 +244,7 @@ For PRACE users, the default production run queue is "qprace". PRACE users can a ### Accounting & Quota -The resources that are currently subject to accounting are the core hours. The core hours are accounted on the wall clock basis. The accounting runs whenever the computational cores are allocated or blocked via the PBS Pro workload manager (the qsub command), regardless of whether the cores are actually used for any calculation. See [example in the general documentation](resource-allocation-and-job-execution/resources-allocation-policy/). +The resources that are currently subject to accounting are the core hours. The core hours are accounted on the wall clock basis. The accounting runs whenever the computational cores are allocated or blocked via the PBS Pro workload manager (the qsub command), regardless of whether the cores are actually used for any calculation. See [example in the general documentation](resources-allocation-policy/). PRACE users should check their project accounting using the [PRACE Accounting Tool (DART)](http://www.prace-ri.eu/accounting-report-tool/). diff --git a/docs.it4i/salomon/resource-allocation-and-job-execution/introduction.md b/docs.it4i/salomon/resource-allocation-and-job-execution.md similarity index 100% rename from docs.it4i/salomon/resource-allocation-and-job-execution/introduction.md rename to docs.it4i/salomon/resource-allocation-and-job-execution.md diff --git a/docs.it4i/salomon/resource-allocation-and-job-execution/resources-allocation-policy.md b/docs.it4i/salomon/resources-allocation-policy.md similarity index 100% rename from docs.it4i/salomon/resource-allocation-and-job-execution/resources-allocation-policy.md rename to docs.it4i/salomon/resources-allocation-policy.md diff --git a/docs.it4i/salomon/shell-and-data-access.md b/docs.it4i/salomon/shell-and-data-access.md index 69d613e8715b8624db59f12cddcc5014c8ab13d3..11825106273d43de0aaf09fdda45bdd322b330c2 100644 --- a/docs.it4i/salomon/shell-and-data-access.md +++ b/docs.it4i/salomon/shell-and-data-access.md @@ -20,7 +20,6 @@ The authentication is by the [private key](../get-started-with-it4innovations/ac !!! Note "Note" Please verify SSH fingerprints during the first logon. They are identical on all login nodes: - f6:28:98:e4:f9:b2:a6:8f:f2:f4:2d:0a:09:67:69:80 (DSA) 70:01:c9:9a:5d:88:91:c7:1b:c0:84:d1:fa:4e:83:5c (RSA) @@ -38,23 +37,17 @@ If you see warning message "UNPROTECTED PRIVATE KEY FILE!", use this command to local $ chmod 600 /path/to/id_rsa ``` -On **Windows**, use [PuTTY ssh client](../get-started-with-it4innovations/accessing-the-clusters/shell-access-and-data-transfer/putty/putty/). +On **Windows**, use [PuTTY ssh client](../get-started-with-it4innovations/accessing-the-clusters/shell-access-and-data-transfer/putty.md). After logging in, you will see the command prompt: -<tty-player controls src=/src/salomon/login_salomon.ttyrec></tty-player> - !!! Note "Note" - The environment is **not** shared between login nodes, except for [shared filesystems](storage/storage/). + The environment is **not** shared between login nodes, except for [shared filesystems](storage/). Data Transfer ------------- Data in and out of the system may be transferred by the [scp](http://en.wikipedia.org/wiki/Secure_copy) and sftp protocols. -In case large volumes of data are transferred, use dedicated data mover nodes cedge[1-3].salomon.it4i.cz for increased performance. - -HTML commented section #1 (removed cedge servers from the table) - |Address|Port|Protocol| |---|---| |salomon.it4i.cz|22|scp, sftp| @@ -65,17 +58,12 @@ HTML commented section #1 (removed cedge servers from the table) The authentication is by the [private key](../get-started-with-it4innovations/accessing-the-clusters/shell-access-and-data-transfer/ssh-keys/) -HTML commented section #2 (ssh transfer performance data need to be verified) - On linux or Mac, use scp or sftp client to transfer the data to Salomon: ```bash local $ scp -i /path/to/id_rsa my-local-file username@salomon.it4i.cz:directory/file ``` -For example how to using scp command: -<tty-player controls src=/src/salomon/transfer_scp_salomon.ttyrec></tty-player> - ```bash local $ scp -i /path/to/id_rsa -r my-local-dir username@salomon.it4i.cz:directory ``` @@ -86,9 +74,6 @@ or local $ sftp -o IdentityFile=/path/to/id_rsa username@salomon.it4i.cz ``` -For example how to using sftp command: -<tty-player controls src=/src/salomon/transfer_sftp_salomon.ttyrec></tty-player> - Very convenient way to transfer files in and out of the Salomon computer is via the fuse filesystem [sshfs](http://linux.die.net/man/1/sshfs) ```bash @@ -107,7 +92,7 @@ $ man sshfs On Windows, use [WinSCP client](http://winscp.net/eng/download.php) to transfer the data. The [win-sshfs client](http://code.google.com/p/win-sshfs/) provides a way to mount the Salomon filesystems directly as an external disc. -More information about the shared file systems is available [here](storage/storage/). +More information about the shared file systems is available [here](storage/). Connection restrictions ----------------------- @@ -157,7 +142,7 @@ Note: Port number 6000 is chosen as an example only. Pick any free port. Remote port forwarding from compute nodes allows applications running on the compute nodes to access hosts outside Salomon Cluster. -First, establish the remote port forwarding form the login node, as [described above](outgoing-connections/#port-forwarding-from-login-nodes). +First, establish the remote port forwarding form the login node, as [described above](#port-forwarding-from-login-nodes). Second, invoke port forwarding from the compute node to the login node. Insert following line into your jobscript or interactive shell @@ -182,10 +167,10 @@ local $ ssh -D 1080 localhost On Windows, install and run the free, open source [Sock Puppet](http://sockspuppet.com/) server. -Once the proxy server is running, establish ssh port forwarding from Salomon to the proxy server, port 1080, exactly as [described above](outgoing-connections/#port-forwarding-from-login-nodes). +Once the proxy server is running, establish ssh port forwarding from Salomon to the proxy server, port 1080, exactly as [described above](#port-forwarding-from-login-nodes). ```bash local $ ssh -R 6000:localhost:1080 salomon.it4i.cz ``` -Now, configure the applications proxy settings to **localhost:6000**. Use port forwarding to access the [proxy server from compute nodes](outgoing-connections/#port-forwarding-from-compute-nodes) as well . +Now, configure the applications proxy settings to **localhost:6000**. Use port forwarding to access the [proxy server from compute nodes](#port-forwarding-from-compute-nodes) as well . diff --git a/docs.it4i/salomon/storage/storage.md b/docs.it4i/salomon/storage.md similarity index 73% rename from docs.it4i/salomon/storage/storage.md rename to docs.it4i/salomon/storage.md index 00de22d4a31a7eb9cb0989b59b295de516def4b0..69b7e2729f4067bb88b13ec8e013877f2d3d7d46 100644 --- a/docs.it4i/salomon/storage/storage.md +++ b/docs.it4i/salomon/storage.md @@ -3,7 +3,7 @@ Storage Introduction ------------ -There are two main shared file systems on Salomon cluster, the [HOME](storage/storage/#home)and [SCRATCH](storage/#shared-filesystems). +There are two main shared file systems on Salomon cluster, the [HOME](#home)and [SCRATCH](#shared-filesystems). All login and compute nodes may access same data on shared filesystems. Compute nodes are also equipped with local (non-shared) scratch, ramdisk and tmp filesystems. @@ -11,26 +11,26 @@ Policy (in a nutshell) ---------------------- !!! Note "Note" Use [ for your most valuable data and programs. - Use [WORK](storage/#work) for your large project files. - Use [TEMP](storage/#temp) for large scratch data. + Use [WORK](#work) for your large project files. + Use [TEMP](#temp) for large scratch data. - Do not use for [archiving](storage/#archiving)! + Do not use for [archiving](#archiving)! Archiving ------------- -Please don't use shared filesystems as a backup for large amount of data or long-term archiving mean. The academic staff and students of research institutions in the Czech Republic can use [CESNET storage service](../../anselm-cluster-documentation/storage/cesnet-data-storage/), which is available via SSHFS. +Please don't use shared filesystems as a backup for large amount of data or long-term archiving mean. The academic staff and students of research institutions in the Czech Republic can use [CESNET storage service](#cesnet-data-storage), which is available via SSHFS. Shared Filesystems ---------------------- -Salomon computer provides two main shared filesystems, the [ HOME filesystem](storage/#home-filesystem) and the [SCRATCH filesystem](storage/#scratch-filesystem). The SCRATCH filesystem is partitioned to [WORK and TEMP workspaces](storage/#shared-workspaces). The HOME filesystem is realized as a tiered NFS disk storage. The SCRATCH filesystem is realized as a parallel Lustre filesystem. Both shared file systems are accessible via the Infiniband network. Extended ACLs are provided on both HOME/SCRATCH filesystems for the purpose of sharing data with other users using fine-grained control. +Salomon computer provides two main shared filesystems, the [ HOME filesystem](#home-filesystem) and the [SCRATCH filesystem](#scratch-filesystem). The SCRATCH filesystem is partitioned to [WORK and TEMP workspaces](#shared-workspaces). The HOME filesystem is realized as a tiered NFS disk storage. The SCRATCH filesystem is realized as a parallel Lustre filesystem. Both shared file systems are accessible via the Infiniband network. Extended ACLs are provided on both HOME/SCRATCH filesystems for the purpose of sharing data with other users using fine-grained control. ###HOME filesystem -The HOME filesystem is realized as a Tiered filesystem, exported via NFS. The first tier has capacity 100TB, second tier has capacity 400TB. The filesystem is available on all login and computational nodes. The Home filesystem hosts the [HOME workspace](storage/#home). +The HOME filesystem is realized as a Tiered filesystem, exported via NFS. The first tier has capacity 100TB, second tier has capacity 400TB. The filesystem is available on all login and computational nodes. The Home filesystem hosts the [HOME workspace](#home). ###SCRATCH filesystem -The architecture of Lustre on Salomon is composed of two metadata servers (MDS) and six data/object storage servers (OSS). Accessible capacity is 1.69 PB, shared among all users. The SCRATCH filesystem hosts the [WORK and TEMP workspaces](storage/#shared-workspaces). +The architecture of Lustre on Salomon is composed of two metadata servers (MDS) and six data/object storage servers (OSS). Accessible capacity is 1.69 PB, shared among all users. The SCRATCH filesystem hosts the [WORK and TEMP workspaces](#shared-workspaces). Configuration of the SCRATCH Lustre storage @@ -321,3 +321,104 @@ Summary |/scratch/work|large project files|Lustre|1.69 PB|30 GB/s|Quota|Compute and login nodes|none| |/scratch/temp|job temporary data|Lustre|1.69 PB|30 GB/s|Quota 100TB|Compute and login nodes|files older 90 days removed| |/ramdisk|job temporary data, node local|local|120GB|90 GB/s|none|Compute nodes|purged after job ends| + +CESNET Data Storage +------------ +Do not use shared filesystems at IT4Innovations as a backup for large amount of data or long-term archiving purposes. + +!!! Note "Note" +../../img/The IT4Innovations does not provide storage capacity for data archiving. Academic staff and students of research institutions in the Czech Republic can use [CESNET Storage service](https://du.cesnet.cz/). + +The CESNET Storage service can be used for research purposes, mainly by academic staff and students of research institutions in the Czech Republic. + +User of data storage CESNET (DU) association can become organizations or an individual person who is either in the current employment relationship (employees) or the current study relationship (students) to a legal entity (organization) that meets the “Principles for access to CESNET Large infrastructure (Access Policy)”. + +User may only use data storage CESNET for data transfer and storage which are associated with activities in science, research, development, the spread of education, culture and prosperity. In detail see “Acceptable Use Policy CESNET Large Infrastructure (Acceptable Use Policy, AUP)”. + +The service is documented at <https://du.cesnet.cz/wiki/doku.php/en/start>. For special requirements please contact directly CESNET Storage Department via e-mail [du-support(at)cesnet.cz](mailto:du-support@cesnet.cz). + +The procedure to obtain the CESNET access is quick and trouble-free. + +(source [https://du.cesnet.cz/](https://du.cesnet.cz/wiki/doku.php/en/start "CESNET Data Storage")) + +CESNET storage access +--------------------- + +### Understanding Cesnet storage + +!!! Note "Note" + It is very important to understand the Cesnet storage before uploading data. Please read <https://du.cesnet.cz/en/navody/home-migrace-plzen/start> first. + +Once registered for CESNET Storage, you may [access the storage](https://du.cesnet.cz/en/navody/faq/start) in number of ways. We recommend the SSHFS and RSYNC methods. + +### SSHFS Access + +!!! Note "Note" + SSHFS: The storage will be mounted like a local hard drive + +The SSHFS provides a very convenient way to access the CESNET Storage. The storage will be mounted onto a local directory, exposing the vast CESNET Storage as if it was a local removable harddrive. Files can be than copied in and out in a usual fashion. + +First, create the mountpoint + +```bash + $ mkdir cesnet +``` + +Mount the storage. Note that you can choose among the ssh.du1.cesnet.cz (Plzen), ssh.du2.cesnet.cz (Jihlava), ssh.du3.cesnet.cz (Brno) Mount tier1_home **(only 5120M !)**: + +```bash + $ sshfs username@ssh.du1.cesnet.cz:. cesnet/ +``` + +For easy future access from Anselm, install your public key + +```bash + $ cp .ssh/id_rsa.pub cesnet/.ssh/authorized_keys +``` + +Mount tier1_cache_tape for the Storage VO: + +```bash + $ sshfs username@ssh.du1.cesnet.cz:/cache_tape/VO_storage/home/username cesnet/ +``` + +View the archive, copy the files and directories in and out + +```bash + $ ls cesnet/ + $ cp -a mydir cesnet/. + $ cp cesnet/myfile . +``` + +Once done, please remember to unmount the storage + +```bash + $ fusermount -u cesnet +``` + +### Rsync access + +!!! Note "Note" + Rsync provides delta transfer for best performance, can resume interrupted transfers + +Rsync is a fast and extraordinarily versatile file copying tool. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use. + +Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated. + +More about Rsync at <https://du.cesnet.cz/en/navody/rsync/start#pro_bezne_uzivatele> + +Transfer large files to/from Cesnet storage, assuming membership in the Storage VO + +```bash + $ rsync --progress datafile username@ssh.du1.cesnet.cz:VO_storage-cache_tape/. + $ rsync --progress username@ssh.du1.cesnet.cz:VO_storage-cache_tape/datafile . +``` + +Transfer large directories to/from Cesnet storage, assuming membership in the Storage VO + +```bash + $ rsync --progress -av datafolder username@ssh.du1.cesnet.cz:VO_storage-cache_tape/. + $ rsync --progress -av username@ssh.du1.cesnet.cz:VO_storage-cache_tape/datafolder . +``` + +Transfer rates of about 28MB/s can be expected. diff --git a/docs.it4i/salomon/storage/cesnet-data-storage.md b/docs.it4i/salomon/storage/cesnet-data-storage.md deleted file mode 100644 index ab5cd020c1520f2d208c50e8836838d4bd3e3e65..0000000000000000000000000000000000000000 --- a/docs.it4i/salomon/storage/cesnet-data-storage.md +++ /dev/null @@ -1,103 +0,0 @@ -CESNET Data Storage -=================== - -Introduction ------------- -Do not use shared filesystems at IT4Innovations as a backup for large amount of data or long-term archiving purposes. - -!!! Note "Note" -../../img/The IT4Innovations does not provide storage capacity for data archiving. Academic staff and students of research institutions in the Czech Republic can use [CESNET Storage service](https://du.cesnet.cz/). - -The CESNET Storage service can be used for research purposes, mainly by academic staff and students of research institutions in the Czech Republic. - -User of data storage CESNET (DU) association can become organizations or an individual person who is either in the current employment relationship (employees) or the current study relationship (students) to a legal entity (organization) that meets the “Principles for access to CESNET Large infrastructure (Access Policy)”. - -User may only use data storage CESNET for data transfer and storage which are associated with activities in science, research, development, the spread of education, culture and prosperity. In detail see “Acceptable Use Policy CESNET Large Infrastructure (Acceptable Use Policy, AUP)”. - -The service is documented at <https://du.cesnet.cz/wiki/doku.php/en/start>. For special requirements please contact directly CESNET Storage Department via e-mail [du-support(at)cesnet.cz](mailto:du-support@cesnet.cz). - -The procedure to obtain the CESNET access is quick and trouble-free. - -(source [https://du.cesnet.cz/](https://du.cesnet.cz/wiki/doku.php/en/start "CESNET Data Storage")) - -CESNET storage access ---------------------- - -### Understanding Cesnet storage - -!!! Note "Note" - It is very important to understand the Cesnet storage before uploading data. Please read <https://du.cesnet.cz/en/navody/home-migrace-plzen/start> first. - -Once registered for CESNET Storage, you may [access the storage](https://du.cesnet.cz/en/navody/faq/start) in number of ways. We recommend the SSHFS and RSYNC methods. - -### SSHFS Access - -!!! Note "Note" - SSHFS: The storage will be mounted like a local hard drive - -The SSHFS provides a very convenient way to access the CESNET Storage. The storage will be mounted onto a local directory, exposing the vast CESNET Storage as if it was a local removable harddrive. Files can be than copied in and out in a usual fashion. - -First, create the mountpoint - -```bash - $ mkdir cesnet -``` - -Mount the storage. Note that you can choose among the ssh.du1.cesnet.cz (Plzen), ssh.du2.cesnet.cz (Jihlava), ssh.du3.cesnet.cz (Brno) Mount tier1_home **(only 5120M !)**: - -```bash - $ sshfs username@ssh.du1.cesnet.cz:. cesnet/ -``` - -For easy future access from Anselm, install your public key - -```bash - $ cp .ssh/id_rsa.pub cesnet/.ssh/authorized_keys -``` - -Mount tier1_cache_tape for the Storage VO: - -```bash - $ sshfs username@ssh.du1.cesnet.cz:/cache_tape/VO_storage/home/username cesnet/ -``` - -View the archive, copy the files and directories in and out - -```bash - $ ls cesnet/ - $ cp -a mydir cesnet/. - $ cp cesnet/myfile . -``` - -Once done, please remember to unmount the storage - -```bash - $ fusermount -u cesnet -``` - -### Rsync access - -!!! Note "Note" - Rsync provides delta transfer for best performance, can resume interrupted transfers - -Rsync is a fast and extraordinarily versatile file copying tool. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use. - -Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated. - -More about Rsync at <https://du.cesnet.cz/en/navody/rsync/start#pro_bezne_uzivatele> - -Transfer large files to/from Cesnet storage, assuming membership in the Storage VO - -```bash - $ rsync --progress datafile username@ssh.du1.cesnet.cz:VO_storage-cache_tape/. - $ rsync --progress username@ssh.du1.cesnet.cz:VO_storage-cache_tape/datafile . -``` - -Transfer large directories to/from Cesnet storage, assuming membership in the Storage VO - -```bash - $ rsync --progress -av datafolder username@ssh.du1.cesnet.cz:VO_storage-cache_tape/. - $ rsync --progress -av username@ssh.du1.cesnet.cz:VO_storage-cache_tape/datafolder . -``` - -Transfer rates of about 28MB/s can be expected. diff --git a/get_modules.sh b/get_modules.sh index 5122453eca83267398fe4cd67f2fc901aa1d2a03..a92b45bc34b5ba9f65380f5e3027267473eb37e6 100755 --- a/get_modules.sh +++ b/get_modules.sh @@ -1,3 +1,4 @@ #!/bin/bash curl -s https://code.it4i.cz/hrb33/modules-anselm/raw/master/anselm.md -o docs.it4i/modules-anselm.md curl -s https://code.it4i.cz/hrb33/modules-salomon/raw/master/salomon.md -o docs.it4i/modules-salomon.md +curl -s https://code.it4i.cz/hrb33/modules-salomon/raw/master/salomon-uv.md -o docs.it4i/modules-salomon-uv.md diff --git a/mkdocs.yml b/mkdocs.yml index 79166b4c7106300be3c031f10550707008cb8b9d..195df0e238a4501352d2fefd1ac2773bb792cdb4 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -15,14 +15,12 @@ pages: - Accessing the Clusters: get-started-with-it4innovations/accessing-the-clusters/introduction.md # - VPN-Connection-Fail-in-Win-8.1: get-started-with-it4innovations/accessing-the-clusters/shell-access-and-data-transfer/vpn-connection-fail-in-win-8.1.md - SSH keys: get-started-with-it4innovations/accessing-the-clusters/shell-access-and-data-transfer/ssh-keys.md - - Pageant SSH agent: get-started-with-it4innovations/accessing-the-clusters/shell-access-and-data-transfer/pageant.md - - PuTTY key generator: get-started-with-it4innovations/accessing-the-clusters/shell-access-and-data-transfer/puttygen.md - PuTTY: get-started-with-it4innovations/accessing-the-clusters/shell-access-and-data-transfer/putty.md - - GUI Access: - - Introduction: get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/graphical-user-interface.md + - PuTTY Pageant SSH agent: get-started-with-it4innovations/accessing-the-clusters/shell-access-and-data-transfer/pageant.md + - PuTTY key generator: get-started-with-it4innovations/accessing-the-clusters/shell-access-and-data-transfer/puttygen.md - X Window System: get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-system.md + - X Window System using Cygwin: get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/cygwin-and-x11-forwarding.md - VNC: get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/vnc.md - - Cygwin and x11 Forwarding: get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/cygwin-and-x11-forwarding.md - Anselm Cluster: - Introduction: anselm-cluster-documentation/introduction.md - Hardware Overview: anselm-cluster-documentation/hardware-overview.md @@ -33,15 +31,12 @@ pages: - Remote visualization: anselm-cluster-documentation/remote-visualization.md - Network: anselm-cluster-documentation/network.md - PRACE User Support: anselm-cluster-documentation/prace.md - - Anselm Cluster Resource Allocation and Job Execution: - - Introduction: anselm-cluster-documentation/resource-allocation-and-job-execution/introduction.md - - Job Priority: anselm-cluster-documentation/resource-allocation-and-job-execution/job-priority.md - - Job Submission and Execution: anselm-cluster-documentation/resource-allocation-and-job-execution/job-submission-and-execution.md - - Capacity computing: anselm-cluster-documentation/resource-allocation-and-job-execution/capacity-computing.md - - Resource allocation policy: anselm-cluster-documentation/resource-allocation-and-job-execution/resources-allocation-policy.md - - Anselm Cluster Storage: - - Cesnet Data Storage: anselm-cluster-documentation/storage/cesnet-data-storage.md - - Storage: anselm-cluster-documentation/storage/storage.md + - Resource Allocation and Job Execution : anselm-cluster-documentation/resource-allocation-and-job-execution.md + - Resource allocation policy: anselm-cluster-documentation/resources-allocation-policy.md + - Job Priority: anselm-cluster-documentation/job-priority.md + - Job Submission and Execution: anselm-cluster-documentation/job-submission-and-execution.md + - Capacity computing: anselm-cluster-documentation/capacity-computing.md + - Storage: anselm-cluster-documentation/storage.md - Anselm Cluster Software: anselm-cluster-documentation/software/index.md - ' Anselm Cluster Software': - Modules: modules-anselm.md @@ -118,22 +113,19 @@ pages: - VPN Access: salomon/vpn-access.md - Compute Nodes: salomon/compute-nodes.md - List of modules: salomon/list_of_modules.md - - Salomon Cluster Network: - - Introduction: salomon/network/network.md - - IB single-plane topology: salomon/network/ib-single-plane-topology.md - - 7D Enhanced Hypercube: salomon/network/7d-enhanced-hypercube.md - - Salomon Cluster Resource Allocation and Job Execution: - - Introduction: salomon/resource-allocation-and-job-execution/introduction.md - - Job scheduling: salomon/resource-allocation-and-job-execution/job-priority.md - - Job submission and execution: salomon/resource-allocation-and-job-execution/job-submission-and-execution.md - - Capacity computing: salomon/resource-allocation-and-job-execution/capacity-computing.md - - Resources Allocation Policy: salomon/resource-allocation-and-job-execution/resources-allocation-policy.md - - Salomon Cluster Storage: - - CESNET Data Storage: salomon/storage/cesnet-data-storage.md - - Storage: salomon/storage/storage.md + - Salomon Cluster Network: salomon/network.md + - IB single-plane topology: salomon/ib-single-plane-topology.md + - 7D Enhanced Hypercube: salomon/7d-enhanced-hypercube.md + - Resource Allocation and Job Execution : salomon/resource-allocation-and-job-execution.md + - Resources Allocation Policy: salomon/resources-allocation-policy.md + - Job scheduling: salomon/job-priority.md + - Job submission and execution: salomon/job-submission-and-execution.md + - Capacity computing: salomon/capacity-computing.md + - Storage: salomon/storage.md - Salomon Cluster Software: salomon/software/index.md - ' Salomon Cluster Software': - Modules: modules-salomon.md + - UV Modules: modules-salomon-uv.md - Compilers: salomon/software/compilers.md - Intel Xeon Phi: salomon/software/intel-xeon-phi.md - Java: salomon/software/java.md