Skip to content
Snippets Groups Projects
Commit b4474c1c authored by Lukáš Krupčík's avatar Lukáš Krupčík
Browse files

remove tab

parent 35f4dca9
No related branches found
No related tags found
5 merge requests!368Update prace.md to document the change from qprace to qprod as the default...,!367Update prace.md to document the change from qprace to qprod as the default...,!366Update prace.md to document the change from qprace to qprod as the default...,!323extended-acls-storage-section,!70Note clean-up
Pipeline #
Hide whitespace changes
Inline Side-by-side
Showing
with 89 additions and 89 deletions
docs.it4i/anselm-cluster-documentation/software/mpi/Running_OpenMPI.md
+ 5
− 5
View file @ b4474c1c
......@@ -7,7 +7,7 @@ The OpenMPI programs may be executed only via the PBS Workload manager, by enter
### Basic Usage
!!! Note
Use the mpiexec to run the OpenMPI code.
Use the mpiexec to run the OpenMPI code.
Example:
......@@ -28,7 +28,7 @@ Example:
```
!!! Note
Please be aware, that in this example, the directive **-pernode** is used to run only **one task per node**, which is normally an unwanted behaviour (unless you want to run hybrid code with just one MPI and 16 OpenMP tasks per node). In normal MPI programs **omit the -pernode directive** to run up to 16 MPI tasks per each node.
Please be aware, that in this example, the directive **-pernode** is used to run only **one task per node**, which is normally an unwanted behaviour (unless you want to run hybrid code with just one MPI and 16 OpenMP tasks per node). In normal MPI programs **omit the -pernode directive** to run up to 16 MPI tasks per each node.
In this example, we allocate 4 nodes via the express queue interactively. We set up the openmpi environment and interactively run the helloworld_mpi.x program. Note that the executable helloworld_mpi.x must be available within the
same path on all nodes. This is automatically fulfilled on the /home and /scratch filesystem.
......@@ -49,7 +49,7 @@ You need to preload the executable, if running on the local scratch /lscratch fi
In this example, we assume the executable helloworld_mpi.x is present on compute node cn17 on local scratch. We call the mpiexec whith the **--preload-binary** argument (valid for openmpi). The mpiexec will copy the executable from cn17 to the /lscratch/15210.srv11 directory on cn108, cn109 and cn110 and execute the program.
!!! Note
MPI process mapping may be controlled by PBS parameters.
MPI process mapping may be controlled by PBS parameters.
The mpiprocs and ompthreads parameters allow for selection of number of running MPI processes per node as well as number of OpenMP threads per MPI process.
......@@ -98,7 +98,7 @@ In this example, we demonstrate recommended way to run an MPI application, using
### OpenMP Thread Affinity
!!! Note
Important! Bind every OpenMP thread to a core!
Important! Bind every OpenMP thread to a core!
In the previous two examples with one or two MPI processes per node, the operating system might still migrate OpenMP threads between cores. You might want to avoid this by setting these environment variable for GCC OpenMP:
......@@ -153,7 +153,7 @@ In this example, we see that ranks have been mapped on nodes according to the or
Exact control of MPI process placement and resource binding is provided by specifying a rankfile
!!! Note
Appropriate binding may boost performance of your application.
Appropriate binding may boost performance of your application.
Example rankfile
......
docs.it4i/anselm-cluster-documentation/software/mpi/mpi.md
+ 4
− 4
View file @ b4474c1c
......@@ -61,7 +61,7 @@ In this example, the openmpi 1.6.5 using intel compilers is activated
## Compiling MPI Programs
!!! Note
After setting up your MPI environment, compile your program using one of the mpi wrappers
After setting up your MPI environment, compile your program using one of the mpi wrappers
```bash
$ mpicc -v
......@@ -108,7 +108,7 @@ Compile the above example with
## Running MPI Programs
!!! Note
The MPI program executable must be compatible with the loaded MPI module.
The MPI program executable must be compatible with the loaded MPI module.
Always compile and execute using the very same MPI module.
It is strongly discouraged to mix mpi implementations. Linking an application with one MPI implementation and running mpirun/mpiexec form other implementation may result in unexpected errors.
......@@ -120,7 +120,7 @@ The MPI program executable must be available within the same path on all nodes.
Optimal way to run an MPI program depends on its memory requirements, memory access pattern and communication pattern.
!!! Note
Consider these ways to run an MPI program:
Consider these ways to run an MPI program:
1. One MPI process per node, 16 threads per process
2. Two MPI processes per node, 8 threads per process
......@@ -131,7 +131,7 @@ Optimal way to run an MPI program depends on its memory requirements, memory acc
**Two MPI** processes per node, using 8 threads each, bound to processor socket is most useful for memory bandwidth bound applications such as BLAS1 or FFT, with scalable memory demand. However, note that the two processes will share access to the network interface. The 8 threads and socket binding should ensure maximum memory access bandwidth and minimize communication, migration and NUMA effect overheads.
!!! Note
Important! Bind every OpenMP thread to a core!
Important! Bind every OpenMP thread to a core!
In the previous two cases with one or two MPI processes per node, the operating system might still migrate OpenMP threads between cores. You want to avoid this by setting the KMP_AFFINITY or GOMP_CPU_AFFINITY environment variables.
......
docs.it4i/anselm-cluster-documentation/software/mpi/running-mpich2.md
+ 3
− 3
View file @ b4474c1c
......@@ -7,7 +7,7 @@ The MPICH2 programs use mpd daemon or ssh connection to spawn processes, no PBS
### Basic Usage
!!! Note
Use the mpirun to execute the MPICH2 code.
Use the mpirun to execute the MPICH2 code.
Example:
......@@ -44,7 +44,7 @@ You need to preload the executable, if running on the local scratch /lscratch fi
In this example, we assume the executable helloworld_mpi.x is present on shared home directory. We run the cp command via mpirun, copying the executable from shared home to local scratch . Second mpirun will execute the binary in the /lscratch/15210.srv11 directory on nodes cn17, cn108, cn109 and cn110, one process per node.
!!! Note
MPI process mapping may be controlled by PBS parameters.
MPI process mapping may be controlled by PBS parameters.
The mpiprocs and ompthreads parameters allow for selection of number of running MPI processes per node as well as number of OpenMP threads per MPI process.
......@@ -93,7 +93,7 @@ In this example, we demonstrate recommended way to run an MPI application, using
### OpenMP Thread Affinity
!!! Note
Important! Bind every OpenMP thread to a core!
Important! Bind every OpenMP thread to a core!
In the previous two examples with one or two MPI processes per node, the operating system might still migrate OpenMP threads between cores. You might want to avoid this by setting these environment variable for GCC OpenMP:
......
docs.it4i/anselm-cluster-documentation/software/numerical-languages/matlab.md
+ 2
− 2
View file @ b4474c1c
......@@ -42,7 +42,7 @@ plots, images, etc... will be still available.
## Running Parallel Matlab Using Distributed Computing Toolbox / Engine
!!! Note
Distributed toolbox is available only for the EDU variant
Distributed toolbox is available only for the EDU variant
The MPIEXEC mode available in previous versions is no longer available in MATLAB 2015. Also, the programming interface has changed. Refer to [Release Notes](http://www.mathworks.com/help/distcomp/release-notes.html#buanp9e-1).
......@@ -65,7 +65,7 @@ Or in the GUI, go to tab HOME -> Parallel -> Manage Cluster Profiles..., click I
With the new mode, MATLAB itself launches the workers via PBS, so you can either use interactive mode or a batch mode on one node, but the actual parallel processing will be done in a separate job started by MATLAB itself. Alternatively, you can use "local" mode to run parallel code on just a single node.
!!! Note
The profile is confusingly named Salomon, but you can use it also on Anselm.
The profile is confusingly named Salomon, but you can use it also on Anselm.
### Parallel Matlab Interactive Session
......
docs.it4i/anselm-cluster-documentation/software/numerical-languages/matlab_1314.md
+ 1
− 1
View file @ b4474c1c
......@@ -3,7 +3,7 @@
## Introduction
!!! Note
This document relates to the old versions R2013 and R2014. For MATLAB 2015, please use [this documentation instead](matlab/).
This document relates to the old versions R2013 and R2014. For MATLAB 2015, please use [this documentation instead](matlab/).
Matlab is available in the latest stable version. There are always two variants of the release:
......
docs.it4i/anselm-cluster-documentation/software/numerical-languages/octave.md
+ 1
− 1
View file @ b4474c1c
......@@ -97,7 +97,7 @@ Octave is linked with parallel Intel MKL, so it best suited for batch processing
variable.
!!! Note
Calculations that do not employ parallelism (either by using parallel MKL e.g. via matrix operations, fork() function, [parallel package](http://octave.sourceforge.net/parallel/) or other mechanism) will actually run slower than on host CPU.
Calculations that do not employ parallelism (either by using parallel MKL e.g. via matrix operations, fork() function, [parallel package](http://octave.sourceforge.net/parallel/) or other mechanism) will actually run slower than on host CPU.
To use Octave on a node with Xeon Phi:
......
docs.it4i/anselm-cluster-documentation/software/numerical-languages/r.md
+ 3
− 3
View file @ b4474c1c
......@@ -96,7 +96,7 @@ Download the package [parallell](package-parallel-vignette.pdf) vignette.
The forking is the most simple to use. Forking family of functions provide parallelized, drop in replacement for the serial apply() family of functions.
!!! Note
Forking via package parallel provides functionality similar to OpenMP construct
Forking via package parallel provides functionality similar to OpenMP construct
omp parallel for
......@@ -147,7 +147,7 @@ Every evaluation of the integrad function runs in parallel on different process.
## Package Rmpi
!!! Note
package Rmpi provides an interface (wrapper) to MPI APIs.
package Rmpi provides an interface (wrapper) to MPI APIs.
It also provides interactive R slave environment. On Anselm, Rmpi provides interface to the [OpenMPI](../mpi-1/Running_OpenMPI/).
......@@ -297,7 +297,7 @@ Execute the example as:
mpi.apply is a specific way of executing Dynamic Rmpi programs.
!!! Note
mpi.apply() family of functions provide MPI parallelized, drop in replacement for the serial apply() family of functions.
mpi.apply() family of functions provide MPI parallelized, drop in replacement for the serial apply() family of functions.
Execution is identical to other dynamic Rmpi programs.
......
docs.it4i/anselm-cluster-documentation/software/numerical-libraries/hdf5.md
+ 1
− 1
View file @ b4474c1c
......@@ -23,7 +23,7 @@ Versions **1.8.11** and **1.8.13** of HDF5 library are available on Anselm, comp
The module sets up environment variables, required for linking and running HDF5 enabled applications. Make sure that the choice of HDF5 module is consistent with your choice of MPI library. Mixing MPI of different implementations may have unpredictable results.
!!! Note
Be aware, that GCC version of **HDF5 1.8.11** has serious performance issues, since it's compiled with -O0 optimization flag. This version is provided only for testing of code compiled only by GCC and IS NOT recommended for production computations. For more information, please see: <http://www.hdfgroup.org/ftp/HDF5/prev-releases/ReleaseFiles/release5-1811>
Be aware, that GCC version of **HDF5 1.8.11** has serious performance issues, since it's compiled with -O0 optimization flag. This version is provided only for testing of code compiled only by GCC and IS NOT recommended for production computations. For more information, please see: <http://www.hdfgroup.org/ftp/HDF5/prev-releases/ReleaseFiles/release5-1811>
All GCC versions of **HDF5 1.8.13** are not affected by the bug, are compiled with -O3 optimizations and are recommended for production computations.
......
docs.it4i/anselm-cluster-documentation/software/numerical-libraries/magma-for-intel-xeon-phi.md
+ 8
− 8
View file @ b4474c1c
......@@ -13,10 +13,10 @@ To be able to compile and link code with MAGMA library user has to load followin
To make compilation more user friendly module also sets these two environment variables:
!!! Note
MAGMA_INC - contains paths to the MAGMA header files (to be used for compilation step)
MAGMA_INC - contains paths to the MAGMA header files (to be used for compilation step)
!!! Note
MAGMA_LIBS - contains paths to MAGMA libraries (to be used for linking step).
MAGMA_LIBS - contains paths to MAGMA libraries (to be used for linking step).
Compilation example:
......@@ -31,16 +31,16 @@ Compilation example:
MAGMA implementation for Intel MIC requires a MAGMA server running on accelerator prior to executing the user application. The server can be started and stopped using following scripts:
!!! Note
To start MAGMA server use:
**$MAGMAROOT/start_magma_server**
To start MAGMA server use:
**$MAGMAROOT/start_magma_server**
!!! Note
To stop the server use:
**$MAGMAROOT/stop_magma_server**
To stop the server use:
**$MAGMAROOT/stop_magma_server**
!!! Note
For deeper understanding how the MAGMA server is started, see the following script:
**$MAGMAROOT/launch_anselm_from_mic.sh**
For deeper understanding how the MAGMA server is started, see the following script:
**$MAGMAROOT/launch_anselm_from_mic.sh**
To test if the MAGMA server runs properly we can run one of examples that are part of the MAGMA installation:
......
docs.it4i/anselm-cluster-documentation/software/openfoam.md
+ 3
− 3
View file @ b4474c1c
......@@ -58,7 +58,7 @@ To create OpenFOAM environment on ANSELM give the commands:
```
!!! Note
Please load correct module with your requirements “compiler - GCC/ICC, precision - DP/SP”.
Please load correct module with your requirements “compiler - GCC/ICC, precision - DP/SP”.
Create a project directory within the $HOME/OpenFOAM directory named \<USER\>-\<OFversion\> and create a directory named run within it, e.g. by typing:
......@@ -121,7 +121,7 @@ Run the second case for example external incompressible turbulent flow - case -
First we must run serial application bockMesh and decomposePar for preparation of parallel computation.
!!! Note
Create a Bash scrip test.sh:
Create a Bash scrip test.sh:
```bash
#!/bin/bash
......@@ -146,7 +146,7 @@ Job submission
This job create simple block mesh and domain decomposition. Check your decomposition, and submit parallel computation:
!!! Note
Create a PBS script testParallel.pbs:
Create a PBS script testParallel.pbs:
```bash
#!/bin/bash
......
docs.it4i/anselm-cluster-documentation/storage.md
+ 15
− 15
View file @ b4474c1c
......@@ -27,7 +27,7 @@ There is default stripe configuration for Anselm Lustre filesystems. However, us
3. stripe_offset The index of the OST where the first stripe is to be placed; default is -1 which results in random selection; using a non-default value is NOT recommended.
!!! Note
Setting stripe size and stripe count correctly for your needs may significantly impact the I/O performance you experience.
Setting stripe size and stripe count correctly for your needs may significantly impact the I/O performance you experience.
Use the lfs getstripe for getting the stripe parameters. Use the lfs setstripe command for setting the stripe parameters to get optimal I/O performance The correct stripe setting depends on your needs and file access patterns.
......@@ -61,14 +61,14 @@ $ man lfs
### Hints on Lustre Stripping
!!! Note
Increase the stripe_count for parallel I/O to the same file.
Increase the stripe_count for parallel I/O to the same file.
When multiple processes are writing blocks of data to the same file in parallel, the I/O performance for large files will improve when the stripe_count is set to a larger value. The stripe count sets the number of OSTs the file will be written to. By default, the stripe count is set to 1. While this default setting provides for efficient access of metadata (for example to support the ls -l command), large files should use stripe counts of greater than 1. This will increase the aggregate I/O bandwidth by using multiple OSTs in parallel instead of just one. A rule of thumb is to use a stripe count approximately equal to the number of gigabytes in the file.
Another good practice is to make the stripe count be an integral factor of the number of processes performing the write in parallel, so that you achieve load balance among the OSTs. For example, set the stripe count to 16 instead of 15 when you have 64 processes performing the writes.
!!! Note
Using a large stripe size can improve performance when accessing very large files
Using a large stripe size can improve performance when accessing very large files
Large stripe size allows each client to have exclusive access to its own part of a file. However, it can be counterproductive in some cases if it does not match your I/O pattern. The choice of stripe size has no effect on a single-stripe file.
......@@ -103,7 +103,7 @@ The architecture of Lustre on Anselm is composed of two metadata servers (MDS)
The HOME filesystem is mounted in directory /home. Users home directories /home/username reside on this filesystem. Accessible capacity is 320TB, shared among all users. Individual users are restricted by filesystem usage quotas, set to 250GB per user. If 250GB should prove as insufficient for particular user, please contact [support](https://support.it4i.cz/rt), the quota may be lifted upon request.
!!! Note
The HOME filesystem is intended for preparation, evaluation, processing and storage of data generated by active Projects.
The HOME filesystem is intended for preparation, evaluation, processing and storage of data generated by active Projects.
The HOME filesystem should not be used to archive data of past Projects or other unrelated data.
......@@ -115,7 +115,7 @@ The HOME filesystem is realized as Lustre parallel filesystem and is available o
Default stripe size is 1MB, stripe count is 1. There are 22 OSTs dedicated for the HOME filesystem.
!!! Note
Setting stripe size and stripe count correctly for your needs may significantly impact the I/O performance you experience.
Setting stripe size and stripe count correctly for your needs may significantly impact the I/O performance you experience.
| HOME filesystem | |
| -------------------- | ------ |
......@@ -132,7 +132,7 @@ Default stripe size is 1MB, stripe count is 1. There are 22 OSTs dedicated for t
The SCRATCH filesystem is mounted in directory /scratch. Users may freely create subdirectories and files on the filesystem. Accessible capacity is 146TB, shared among all users. Individual users are restricted by filesystem usage quotas, set to 100TB per user. The purpose of this quota is to prevent runaway programs from filling the entire filesystem and deny service to other users. If 100TB should prove as insufficient for particular user, please contact [support](https://support.it4i.cz/rt), the quota may be lifted upon request.
!!! Note
The Scratch filesystem is intended for temporary scratch data generated during the calculation as well as for high performance access to input and output files. All I/O intensive jobs must use the SCRATCH filesystem as their working directory.
The Scratch filesystem is intended for temporary scratch data generated during the calculation as well as for high performance access to input and output files. All I/O intensive jobs must use the SCRATCH filesystem as their working directory.
>Users are advised to save the necessary data from the SCRATCH filesystem to HOME filesystem after the calculations and clean up the scratch files.
......@@ -141,7 +141,7 @@ The SCRATCH filesystem is mounted in directory /scratch. Users may freely create
The SCRATCH filesystem is realized as Lustre parallel filesystem and is available from all login and computational nodes. Default stripe size is 1MB, stripe count is 1. There are 10 OSTs dedicated for the SCRATCH filesystem.
!!! Note
Setting stripe size and stripe count correctly for your needs may significantly impact the I/O performance you experience.
Setting stripe size and stripe count correctly for your needs may significantly impact the I/O performance you experience.
| SCRATCH filesystem | |
| -------------------- | -------- |
......@@ -261,7 +261,7 @@ Default ACL mechanism can be used to replace setuid/setgid permissions on direct
### Local Scratch
!!! Note
Every computational node is equipped with 330GB local scratch disk.
Every computational node is equipped with 330GB local scratch disk.
Use local scratch in case you need to access large amount of small files during your calculation.
......@@ -270,7 +270,7 @@ The local scratch disk is mounted as /lscratch and is accessible to user at /lsc
The local scratch filesystem is intended for temporary scratch data generated during the calculation as well as for high performance access to input and output files. All I/O intensive jobs that access large number of small files within the calculation must use the local scratch filesystem as their working directory. This is required for performance reasons, as frequent access to number of small files may overload the metadata servers (MDS) of the Lustre filesystem.
!!! Note
The local scratch directory /lscratch/$PBS_JOBID will be deleted immediately after the calculation end. Users should take care to save the output data from within the jobscript.
The local scratch directory /lscratch/$PBS_JOBID will be deleted immediately after the calculation end. Users should take care to save the output data from within the jobscript.
| local SCRATCH filesystem | |
| ------------------------ | -------------------- |
......@@ -285,14 +285,14 @@ The local scratch filesystem is intended for temporary scratch data generated d
Every computational node is equipped with filesystem realized in memory, so called RAM disk.
!!! Note
Use RAM disk in case you need really fast access to your data of limited size during your calculation. Be very careful, use of RAM disk filesystem is at the expense of operational memory.
Use RAM disk in case you need really fast access to your data of limited size during your calculation. Be very careful, use of RAM disk filesystem is at the expense of operational memory.
The local RAM disk is mounted as /ramdisk and is accessible to user at /ramdisk/$PBS_JOBID directory.
The local RAM disk filesystem is intended for temporary scratch data generated during the calculation as well as for high performance access to input and output files. Size of RAM disk filesystem is limited. Be very careful, use of RAM disk filesystem is at the expense of operational memory. It is not recommended to allocate large amount of memory and use large amount of data in RAM disk filesystem at the same time.
!!! Note
The local RAM disk directory /ramdisk/$PBS_JOBID will be deleted immediately after the calculation end. Users should take care to save the output data from within the jobscript.
The local RAM disk directory /ramdisk/$PBS_JOBID will be deleted immediately after the calculation end. Users should take care to save the output data from within the jobscript.
| RAM disk | |
| ----------- | ------------------------------------------------------------------------------------------------------- |
......@@ -321,7 +321,7 @@ Each node is equipped with local /tmp directory of few GB capacity. The /tmp dir
Do not use shared filesystems at IT4Innovations as a backup for large amount of data or long-term archiving purposes.
!!! Note
The IT4Innovations does not provide storage capacity for data archiving. Academic staff and students of research institutions in the Czech Republic can use [CESNET Storage service](https://du.cesnet.cz/).
The IT4Innovations does not provide storage capacity for data archiving. Academic staff and students of research institutions in the Czech Republic can use [CESNET Storage service](https://du.cesnet.cz/).
The CESNET Storage service can be used for research purposes, mainly by academic staff and students of research institutions in the Czech Republic.
......@@ -340,14 +340,14 @@ The procedure to obtain the CESNET access is quick and trouble-free.
### Understanding CESNET Storage
!!! Note
It is very important to understand the CESNET storage before uploading data. Please read <https://du.cesnet.cz/en/navody/home-migrace-plzen/start> first.
It is very important to understand the CESNET storage before uploading data. Please read <https://du.cesnet.cz/en/navody/home-migrace-plzen/start> first.
Once registered for CESNET Storage, you may [access the storage](https://du.cesnet.cz/en/navody/faq/start) in number of ways. We recommend the SSHFS and RSYNC methods.
### SSHFS Access
!!! Note
SSHFS: The storage will be mounted like a local hard drive
SSHFS: The storage will be mounted like a local hard drive
The SSHFS provides a very convenient way to access the CESNET Storage. The storage will be mounted onto a local directory, exposing the vast CESNET Storage as if it was a local removable hard drive. Files can be than copied in and out in a usual fashion.
......@@ -392,7 +392,7 @@ Once done, please remember to unmount the storage
### Rsync Access
!!! Note
Rsync provides delta transfer for best performance, can resume interrupted transfers
Rsync provides delta transfer for best performance, can resume interrupted transfers
Rsync is a fast and extraordinarily versatile file copying tool. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
......
docs.it4i/salomon/capacity-computing.md
+ 6
− 6
View file @ b4474c1c
......@@ -7,7 +7,7 @@ In many cases, it is useful to submit huge (100+) number of computational jobs i
However, executing huge number of jobs via the PBS queue may strain the system. This strain may result in slow response to commands, inefficient scheduling and overall degradation of performance and user experience, for all users. For this reason, the number of jobs is **limited to 100 per user, 1500 per job array**
!!! Note
Please follow one of the procedures below, in case you wish to schedule more than 100 jobs at a time.
Please follow one of the procedures below, in case you wish to schedule more than 100 jobs at a time.
- Use [Job arrays](capacity-computing.md#job-arrays) when running huge number of [multithread](capacity-computing/#shared-jobscript-on-one-node) (bound to one node only) or multinode (multithread across several nodes) jobs
- Use [GNU parallel](capacity-computing/#gnu-parallel) when running single core jobs
......@@ -21,7 +21,7 @@ However, executing huge number of jobs via the PBS queue may strain the system.
## Job Arrays
!!! Note
Huge number of jobs may be easily submitted and managed as a job array.
Huge number of jobs may be easily submitted and managed as a job array.
A job array is a compact representation of many jobs, called subjobs. The subjobs share the same job script, and have the same values for all attributes and resources, with the following exceptions:
......@@ -152,7 +152,7 @@ Read more on job arrays in the [PBSPro Users guide](../../pbspro-documentation/)
## GNU Parallel
!!! Note
Use GNU parallel to run many single core tasks on one node.
Use GNU parallel to run many single core tasks on one node.
GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. GNU parallel is most useful in running single core jobs via the queue system on Anselm.
......@@ -224,12 +224,12 @@ In this example, we submit a job of 101 tasks. 24 input files will be processed
## Job Arrays and GNU Parallel
!!! Note
Combine the Job arrays and GNU parallel for best throughput of single core jobs
Combine the Job arrays and GNU parallel for best throughput of single core jobs
While job arrays are able to utilize all available computational nodes, the GNU parallel can be used to efficiently run multiple single-core jobs on single node. The two approaches may be combined to utilize all available (current and future) resources to execute single core jobs.
!!! Note
Every subjob in an array runs GNU parallel to utilize all cores on the node
Every subjob in an array runs GNU parallel to utilize all cores on the node
### GNU Parallel, Shared jobscript
......@@ -284,7 +284,7 @@ cp output $PBS_O_WORKDIR/$TASK.out
In this example, the jobscript executes in multiple instances in parallel, on all cores of a computing node. Variable $TASK expands to one of the input filenames from tasklist. We copy the input file to local scratch, execute the myprog.x and copy the output file back to the submit directory, under the $TASK.out name. The numtasks file controls how many tasks will be run per subjob. Once an task is finished, new task starts, until the number of tasks in numtasks file is reached.
!!! Note
Select subjob walltime and number of tasks per subjob carefully
Select subjob walltime and number of tasks per subjob carefully
When deciding this values, think about following guiding rules :
......
docs.it4i/salomon/environment-and-modules.md
+ 2
− 2
View file @ b4474c1c
......@@ -24,7 +24,7 @@ fi
```
!!! Note
Do not run commands outputting to standard output (echo, module list, etc) in .bashrc for non-interactive SSH sessions. It breaks fundamental functionality (scp, PBS) of your account! Take care for SSH session interactivity for such commands as stated in the previous example.
Do not run commands outputting to standard output (echo, module list, etc) in .bashrc for non-interactive SSH sessions. It breaks fundamental functionality (scp, PBS) of your account! Take care for SSH session interactivity for such commands as stated in the previous example.
### Application Modules
......@@ -57,7 +57,7 @@ Application modules on Salomon cluster are built using [EasyBuild](http://hpcuge
```
!!! Note
The modules set up the application paths, library paths and environment variables for running particular application.
The modules set up the application paths, library paths and environment variables for running particular application.
The modules may be loaded, unloaded and switched, according to momentary needs.
......
docs.it4i/salomon/job-priority.md
+ 2
− 2
View file @ b4474c1c
......@@ -37,7 +37,7 @@ Usage counts allocated core-hours (`ncpus x walltime`). Usage is decayed, or cut
# Jobs Queued in Queue qexp Are Not Calculated to Project's Usage.
!!! Note
Calculated usage and fair-share priority can be seen at <https://extranet.it4i.cz/rsweb/salomon/projects>.
Calculated usage and fair-share priority can be seen at <https://extranet.it4i.cz/rsweb/salomon/projects>.
Calculated fair-share priority can be also seen as Resource_List.fairshare attribute of a job.
......@@ -66,7 +66,7 @@ The scheduler makes a list of jobs to run in order of execution priority. Schedu
It means, that jobs with lower execution priority can be run before jobs with higher execution priority.
!!! Note
It is **very beneficial to specify the walltime** when submitting jobs.
It is **very beneficial to specify the walltime** when submitting jobs.
Specifying more accurate walltime enables better scheduling, better execution times and better resource usage. Jobs with suitable (small) walltime could be backfilled - and overtake job(s) with higher priority.
......
docs.it4i/salomon/job-submission-and-execution.md
+ 16
− 16
View file @ b4474c1c
......@@ -12,7 +12,7 @@ When allocating computational resources for the job, please specify
6. Jobscript or interactive switch
!!! Note
Use the **qsub** command to submit your job to a queue for allocation of the computational resources.
Use the **qsub** command to submit your job to a queue for allocation of the computational resources.
Submit the job using the qsub command:
......@@ -23,7 +23,7 @@ $ qsub -A Project_ID -q queue -l select=x:ncpus=y,walltime=[[hh:]mm:]ss[.ms] job
The qsub submits the job into the queue, in another words the qsub command creates a request to the PBS Job manager for allocation of specified resources. The resources will be allocated when available, subject to above described policies and constraints. **After the resources are allocated the jobscript or interactive shell is executed on first of the allocated nodes.**
!!! Note
PBS statement nodes (qsub -l nodes=nodespec) is not supported on Salomon cluster.
PBS statement nodes (qsub -l nodes=nodespec) is not supported on Salomon cluster.
### Job Submission Examples
......@@ -72,7 +72,7 @@ In this example, we allocate 4 nodes, with 24 cores per node (totalling 96 cores
### UV2000 SMP
!!! Note
14 NUMA nodes available on UV2000
14 NUMA nodes available on UV2000
Per NUMA node allocation.
Jobs are isolated by cpusets.
......@@ -109,7 +109,7 @@ $ qsub -m n
### Placement by Name
!!! Note
Not useful for ordinary computing, suitable for node testing/bechmarking and management tasks.
Not useful for ordinary computing, suitable for node testing/bechmarking and management tasks.
Specific nodes may be selected using PBS resource attribute host (for hostnames):
......@@ -136,7 +136,7 @@ For communication intensive jobs it is possible to set stricter requirement - to
Nodes directly connected to the same InifiBand switch can communicate most efficiently. Using the same switch prevents hops in the network and provides for unbiased, most efficient network communication. There are 9 nodes directly connected to every InifiBand switch.
!!! Note
We recommend allocating compute nodes of a single switch when the best possible computational network performance is required to run job efficiently.
We recommend allocating compute nodes of a single switch when the best possible computational network performance is required to run job efficiently.
Nodes directly connected to the one InifiBand switch can be allocated using node grouping on PBS resource attribute switch.
......@@ -149,7 +149,7 @@ $ qsub -A OPEN-0-0 -q qprod -l select=9:ncpus=24 -l place=group=switch ./myjob
### Placement by Specific InifiBand Switch
!!! Note
Not useful for ordinary computing, suitable for testing and management tasks.
Not useful for ordinary computing, suitable for testing and management tasks.
Nodes directly connected to the specific InifiBand switch can be selected using the PBS resource attribute _switch_.
......@@ -234,7 +234,7 @@ r1i0n11
## Job Management
!!! Note
Check status of your jobs using the **qstat** and **check-pbs-jobs** commands
Check status of your jobs using the **qstat** and **check-pbs-jobs** commands
```bash
$ qstat -a
......@@ -313,7 +313,7 @@ Run loop 3
In this example, we see actual output (some iteration loops) of the job 35141.dm2
!!! Note
Manage your queued or running jobs, using the **qhold**, **qrls**, **qdel,** **qsig** or **qalter** commands
Manage your queued or running jobs, using the **qhold**, **qrls**, **qdel,** **qsig** or **qalter** commands
You may release your allocation at any time, using qdel command
......@@ -338,12 +338,12 @@ $ man pbs_professional
### Jobscript
!!! Note
Prepare the jobscript to run batch jobs in the PBS queue system
Prepare the jobscript to run batch jobs in the PBS queue system
The Jobscript is a user made script, controlling sequence of commands for executing the calculation. It is often written in bash, other scripts may be used as well. The jobscript is supplied to PBS **qsub** command as an argument and executed by the PBS Professional workload manager.
!!! Note
The jobscript or interactive shell is executed on first of the allocated nodes.
The jobscript or interactive shell is executed on first of the allocated nodes.
```bash
$ qsub -q qexp -l select=4:ncpus=24 -N Name0 ./myjob
......@@ -360,7 +360,7 @@ Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
In this example, the nodes r21u01n577, r21u02n578, r21u03n579, r21u04n580 were allocated for 1 hour via the qexp queue. The jobscript myjob will be executed on the node r21u01n577, while the nodes r21u02n578, r21u03n579, r21u04n580 are available for use as well.
!!! Note
The jobscript or interactive shell is by default executed in home directory
The jobscript or interactive shell is by default executed in home directory
```bash
$ qsub -q qexp -l select=4:ncpus=24 -I
......@@ -374,7 +374,7 @@ $ pwd
In this example, 4 nodes were allocated interactively for 1 hour via the qexp queue. The interactive shell is executed in the home directory.
!!! Note
All nodes within the allocation may be accessed via ssh. Unallocated nodes are not accessible to user.
All nodes within the allocation may be accessed via ssh. Unallocated nodes are not accessible to user.
The allocated nodes are accessible via ssh from login nodes. The nodes may access each other via ssh as well.
......@@ -406,7 +406,7 @@ In this example, the hostname program is executed via pdsh from the interactive
### Example Jobscript for MPI Calculation
!!! Note
Production jobs must use the /scratch directory for I/O
Production jobs must use the /scratch directory for I/O
The recommended way to run production jobs is to change to /scratch directory early in the jobscript, copy all inputs to /scratch, execute the calculations and copy outputs to home directory.
......@@ -438,12 +438,12 @@ exit
In this example, some directory on the /home holds the input file input and executable mympiprog.x . We create a directory myjob on the /scratch filesystem, copy input and executable files from the /home directory where the qsub was invoked ($PBS_O_WORKDIR) to /scratch, execute the MPI programm mympiprog.x and copy the output file back to the /home directory. The mympiprog.x is executed as one process per node, on all allocated nodes.
!!! Note
Consider preloading inputs and executables onto [shared scratch](storage/) before the calculation starts.
Consider preloading inputs and executables onto [shared scratch](storage/) before the calculation starts.
In some cases, it may be impractical to copy the inputs to scratch and outputs to home. This is especially true when very large input and output files are expected, or when the files should be reused by a subsequent calculation. In such a case, it is users responsibility to preload the input files on shared /scratch before the job submission and retrieve the outputs manually, after all calculations are finished.
!!! Note
Store the qsub options within the jobscript. Use **mpiprocs** and **ompthreads** qsub options to control the MPI job execution.
Store the qsub options within the jobscript. Use **mpiprocs** and **ompthreads** qsub options to control the MPI job execution.
### Example Jobscript for MPI Calculation With Preloaded Inputs
......@@ -477,7 +477,7 @@ HTML commented section #2 (examples need to be reworked)
### Example Jobscript for Single Node Calculation
!!! Note
Local scratch directory is often useful for single node jobs. Local scratch will be deleted immediately after the job ends. Be very careful, use of RAM disk filesystem is at the expense of operational memory.
Local scratch directory is often useful for single node jobs. Local scratch will be deleted immediately after the job ends. Be very careful, use of RAM disk filesystem is at the expense of operational memory.
Example jobscript for single node calculation, using [local scratch](storage/) on the node:
......
docs.it4i/salomon/prace.md
+ 1
− 1
View file @ b4474c1c
......@@ -249,7 +249,7 @@ PRACE users should check their project accounting using the [PRACE Accounting To
Users who have undergone the full local registration procedure (including signing the IT4Innovations Acceptable Use Policy) and who have received local password may check at any time, how many core-hours have been consumed by themselves and their projects using the command "it4ifree". You need to know your user password to use the command and that the displayed core hours are "system core hours" which differ from PRACE "standardized core hours".
!!! Note
The **it4ifree** command is a part of it4i.portal.clients package, located here: <https://pypi.python.org/pypi/it4i.portal.clients>
The **it4ifree** command is a part of it4i.portal.clients package, located here: <https://pypi.python.org/pypi/it4i.portal.clients>
```bash
$ it4ifree
......
docs.it4i/salomon/resource-allocation-and-job-execution.md
+ 2
− 2
View file @ b4474c1c
......@@ -14,14 +14,14 @@ The resources are allocated to the job in a fair-share fashion, subject to const
- **qfree**, the Free resource utilization queue
!!! Note
Check the queue status at <https://extranet.it4i.cz/rsweb/salomon/>
Check the queue status at <https://extranet.it4i.cz/rsweb/salomon/>
Read more on the [Resource Allocation Policy](resources-allocation-policy/) page.
## Job Submission and Execution
!!! Note
Use the **qsub** command to submit your jobs.
Use the **qsub** command to submit your jobs.
The qsub submits the job into the queue. The qsub command creates a request to the PBS Job manager for allocation of specified resources. The **smallest allocation unit is entire node, 24 cores**, with exception of the qexp queue. The resources will be allocated when available, subject to allocation policies and constraints. **After the resources are allocated the jobscript or interactive shell is executed on first of the allocated nodes.**
......
docs.it4i/salomon/resources-allocation-policy.md
+ 5
− 5
View file @ b4474c1c
......@@ -5,7 +5,7 @@
The resources are allocated to the job in a fair-share fashion, subject to constraints set by the queue and resources available to the Project. The fair-share at Anselm ensures that individual users may consume approximately equal amount of resources per week. Detailed information in the [Job scheduling](job-priority/) section. The resources are accessible via several queues for queueing the jobs. The queues provide prioritized and exclusive access to the computational resources. Following table provides the queue partitioning overview:
!!! Note
Check the queue status at <https://extranet.it4i.cz/rsweb/salomon/>
Check the queue status at <https://extranet.it4i.cz/rsweb/salomon/>
| queue | active project | project resources | nodes | min ncpus | priority | authorization | walltime |
| ------------------------------- | -------------- | ----------------- | ------------------------------------------------------------- | --------- | -------- | ------------- | --------- |
......@@ -18,7 +18,7 @@ The resources are allocated to the job in a fair-share fashion, subject to const
| **qviz** Visualization queue | yes | none required | 2 (with NVIDIA Quadro K5000) | 4 | 150 | no | 1 / 8h |
!!! Note
**The qfree queue is not free of charge**. [Normal accounting](resources-allocation-policy/#resources-accounting-policy) applies. However, it allows for utilization of free resources, once a Project exhausted all its allocated computational resources. This does not apply for Directors Discreation's projects (DD projects) by default. Usage of qfree after exhaustion of DD projects computational resources is allowed after request for this queue.
**The qfree queue is not free of charge**. [Normal accounting](resources-allocation-policy/#resources-accounting-policy) applies. However, it allows for utilization of free resources, once a Project exhausted all its allocated computational resources. This does not apply for Directors Discreation's projects (DD projects) by default. Usage of qfree after exhaustion of DD projects computational resources is allowed after request for this queue.
- **qexp**, the Express queue: This queue is dedicated for testing and running very small jobs. It is not required to specify a project to enter the qexp. There are 2 nodes always reserved for this queue (w/o accelerator), maximum 8 nodes are available via the qexp for a particular user. The nodes may be allocated on per core basis. No special authorization is required to use it. The maximum runtime in qexp is 1 hour.
- **qprod**, the Production queue: This queue is intended for normal production runs. It is required that active project with nonzero remaining resources is specified to enter the qprod. All nodes may be accessed via the qprod queue, however only 86 per job. Full nodes, 24 cores per node are allocated. The queue runs with medium priority and no special authorization is required to use it. The maximum runtime in qprod is 48 hours.
......@@ -29,7 +29,7 @@ The resources are allocated to the job in a fair-share fashion, subject to const
- **qviz**, the Visualization queue: Intended for pre-/post-processing using OpenGL accelerated graphics. Currently when accessing the node, each user gets 4 cores of a CPU allocated, thus approximately 73 GB of RAM and 1/7 of the GPU capacity (default "chunk"). If more GPU power or RAM is required, it is recommended to allocate more chunks (with 4 cores each) up to one whole node per user, so that all 28 cores, 512 GB RAM and whole GPU is exclusive. This is currently also the maximum allowed allocation per one user. One hour of work is allocated by default, the user may ask for 2 hours maximum.
!!! Note
To access node with Xeon Phi co-processor user needs to specify that in [job submission select statement](job-submission-and-execution/).
To access node with Xeon Phi co-processor user needs to specify that in [job submission select statement](job-submission-and-execution/).
### Notes
......@@ -42,7 +42,7 @@ Salomon users may check current queue configuration at <https://extranet.it4i.cz
### Queue Status
!!! Note
Check the status of jobs, queues and compute nodes at [https://extranet.it4i.cz/rsweb/salomon/](https://extranet.it4i.cz/rsweb/salomon)
Check the status of jobs, queues and compute nodes at [https://extranet.it4i.cz/rsweb/salomon/](https://extranet.it4i.cz/rsweb/salomon)
![RSWEB Salomon](../img/rswebsalomon.png "RSWEB Salomon")
......@@ -120,7 +120,7 @@ The resources that are currently subject to accounting are the core-hours. The c
### Check Consumed Resources
!!! Note
The **it4ifree** command is a part of it4i.portal.clients package, located here: <https://pypi.python.org/pypi/it4i.portal.clients>
The **it4ifree** command is a part of it4i.portal.clients package, located here: <https://pypi.python.org/pypi/it4i.portal.clients>
User may check at any time, how many core-hours have been consumed by himself/herself and his/her projects. The command is available on clusters' login nodes.
......
docs.it4i/salomon/shell-and-data-access.md
+ 8
− 8
View file @ b4474c1c
......@@ -5,7 +5,7 @@
The Salomon cluster is accessed by SSH protocol via login nodes login1, login2, login3 and login4 at address salomon.it4i.cz. The login nodes may be addressed specifically, by prepending the login node name to the address.
!!! Note
The alias salomon.it4i.cz is currently not available through VPN connection. Please use loginX.salomon.it4i.cz when connected to VPN.
The alias salomon.it4i.cz is currently not available through VPN connection. Please use loginX.salomon.it4i.cz when connected to VPN.
| Login address | Port | Protocol | Login node |
| ---------------------- | ---- | -------- | ------------------------------------- |
......@@ -18,9 +18,9 @@ The Salomon cluster is accessed by SSH protocol via login nodes login1, login2,
The authentication is by the [private key](../get-started-with-it4innovations/accessing-the-clusters/shell-access-and-data-transfer/ssh-keys/)
!!! Note
Please verify SSH fingerprints during the first logon. They are identical on all login nodes:
f6:28:98:e4:f9:b2:a6:8f:f2:f4:2d:0a:09:67:69:80 (DSA)
70:01:c9:9a:5d:88:91:c7:1b:c0:84:d1:fa:4e:83:5c (RSA)
Please verify SSH fingerprints during the first logon. They are identical on all login nodes:
f6:28:98:e4:f9:b2:a6:8f:f2:f4:2d:0a:09:67:69:80 (DSA)
70:01:c9:9a:5d:88:91:c7:1b:c0:84:d1:fa:4e:83:5c (RSA)
Private key authentication:
......@@ -57,7 +57,7 @@ Last login: Tue Jul 9 15:57:38 2013 from your-host.example.com
```
!!! Note
The environment is **not** shared between login nodes, except for [shared filesystems](storage/).
The environment is **not** shared between login nodes, except for [shared filesystems](storage/).
## Data Transfer
......@@ -121,7 +121,7 @@ Outgoing connections, from Salomon Cluster login nodes to the outside world, are
| 9418 | git |
!!! Note
Please use **ssh port forwarding** and proxy servers to connect from Salomon to all other remote ports.
Please use **ssh port forwarding** and proxy servers to connect from Salomon to all other remote ports.
Outgoing connections, from Salomon Cluster compute nodes are restricted to the internal network. Direct connections form compute nodes to outside world are cut.
......@@ -130,7 +130,7 @@ Outgoing connections, from Salomon Cluster compute nodes are restricted to the i
### Port Forwarding From Login Nodes
!!! Note
Port forwarding allows an application running on Salomon to connect to arbitrary remote host and port.
Port forwarding allows an application running on Salomon to connect to arbitrary remote host and port.
It works by tunneling the connection from Salomon back to users workstation and forwarding from the workstation to the remote host.
......@@ -171,7 +171,7 @@ In this example, we assume that port forwarding from login1:6000 to remote.host.
Port forwarding is static, each single port is mapped to a particular port on remote host. Connection to other remote host, requires new forward.
!!! Note
Applications with inbuilt proxy support, experience unlimited access to remote hosts, via single proxy server.
Applications with inbuilt proxy support, experience unlimited access to remote hosts, via single proxy server.
To establish local proxy server on your workstation, install and run SOCKS proxy server software. On Linux, sshd demon provides the functionality. To establish SOCKS proxy server listening on port 1080 run:
......
docs.it4i/salomon/software/chemistry/molpro.md
+ 1
− 1
View file @ b4474c1c
......@@ -33,7 +33,7 @@ Compilation parameters are default:
Molpro is compiled for parallel execution using MPI and OpenMP. By default, Molpro reads the number of allocated nodes from PBS and launches a data server on one node. On the remaining allocated nodes, compute processes are launched, one process per node, each with 16 threads. You can modify this behavior by using -n, -t and helper-server options. Please refer to the [Molpro documentation](http://www.molpro.net/info/2010.1/doc/manual/node9.html) for more details.
!!! Note
The OpenMP parallelization in Molpro is limited and has been observed to produce limited scaling. We therefore recommend to use MPI parallelization only. This can be achieved by passing option mpiprocs=16:ompthreads=1 to PBS.
The OpenMP parallelization in Molpro is limited and has been observed to produce limited scaling. We therefore recommend to use MPI parallelization only. This can be achieved by passing option mpiprocs=16:ompthreads=1 to PBS.
You are advised to use the -d option to point to a directory in [SCRATCH filesystem](../../storage/storage/). Molpro can produce a large amount of temporary data during its run, and it is important that these are placed in the fast scratch filesystem.
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment