4 merge requests!368Update prace.md to document the change from qprace to qprod as the default...,!367Update prace.md to document the change from qprace to qprod as the default...,!366Update prace.md to document the change from qprace to qprod as the default...,!323extended-acls-storage-section
@@ -11,10 +11,7 @@ When allocating computational resources for the job, specify:
1. your Project ID
1. a Jobscript or interactive switch
!!! note
Use the **qsub** command to submit your job to a queue for allocation of computational resources.
Submit the job using the qsub command:
Submit the job using the `qsub` command:
```console
$qsub -A Project_ID -q queue -lselect=x:ncpus=y,walltime=[[hh:]mm:]ss[.ms] jobscript
...
...
@@ -68,7 +65,7 @@ $ qsub -m n
### Salomon - Intel Xeon Phi Co-Processors
To allocate a node with Xeon Phi co-processor, the user needs to specify that in the select statement. Currently only allocation of whole nodes with both Phi cards as the smallest chunk is supported. A standard PBSPro approach through the "accelerator", "naccelerators", and "accelerator_model" attributes is used. The "accelerator_model" can be omitted since on Salomon, only one type of accelerator type/model is available.
To allocate a node with Xeon Phi co-processor, the user needs to specify that in the select statement. Currently only allocation of whole nodes with both Phi cards as the smallest chunk is supported. A standard PBSPro approach through the `accelerator`, `naccelerators`, and `accelerator_model` attributes is used. The `accelerator_model` can be omitted since on Salomon, only one type of accelerator type/model is available.
The absence of specialized queue for accessing the nodes with cards means, that the Phi cards can be utilized in any queue, including qexp for testing/experiments, qlong for longer jobs, qfree after the project resources have been spent, etc. The Phi cards are thus also available to PRACE users. There is no need to ask for permission to utilize the Phi cards in project proposals.
The UV2000 (node uv1) offers 3TB of RAM and 104 cores, distributed in 13 NUMA nodes. A NUMA node packs 8 cores and approx. 247GB RAM (with exception, node 11 has only 123GB RAM). In the PBS the UV2000 provides 13 chunks, a chunk per NUMA node (see [Resource allocation policy][1]). The jobs on UV2000 are isolated from each other by cpusets, so that a job by one user may not utilize CPU or memory allocated to a job by other user. Always, full chunks are allocated, a job may only use resources of the NUMA nodes allocated to itself.
The UV2000 (node uv1) offers 3TB of RAM and 104 cores, distributed in 13 NUMA nodes. A NUMA node packs 8 cores and approx. 247GB RAM (with exception, node 11 has only 123GB RAM). In the PBS the UV2000 provides 13 chunks, a chunk per NUMA node (see [Resource allocation policy][2]). The jobs on UV2000 are isolated from each other by cpusets, so that a job by one user may not utilize CPU or memory allocated to a job by other user. Always, full chunks are allocated, a job may only use resources of the NUMA nodes allocated to itself.
```console
$qsub -A OPEN-0-0 -q qfat -lselect=13 ./myjob
...
...
@@ -139,7 +136,7 @@ In this example, we allocate 2000GB of memory and 16 cores on the UV2000 for 48
### Useful Tricks
All qsub options may be [saved directly into the jobscript][2]. In such a case, no options to qsub are needed.
All qsub options may be [saved directly into the jobscript][1]. In such a case, no options to qsub are needed.
```console
$qsub ./myjob
...
...
@@ -186,9 +183,9 @@ In this example, we allocate 4 nodes, 16 cores per node, selecting only the node
### Anselm - Placement by IB Switch
Groups of computational nodes are connected to chassis integrated Infiniband switches. These switches form the leaf switch layer of the [Infiniband network][2] fat tree topology. Nodes sharing the leaf switch can communicate most efficiently. Sharing the same switch prevents hops in the network and facilitates unbiased, highly efficient network communication.
Groups of computational nodes are connected to chassis integrated Infiniband switches. These switches form the leaf switch layer of the [Infiniband network][3] fat tree topology. Nodes sharing the leaf switch can communicate most efficiently. Sharing the same switch prevents hops in the network and facilitates unbiased, highly efficient network communication.
Nodes sharing the same switch may be selected via the PBS resource attribute ibswitch. Values of this attribute are iswXX, where XX is the switch number. The node-switch mapping can be seen in the [Hardware Overview][3] section.
Nodes sharing the same switch may be selected via the PBS resource attribute `ibswitch`. Values of this attribute are `iswXX`, where `XX` is the switch number. The node-switch mapping can be seen in the [Hardware Overview][4] section.
We recommend allocating compute nodes to a single switch when best possible computational network performance is required to run the job efficiently:
...
...
@@ -211,7 +208,7 @@ Nodes directly connected to the same InifiBand switch can communicate most effic
!!! note
We recommend allocating compute nodes of a single switch when the best possible computational network performance is required to run job efficiently.
Nodes directly connected to the one InifiBand switch can be allocated using node grouping on the PBS resource attribute _switch_.
Nodes directly connected to the one InifiBand switch can be allocated using node grouping on the PBS resource attribute `switch`.
In this example, we request all 9 nodes directly connected to the same switch using node grouping placement.
Not useful for ordinary computing, suitable for testing and management tasks.
Nodes directly connected to the specific InifiBand switch can be selected using the PBS resource attribute _switch_.
Nodes directly connected to the specific InifiBand switch can be selected using the PBS resource attribute `switch`.
In this example, we request all 9 nodes directly connected to the r4i1s0sw1 switch.
...
...
@@ -273,7 +270,7 @@ Nodes located in the same dimension group may be allocated using node grouping o
| 6D | ehc_6d | 432,576 |
| 7D | ehc_7d | all |
In this example, we allocate 16 nodes in the same [hypercube dimension][4] 1 group.
In this example, we allocate 16 nodes in the same [hypercube dimension][5] 1 group.
```console
$qsub -A OPEN-0-0 -q qprod -lselect=16:ncpus=24 -lplace=group=ehc_1d -I
...
...
@@ -335,7 +332,7 @@ Although this example is somewhat artificial, it demonstrates the flexibility of
## Job Management
!!! note
Check status of your jobs using the **qstat** and **check-pbs-jobs** commands
Check status of your jobs using the `qstat` and `check-pbs-jobs` commands
```console
$qstat -a
...
...
@@ -414,21 +411,21 @@ Run loop 3
In this example, we see the actual output (some iteration loops) of the job 35141.dm2.
!!! note
Manage your queued or running jobs, using the **qhold**, **qrls**, **qdel**, **qsig**, or **qalter** commands
Manage your queued or running jobs, using the `qhold`, `qrls`, `qdel`, `qsig`, or `qalter` commands
You may release your allocation at any time, using the qdel command
You may release your allocation at any time, using the `qdel` command
```console
$qdel 12345.srv11
```
You may kill a running job by force, using the qsig command
You may kill a running job by force, using the `qsig` command
```console
$qsig -s 9 12345.srv11
```
Learn more by reading the pbs man page
Learn more by reading the PBS man page
```console
$man pbs_professional
...
...
@@ -441,7 +438,7 @@ $ man pbs_professional
!!! note
Prepare the jobscript to run batch jobs in the PBS queue system
The Jobscript is a user made script controlling a sequence of commands for executing the calculation. It is often written in bash, though other scripts may be used as well. The jobscript is supplied to the PBS **qsub** command as an argument, and is executed by the PBS Professional workload manager.
The Jobscript is a user made script controlling a sequence of commands for executing the calculation. It is often written in bash, though other scripts may be used as well. The jobscript is supplied to the PBS `qsub` command as an argument, and is executed by the PBS Professional workload manager.
!!! note
The jobscript or interactive shell is executed on first of the allocated nodes.
...
...
@@ -474,7 +471,7 @@ $ pwd
In this example, 4 nodes were allocated interactively for 1 hour via the qexp queue. The interactive shell is executed in the home directory.
!!! note
All nodes within the allocation may be accessed via SSH. Unallocated nodes are not accessible to the user.
All nodes within the allocation may be accessed via SSH. Unallocated nodes are not accessible to the user.
The allocated nodes are accessible via SSH from login nodes. The nodes may access each other via SSH as well.
...
...
@@ -538,12 +535,12 @@ exit
In this example, a directory in /home holds the input file input and the mympiprog.x executable. We create the myjob directory on the /scratch filesystem, copy input and executable files from the /home directory where the qsub was invoked ($PBS_O_WORKDIR) to /scratch, execute the MPI program mympiprog.x and copy the output file back to the /home directory. mympiprog.x is executed as one process per node, on all allocated nodes.
!!! note
Consider preloading inputs and executables onto [shared scratch][4] memory before the calculation starts.
Consider preloading inputs and executables onto [shared scratch][6] memory before the calculation starts.
In some cases, it may be impractical to copy the inputs to the scratch memory and the outputs to the home directory. This is especially true when very large input and output files are expected, or when the files should be reused by a subsequent calculation. In such cases, it is the users' responsibility to preload the input files on shared /scratch memory before the job submission, and retrieve the outputs manually after all calculations are finished.
!!! note
Store the qsub options within the jobscript. Use the **mpiprocs** and **ompthreads** qsub options to control the MPI job execution.
Store the qsub options within the jobscript. Use the `mpiprocs` and `ompthreads` qsub options to control the MPI job execution.
### Example Jobscript for MPI Calculation With Preloaded Inputs
...
...
@@ -570,16 +567,16 @@ mpirun ./mympiprog.x
exit
```
In this example, input and executable files are assumed to be preloaded manually in the /scratch/$USER/myjob directory. Note the **mpiprocs** and **ompthreads** qsub options controlling the behavior of the MPI execution. mympiprog.x is executed as one process per node, on all 100 allocated nodes. If mympiprog.x implements OpenMP threads, it will run 16 threads per node.
In this example, input and executable files are assumed to be preloaded manually in the /scratch/$USER/myjob directory. Note the `mpiprocs` and `ompthreads` qsub options controlling the behavior of the MPI execution. mympiprog.x is executed as one process per node, on all 100 allocated nodes. If mympiprog.x implements OpenMP threads, it will run 16 threads per node.
More information can be found in the [Running OpenMPI][5] and [Running MPICH2][6] sections.
More information can be found in the [Running OpenMPI][7] and [Running MPICH2][8] sections.
### Example Jobscript for Single Node Calculation
!!! note
The local scratch directory is often useful for single node jobs. Local scratch memory will be deleted immediately after the job ends.
Example jobscript for single node calculation, using [local scratch][4] memory on the node:
Example jobscript for single node calculation, using [local scratch][6] memory on the node:
```bash
#!/bin/bash
...
...
@@ -605,12 +602,14 @@ In this example, a directory in /home holds the input file input and executable
### Other Jobscript Examples
Further jobscript examples may be found in the software section and the [Capacity computing][7] section.
Further jobscript examples may be found in the software section and the [Capacity computing][9] section.
