4 merge requests!368Update prace.md to document the change from qprace to qprod as the default...,!367Update prace.md to document the change from qprace to qprod as the default...,!366Update prace.md to document the change from qprace to qprod as the default...,!323extended-acls-storage-section
The qsub command submits the job to the queue, i.e. the qsub command creates a request to the PBS Job manager for allocation of specified resources. The resources will be allocated when available, subject to the above described policies and constraints. **After the resources are allocated, the jobscript or interactive shell is executed on the first of the allocated nodes.**
!!! note
PBS statement nodes (qsub -l nodes=nodespec) are not supported on the Anselm cluster.
PBS statement nodes (qsub -l nodes=nodespec) are not supported on Anselm.
### Job Submission Examples
...
...
@@ -46,7 +46,7 @@ In this example, we allocate 4 nodes, 16 cores per node, for 1 hour. We allocate
$qsub -A OPEN-0-0 -q qnvidia -lselect=10:ncpus=16 ./myjob
```
In this example, we allocate 10 nvidia accelerated nodes, 16 cores per node, for 24 hours. We allocate these resources via the qnvidia queue. the jobscript 'myjob' will be executed on the first node in the allocation.
In this example, we allocate 10 NVIDIA accelerated nodes, 16 cores per node, for 24 hours. We allocate these resources via the qnvidia queue. The jobscript 'myjob' will be executed on the first node in the allocation.
```console
$qsub -A OPEN-0-0 -q qfree -lselect=10:ncpus=16 ./myjob
...
...
@@ -60,7 +60,7 @@ All qsub options may be [saved directly into the jobscript][1]. In such cases, i
$qsub ./myjob
```
By default, the PBS batch system sends an e-mail only when the job is aborted. Disabling mail events completely can be done as follows:
By default, the PBS batch system sends an email only when the job is aborted. Disabling mail events completely can be done as follows:
```console
$qsub -m n
...
...
@@ -68,24 +68,24 @@ $ qsub -m n
### Salomon - Intel Xeon Phi Co-Processors
To allocate a node with Xeon Phi co-processor, user needs to specify that in select statement. Currently only allocation of whole nodes with both Phi cards as the smallest chunk is supported. Standard PBSPro approach through attributes "accelerator", "naccelerators" and "accelerator_model" is used. The "accelerator_model" can be omitted, since on Salomon only one type of accelerator type/model is available.
The absence of specialized queue for accessing the nodes with cards means, that the Phi cards can be utilized in any queue, including qexp for testing/experiments, qlong for longer jobs, qfree after the project resources have been spent, etc. The Phi cards are thus also available to PRACE users. There's no need to ask for permission to utilize the Phi cards in project proposals.
To allocate a node with Xeon Phi co-processor, the user needs to specify that in the select statement. Currently only allocation of whole nodes with both Phi cards as the smallest chunk is supported. A standard PBSPro approach through the "accelerator", "naccelerators", and "accelerator_model" attributes is used. The "accelerator_model" can be omitted since on Salomon, only one type of accelerator type/model is available.
The absence of specialized queue for accessing the nodes with cards means, that the Phi cards can be utilized in any queue, including qexp for testing/experiments, qlong for longer jobs, qfree after the project resources have been spent, etc. The Phi cards are thus also available to PRACE users. There is no need to ask for permission to utilize the Phi cards in project proposals.
```console
$qsub -A OPEN-0-0 -I-q qprod -lselect=1:ncpus=24:accelerator=True:naccelerators=2:accelerator_model=phi7120 ./myjob
```
In this example, we allocate 1 node, with 24 cores, with 2 Xeon Phi 7120p cards, running batch job ./myjob. The default time for qprod is used, e.g. 24 hours.
In this example, we allocate 1 node with 24 cores, with 2 Xeon Phi 7120p cards, running batch job ./myjob. The default time for qprod is used, e.g. 24 hours.
```console
$qsub -A OPEN-0-0 -I-q qlong -lselect=4:ncpus=24:accelerator=True:naccelerators=2 -lwalltime=56:00:00 -I
```
In this example, we allocate 4 nodes, with 24 cores per node (totalling 96 cores), with 2 Xeon Phi 7120p cards per node (totalling 8 Phi cards), running interactive job for 56 hours. The accelerator model name was omitted.
In this example, we allocate 4 nodes with 24 cores per node (totaling 96 cores), with 2 Xeon Phi 7120p cards per node (totaling 8 Phi cards), running interactive job for 56 hours. The accelerator model name was omitted.
#### Salomon - Intel Xeon Phi - Queue QMIC
Examples executions
Examples executions:
```console
-l select=1
...
...
@@ -117,25 +117,25 @@ The UV2000 (node uv1) offers 3TB of RAM and 104 cores, distributed in 13 NUMA no
$qsub -A OPEN-0-0 -q qfat -lselect=13 ./myjob
```
In this example, we allocate all 13 NUMA nodes (corresponds to 13 chunks), 104 cores of the SGI UV2000 node for 24 hours. Jobscript myjob will be executed on the node uv1.
In this example, we allocate all 13 NUMA nodes (corresponds to 13 chunks), 104 cores of the SGI UV2000 node for 24 hours. The myjob jobscript will be executed on the uv1 node.
```console
$qsub -A OPEN-0-0 -q qfat -lselect=1:mem=2000GB ./myjob
```
In this example, we allocate 2000GB of memory on the UV2000 for 24 hours. By requesting 2000GB of memory, memory from 10 chunks and 8 cores are allocated. Jobscript myjob will be executed on the node uv1.
In this example, we allocate 2000GB of memory on the UV2000 for 24 hours. By requesting 2000GB of memory, memory from 10 chunks and 8 cores is allocated. The myjob jobscript will be executed on the uv1 node.
```console
$qsub -A OPEN-0-0 -q qfat -lselect=1:mem=3099GB,walltime=48:00:00 ./myjob
```
In this example, we allocate 3099GB of memory on the UV2000 for 48 hours. By requesting 3099GB of memory, memory from all 13 chunks and 8 cores are allocated. Jobscript myjob will be executed on the node uv1.
In this example, we allocate 3099GB of memory on the UV2000 for 48 hours. By requesting 3099GB of memory, memory from all 13 chunks and 8 cores is allocated. The myjob jobscript will be executed on the uv1 node.
```console
$qsub -A OPEN-0-0 -q qfat -lselect=2:mem=1000GB,walltime=48:00:00 ./myjob
```
In this example, we allocate 2000GB of memory and 16 cores on the UV2000 for 48 hours. By requesting 1000GB of memory per chunk, 2000GB of memory and 16 cores are allocated. Jobscript myjob will be executed on the node uv1.
In this example, we allocate 2000GB of memory and 16 cores on the UV2000 for 48 hours. By requesting 1000GB of memory per chunk, 2000GB of memory and 16 cores are allocated. The myjob jobscript will be executed on the uv1 node.
### Useful Tricks
...
...
@@ -145,7 +145,7 @@ All qsub options may be [saved directly into the jobscript][2]. In such a case,
$qsub ./myjob
```
By default, the PBS batch system sends an e-mail only when the job is aborted. Disabling mail events completely can be done like this:
By default, the PBS batch system sends an email only when the job is aborted. Disabling mail events completely can be done like this:
```console
$qsub -m n
...
...
@@ -155,7 +155,7 @@ $ qsub -m n
### Placement by Name
Specific nodes may be allocated via the PBS
Specific nodes may be allocated via PBS:
```console
$qsub -A OPEN-0-0 -q qprod -lselect=1:ncpus=16:host=cn171+1:ncpus=16:host=cn172 -I
...
...
@@ -169,7 +169,7 @@ In this example, we allocate nodes cn171 and cn172, all 16 cores per node, for 2
### Anselm - Placement by CPU Type
Nodes equipped with an Intel Xeon E5-2665 CPU have a base clock frequency of 2.4GHz, nodes equipped with an Intel Xeon E5-2470 CPU have a base frequency of 2.3 GHz (see the section Compute Nodes for details). Nodes may be selected via the PBS resource attribute cpu_freq.
Nodes equipped with an Intel Xeon E5-2665 CPU have a base clock frequency of 2.4GHz; nodes equipped with an Intel Xeon E5-2470 CPU have a base frequency of 2.3 GHz (for details, see the Compute Nodes section). Nodes may be selected via the PBS resource attribute cpu_freq.
#### Anselm
...
...
@@ -182,7 +182,7 @@ Nodes equipped with an Intel Xeon E5-2665 CPU have a base clock frequency of 2.4
$qsub -A OPEN-0-0 -q qprod -lselect=4:ncpus=16:cpu_freq=24 -I
```
In this example, we allocate 4 nodes, 16 cores per node, selecting only the nodes with Intel Xeon E5-2665 CPU.
In this example, we allocate 4 nodes, 16 cores per node, selecting only the nodes with the Intel Xeon E5-2665 CPU.
### Anselm - Placement by IB Switch
...
...
@@ -196,13 +196,13 @@ We recommend allocating compute nodes to a single switch when best possible comp
$qsub -A OPEN-0-0 -q qprod -lselect=18:ncpus=16:ibswitch=isw11 ./myjob
```
In this example, we request all of the 18 nodes sharing the isw11 switch for 24 hours. a full chassis will be allocated.
In this example, we request all of the 18 nodes sharing the isw11 switch for 24 hours. A full chassis will be allocated.
### Salomon - Placement by Network Location
Network location of allocated nodes in the [InifiBand network][3] influences efficiency of network communication between nodes of job. Nodes on the same InifiBand switch communicate faster with lower latency than distant nodes. To improve communication efficiency of jobs, PBS scheduler on Salomon is configured to allocate nodes - from currently available resources - which are as close as possible in the network topology.
The network location of allocated nodes in the [InifiBand network][3] influences efficiency of network communication between nodes of job. Nodes on the same InifiBand switch communicate faster with lower latency than distant nodes. To improve communication efficiency of jobs, PBS scheduler on Salomon is configured to allocate nodes (from currently available resources), which are as close as possible in the network topology.
For communication intensive jobs it is possible to set stricter requirement - to require nodes directly connected to the same InifiBand switch or to require nodes located in the same dimension group of the InifiBand network.
For communication intensive jobs, it is possible to set stricter requirement - to require nodes directly connected to the same InifiBand switch or to require nodes located in the same dimension group of the InifiBand network.
### Salomon - Placement by InifiBand Switch
...
...
@@ -211,7 +211,7 @@ Nodes directly connected to the same InifiBand switch can communicate most effic
!!! note
We recommend allocating compute nodes of a single switch when the best possible computational network performance is required to run job efficiently.
Nodes directly connected to the one InifiBand switch can be allocated using node grouping on PBS resource attribute switch.
Nodes directly connected to the one InifiBand switch can be allocated using node grouping on the PBS resource attribute _switch_.
In this example, we request all 9 nodes directly connected to the same switch using node grouping placement.
Intel Turbo Boost Technology is on by default. We strongly recommend keeping the default.
If necessary (such as in the case of benchmarking) you can disable the Turbo for all nodes of the job by using the PBS resource attribute cpu_turbo_boost:
If necessary (such as in the case of benchmarking), you can disable the Turbo for all nodes of the job by using the PBS resource attribute cpu_turbo_boost:
```console
$qsub -A OPEN-0-0 -q qprod -lselect=4:ncpus=16 -lcpu_turbo_boost=0 -I
...
...
@@ -358,9 +358,9 @@ Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
In this example user1 and user2 are running jobs named job1, job2 and job3x. The jobs job1 and job2 are using 4 nodes, 16 cores per node each. job1 has already run for 38 hours and 25 minutes, and job2 for 17 hours 44 minutes. job1 has already consumed `64 x 38.41 = 2458.6` core hours. job3x has already consumed `0.96 x 32 = 30.93` core hours. These consumed core hours will be accounted for on the respective project accounts, regardless of whether the allocated cores were actually used for computations.
In this example user1 and user2 are running jobs named job1, job2, and job3x. The jobs job1 and job2 are using 4 nodes, 16 cores per node each. job1 has already run for 38 hours and 25 minutes, and job2 for 17 hours 44 minutes. job1 has already consumed `64 x 38.41 = 2458.6` core hours. job3x has already consumed `0.96 x 32 = 30.93` core hours. These consumed core hours will be accounted for on the respective project accounts, regardless of whether the allocated cores were actually used for computations.
The following commands allow you to; check the status of your jobs using the check-pbs-jobs command; check for the presence of user's PBS jobs' processes on execution hosts; display load and processes; display job standard and error output; continuously display (tail -f) job standard or error output;
The following commands allow you to check the status of your jobs using the check-pbs-jobs command, check for the presence of user's PBS jobs' processes on execution hosts, display load and processes, display job standard and error output, and continuously display (tail -f) job standard or error output.
```console
$check-pbs-jobs --check-all
...
...
@@ -381,7 +381,7 @@ cn164: OK
cn165: No process
```
In this example we see that job 35141.dm2 is not currently running any processes on the allocated node cn165, which may indicate an execution error.
In this example we see that job 35141.dm2 is not currently running any processes on the allocated node cn165, which may indicate an execution error:
```console
$check-pbs-jobs --print-load--print-processes
...
...
@@ -397,7 +397,7 @@ cn164: 99.7 run-task
...
```
In this example we see that job 35141.dm2 is currently running a process run-task on node cn164, using one thread only, while node cn165 is empty, which may indicate an execution error.
In this example, we see that job 35141.dm2 is currently running a process run-task on node cn164, using one thread only, while node cn165 is empty, which may indicate an execution error.
```console
$check-pbs-jobs --jobid 35141.dm2 --print-job-out
...
...
@@ -411,10 +411,10 @@ Run loop 2
Run loop 3
```
In this example, we see actual output (some iteration loops) of the job 35141.dm2
In this example, we see the actual output (some iteration loops) of the job 35141.dm2.
!!! note
Manage your queued or running jobs, using the **qhold**, **qrls**, **qdel**, **qsig** or **qalter** commands
Manage your queued or running jobs, using the **qhold**, **qrls**, **qdel**, **qsig**, or **qalter** commands
You may release your allocation at any time, using the qdel command
...
...
@@ -458,9 +458,9 @@ Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
cn17/0*16+cn108/0*16+cn109/0*16+cn110/0*16
```
In this example, the nodes cn17, cn108, cn109, and cn110 were allocated for 1 hour via the qexp queue. The jobscript myjob will be executed on the node cn17, while the nodes cn108, cn109, and cn110 are available for use as well.
In this example, the nodes cn17, cn108, cn109, and cn110 were allocated for 1 hour via the qexp queue. The myjob jobscript will be executed on the node cn17, while the nodes cn108, cn109, and cn110 are available for use as well.
The jobscript or interactive shell is by default executed in the home directory
The jobscript or interactive shell is by default executed in the home directory:
```console
$qsub -q qexp -lselect=4:ncpus=16 -I
...
...
@@ -474,11 +474,11 @@ $ pwd
In this example, 4 nodes were allocated interactively for 1 hour via the qexp queue. The interactive shell is executed in the home directory.
!!! note
All nodes within the allocation may be accessed via ssh. Unallocated nodes are not accessible to the user.
All nodes within the allocation may be accessed via SSH. Unallocated nodes are not accessible to the user.
The allocated nodes are accessible via ssh from login nodes. The nodes may access each other via ssh as well.
The allocated nodes are accessible via SSH from login nodes. The nodes may access each other via SSH as well.
Calculations on allocated nodes may be executed remotely via the MPI, ssh, pdsh or clush. You may find out which nodes belong to the allocation by reading the $PBS_NODEFILE file
Calculations on allocated nodes may be executed remotely via the MPI, SSH, pdsh, or clush. You may find out which nodes belong to the allocation by reading the $PBS_NODEFILE file
```console
qsub -q qexp -l select=4:ncpus=16 -I
...
...
@@ -501,14 +501,14 @@ cn109: cn109
cn110: cn110
```
In this example, the hostname program is executed via pdsh from the interactive shell. The execution runs on all four allocated nodes. The same result would be achieved if the pdsh is called from any of the allocated nodes or from the login nodes.
In this example, the hostname program is executed via pdsh from the interactive shell. The execution runs on all four allocated nodes. The same result would be achieved if the pdsh were called from any of the allocated nodes or from the login nodes.
### Example Jobscript for MPI Calculation
!!! note
Production jobs must use the /scratch directory for I/O
The recommended way to run production jobs is to change to the /scratch directory early in the jobscript, copy all inputs to /scratch, execute the calculations and copy outputs to the home directory.
The recommended way to run production jobs is to change to the /scratch directory early in the jobscript, copy all inputs to /scratch, execute the calculations, and copy outputs to the home directory.
```bash
#!/bin/bash
...
...
@@ -535,7 +535,7 @@ cp output $PBS_O_WORKDIR/.
exit
```
In this example, a directory in /home holds the input file input and executable mympiprog.x . We create the directory myjob on the /scratch filesystem, copy input and executable files from the /home directory where the qsub was invoked ($PBS_O_WORKDIR) to /scratch, execute the MPI program mympiprog.x and copy the output file back to the /home directory. mympiprog.x is executed as one process per node, on all allocated nodes.
In this example, a directory in /home holds the input file input and the mympiprog.x executable. We create the myjob directory on the /scratch filesystem, copy input and executable files from the /home directory where the qsub was invoked ($PBS_O_WORKDIR) to /scratch, execute the MPI program mympiprog.x and copy the output file back to the /home directory. mympiprog.x is executed as one process per node, on all allocated nodes.
!!! note
Consider preloading inputs and executables onto [shared scratch][4] memory before the calculation starts.
...
...
@@ -543,7 +543,7 @@ In this example, a directory in /home holds the input file input and executable
In some cases, it may be impractical to copy the inputs to the scratch memory and the outputs to the home directory. This is especially true when very large input and output files are expected, or when the files should be reused by a subsequent calculation. In such cases, it is the users' responsibility to preload the input files on shared /scratch memory before the job submission, and retrieve the outputs manually after all calculations are finished.
!!! note
Store the qsub options within the jobscript. Use **mpiprocs** and **ompthreads** qsub options to control the MPI job execution.
Store the qsub options within the jobscript. Use the **mpiprocs** and **ompthreads** qsub options to control the MPI job execution.
### Example Jobscript for MPI Calculation With Preloaded Inputs
