job-submission-and-execution.md

Job submission and execution

Job Submission

When allocating computational resources for the job, please specify

1. suitable queue for your job (default is qprod)
2. number of computational nodes required
3. number of cores per node required
4. maximum wall time allocated to your calculation, note that jobs exceeding maximum wall time will be killed
5. Project ID
6. Jobscript or interactive switch

!!! Note "Note" Use the qsub command to submit your job to a queue for allocation of the computational resources.

Submit the job using the qsub command:

$ qsub -A Project_ID -q queue -l select=x:ncpus=y,walltime=[[hh:]mm:]ss[.ms] jobscript

The qsub submits the job into the queue, in another words the qsub command creates a request to the PBS Job manager for allocation of specified resources. The resources will be allocated when available, subject to above described policies and constraints. After the resources are allocated the jobscript or interactive shell is executed on first of the allocated nodes.

Job Submission Examples

$ qsub -A OPEN-0-0 -q qprod -l select=64:ncpus=16,walltime=03:00:00 ./myjob

In this example, we allocate 64 nodes, 16 cores per node, for 3 hours. We allocate these resources via the qprod queue, consumed resources will be accounted to the Project identified by Project ID OPEN-0-0. Jobscript myjob will be executed on the first node in the allocation.

$ qsub -q qexp -l select=4:ncpus=16 -I

In this example, we allocate 4 nodes, 16 cores per node, for 1 hour. We allocate these resources via the qexp queue. The resources will be available interactively

$ qsub -A OPEN-0-0 -q qnvidia -l select=10:ncpus=16 ./myjob

In this example, we allocate 10 nvidia accelerated nodes, 16 cores per node, for  24 hours. We allocate these resources via the qnvidia queue. Jobscript myjob will be executed on the first node in the allocation.

$ qsub -A OPEN-0-0 -q qfree -l select=10:ncpus=16 ./myjob

In this example, we allocate 10  nodes, 16 cores per node, for 12 hours. We allocate these resources via the qfree queue. It is not required that the project OPEN-0-0 has any available resources left. Consumed resources are still accounted for. Jobscript myjob will be executed on the first node in the allocation.

All qsub options may be saved directly into the jobscript. In such a case, no options to qsub are needed.

$ qsub ./myjob

By default, the PBS batch system sends an e-mail only when the job is aborted. Disabling mail events completely can be done like this:

$ qsub -m n

Advanced job placement

Placement by name

Specific nodes may be allocated via the PBS

qsub -A OPEN-0-0 -q qprod -l select=1:ncpus=16:host=cn171+1:ncpus=16:host=cn172 -I

In this example, we allocate nodes cn171 and cn172, all 16 cores per node, for 24 hours.  Consumed resources will be accounted to the Project identified by Project ID OPEN-0-0. The resources will be available interactively.

Placement by CPU type

Nodes equipped with Intel Xeon E5-2665 CPU have base clock frequency 2.4GHz, nodes equipped with Intel Xeon E5-2470 CPU have base frequency 2.3 GHz (see section Compute Nodes for details).  Nodes may be selected via the PBS resource attribute cpu_freq .

|CPU Type|base freq.|Nodes|cpu_freq attribute| |---|---| |Intel Xeon E5-2665|2.4GHz|cn[1-180], cn[208-209]|24| |Intel Xeon E5-2470|2.3GHz|cn[181-207]|23|

$ qsub -A OPEN-0-0 -q qprod -l select=4:ncpus=16:cpu_freq=24 -I

In this example, we allocate 4 nodes, 16 cores, selecting only the nodes with Intel Xeon E5-2665 CPU.

Placement by IB switch

Groups of computational nodes are connected to chassis integrated Infiniband switches. These switches form the leaf switch layer of the Infiniband network fat tree topology. Nodes sharing the leaf switch can communicate most efficiently. Sharing the same switch prevents hops in the network and provides for unbiased, most efficient network communication.

Nodes sharing the same switch may be selected via the PBS resource attribute ibswitch. Values of this attribute are iswXX, where XX is the switch number. The node-switch mapping can be seen at Hardware Overview section.

We recommend allocating compute nodes of a single switch when best possible computational network performance is required to run the job efficiently:

    qsub -A OPEN-0-0 -q qprod -l select=18:ncpus=16:ibswitch=isw11 ./myjob

In this example, we request all the 18 nodes sharing the isw11 switch for 24 hours. Full chassis will be allocated.

Advanced job handling

Selecting Turbo Boost off

Intel Turbo Boost Technology is on by default. We strongly recommend keeping the default.