slurm-job-submission-and-execution.md

# Slurm Job Submission and Execution

!!! Important
    Slurm scheduler is currently in use on Complementary System only.
    Starting July 19. 9AM, Slurm will be in use on the Barbora cluster also.

## Slurm Introduction

[Slurm][a] is a workload manager which facilitates access to cluster resources, and manages scheduling, starting, and executing jobs, and job monitoring. We highly encourage you have a look at the [interactive Slurm tutorial][b].

A `man` page exists for all Slurm commands, as well as `--help` command option, which provides a brief summary of options. Slurm [documentation][c] and [man pages][d] are also available online.

### Quick Overview of Common Commands

| Command | Explanation                                                                                                                 |
| :-----: | :-------------------------------------------------------------------------------------------------------------------------- |
| sinfo   | View information about nodes and partitions.                                                                                |
| squeue  | View information about jobs located in the scheduling queue.                                                                |
| sacct   | Display accounting data for all jobs and job steps in the job accounting log or Slurm database.                             |
|                                                                                                                                       |
| salloc  | Obtain  a job allocation (a set of nodes), execute a command, and then release the allocation when the command is finished. |
| sattach | Attach to a job step.                                                                                                       |
| sbatch  | Submit a batch script to Slurm.                                                                                             |
| sbcast  | Transmit a file to the nodes allocated to a job.                                                                            |
| scancel | Used to signal jobs or job steps that are under the control of Slurm.                                                       |
|
| srun    | Run parallel jobs.                                                                                                          |

### Job Submission Options

Slurm provides three different commands for job submission: `salloc`, `srun`, and `sbatch`. Each of those serves a slightly different purpose; `salloc` is used to obtain an allocation, `srun` runs parallel jobs (and obtains appropriate allocation first if necessary), and serves as Slurm's native way of executing MPI-enabled applications, and `sbatch` reads a batch script, asks for required allocation, and then executes it. While all of these may be used interchangeably, their specifics make them more useful in different scenarios:

| Command | Preferred use case                                                              |
| :-----: | :------------------------------------------------------------------------------ |
| sbatch  | Production jobs                                                                 |
| salloc  | Interactive jobs, debugging                                                     |
| srun    | Quick interactive jobs, short analysis of output data, running MPI applications |

## Batch Mode

`sbatch` command can be used to submit a script for later execution. A batch script usually consists of three parts:

* interpreter used for code execution, denoted by shebang sign (`#!`) followed by an absolute path to the interpreter
* job options, denoted by a hash sign (`#`), a keyword for scheduler to pick upon, and submission options
* job instructions; these contain everything an interpreter should do once the job has been initialized, including scheduler-independent environment setup

Your most commonly used interpreter will be `bash`, or some other shell command language interpreter, such as `dash`, `zsh`, etc. You would then define your interpreter on the first line of your batch script like this:

```shell
#!/usr/bin/bash
```

To define Slurm job options within the batch script, use `SBATCH` keyword followed by your job option. Multiple `SBATCH` statements can be specified:

```shell
#SBATCH -A OPEN-00-00
#SBATCH -p p03-amd
#SBATCH -n 4
```

Here we asked for 4 tasks in total to be executed on partition p03-amd using OPEN-00-00's project resources.

Job instructions should contain everything you'd like your job to do; that is, every single command the job is supposed to execute:

```shell
ml OpenMPI/4.1.4-GCC-11.3.0

srun hostname
```

Combined together, the previous examples make up a following script:

```shell
#!/usr/bin/bash
#SBATCH -A OPEN-00-00
#SBATCH -p p03-amd
#SBATCH -n 4

ml OpenMPI/4.1.4-GCC-11.3.0

srun hostname
```

And by submitting it from the login node of Complementary System

```console
$ sbatch ./batchscript.sh
Submitted batch job 1511
```

we get an output file with the following contents:

```console
$ cat slurm-1511.out
      1 p03-amd01.cs.it4i.cz
      3 p03-amd02.cs.it4i.cz
```

Notice that Slurm spread our job across 2 different nodes; by default, Slurm selects the number of nodes to minimize wait time before job execution. However, sometimes you may want to restrict your job to only a certain minimum or maximum number of nodes (or both). You may also require more time for your calculation to finish than the default allocated time. For an overview of such job options, see table below.

### Quick Overview of Common Batch Job Options

| Job Option        | Specification                                       |
| :---------------: | :-------------------------------------------------- |
| Job Name          | `-J`, `--job-name=<name>`                           |
| Account           | `-A`, `--account=<account>`                         |
| Queue / Partition | `-p`, `--partition=<partition_names>`               |
| Node Count        | `-N`, `--nodes=<minnodes>[-maxnodes]`               |
| Total Task Count  | `-n`, `--ntasks=<number>`                           |
| Tasks Per Node\*  | `--ntasks-per-node=<ntasks>`                        |
| CPUs Per Task\*   | `-c`, `--cpus-per-task=<ncpus>`                     |
| Wall Clock Limit  | `-t`, `-t <time>`                                   |
| Copy Environment  | `--export={[ALL,]<environment_variables>|ALL|NONE}` |
| Job Dependency    | `-d`, `--dependency=<dependency_list>`              |
| Job Arrays        | `-a`, `--array=<indexes>`                           |

* When the `--ntasks-per-node` option is used in conjunction with `--ntasks`, it is treated as a maximum count of tasks per node instead. For more information, see the `man` page.

* Beginning with 22.05, `srun` will not inherit the `--cpus-per-task` value requested by `salloc` or `sbatch`. It must be requested again with the call to `srun` or set with  the `SRUN_CPUS_PER_TASK` environment variable if desired for the task(s).

#### Job Environment Variables

Slurm exposes useful information about job to job main process (usually shell) via environment variables.
To view all of the Slurm's environment variables, use the command:

```shell
set | grep ^SLURM
```

#### Common Job Workflow

!!! Note
    Production jobs must use the `/scratch` directory for I/O

The recommended way to run production jobs is to change to the `/scratch` directory early in the jobscript, copy all inputs to `/scratch`, execute your calculations, and copy outputs back to the `/home` directory. While it is possible to prepare and submit your jobs from the `/scratch` itself, note that it is meant only for temporary data storage during job execution; **files older than 90 days may be automatically deleted**.

```shell
#!/bin/bash
#SBATCH -J job_example
#SBATCH -A OPEN-00-00
#SBATCH -p p03-amd
#SBATCH -n 4

cd $SLURM_SUBMIT_DIR

SCRDIR="/scratch/project/open-00-00/${USER}/myjob"
mkdir -p "${SCRDIR}"

# change to scratch directory, exit on failure
cd "${SCRDIR}" || exit 1

# copy input files to scratch
cp "${SLURM_SUBMIT_DIR}/input" .
cp "${SLURM_SUBMIT_DIR}/mympiprog.x" .

# load required module(s)
# (Always specify the module's name and version in your script;
#  see https://docs.it4i.cz/software/modules/lmod/#loading-modules.)
ml OpenMPI/4.1.4-GCC-11.3.0

# execute the calculation
srun ./mympiprog.x

# copy output file to home
cp output "${SLURM_SUBMIT_DIR}/"

exit
```

In this example, a directory in `/home` holds the input file `input` and the `mympiprog.x` executable. We create the `myjob` directory on the `/scratch` filesystem, copy input and executable files from the `/home` directory where the `sbatch` was invoked (`$SLURM_SUBMIT_DIR`) to `/scratch`, execute the MPI program `mympiprog.x`, and copy the `output` file back to the `/home` directory. `mympiprog.x` is run with 4 tasks, allocated according to Slurm configuration.

#### Selecting the Number of Nodes

Some jobs may run significantly slower when the tasks are spread too thin; in that case, it may be beneficial to specify the number of nodes:

```shell
#!/bin/bash
#SBATCH -J job_example
#SBATCH -A OPEN-00-00
#SBATCH -p p01-arm
#SBATCH -n 4
#SBATCH -N 2

srun hostname | sort | uniq -c

exit
```

The `-N`, or `--nodes` option takes two arguments: `minnodes`, minimum number of nodes to allocate, and `maxnodes`, maximum number of nodes to allocate. When only one number is specified, it means to allocate exectly the specified number of nodes. Output of the above batchscript may then look something like this:

```console
$ cat slurm-1642.out
      2 p01-arm01.cs.it4i.cz
      2 p01-arm02.cs.it4i.cz
```

On the other hand, you may also want to allow Slurm some flexibility in scheduling your job, which may result in earlier job execution. Taking the last jobscript as an example, Slurm may not be able to satisfy a request for 4 tasks on 2 nodes. In that case, the following jobscript

```shell
#!/bin/bash
#SBATCH -J job_example
#SBATCH -A OPEN-00-00
#SBATCH -p p01-arm
#SBATCH -n 4
#SBATCH -N 2-4

srun hostname | sort | uniq -c

exit
```

may run earlier than jobs asking strictly for 4 tasks on 2 nodes, because the number of required slots may be easier to satisfy by simply spreading them out across more nodes:

```console
$ cat slurm-1643.out
      1 p01-arm01.cs.it4i.cz
      1 p01-arm02.cs.it4i.cz
      1 p01-arm03.cs.it4i.cz
      1 p01-arm04.cs.it4i.cz
```

## Interactive Mode

Sometimes you may want to run your job interactively, for example for debugging, running your commands one by one from the command line. Slurm provides two diferrent commands for these cases, each with a slightly different purpose: `salloc` and `srun`. Both of these take [options similar to `sbatch`][1].

### Resource Allocation

`salloc` requests an allocation, and once obtained, starts a shell on one of the nodes (master node). For example, you can allocate 2 nodes for 10 minutes with the following command:

```console
user@login$ salloc -N 2 -A OPEN-00-00 -p p01-arm -t 10
salloc: Granted job allocation 1539
salloc: Waiting for resource configuration
salloc: Nodes p01-arm[01-02] are ready for job

user@p01-arm01$
```

Here we can see that Slurm started a shell session on node `p01-arm01`. To start a parallel execution on allocated nodes, you can run:

```console
$ srun hostname
p01-arm01.cs.it4i.cz
p01-arm02.cs.it4i.cz
```

To finish the job, you can either use the `exit` keyword, or Ctrl+D (`^D`) control sequence:

```console
$ exit
salloc: Relinquishing job allocation 1539
```

If your job exceeds either the specified time limit or the maximum possible time limit for the selected partition, you will instead be booted out by the scheduler with the following message:

```console
$ salloc: Job 1544 has exceeded its time limit and its allocation has been revoked.
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
slurmstepd: error: *** STEP 1544.interactive ON p01-arm01 CANCELLED AT 2023-02-07T15:18:07 DUE TO TIME LIMIT ***
exit
$
```

### Job Submission

!!! Note
    `srun` functions as Slurm's native way of executing MPI-aware applications, with several benefits over `mpirun`/`mpiexec`; it knows the exact machine configuration, job allocation, and Slurm's job environment variables. While our OpenMPI modules are compiled with Slurm support, and Intel MPI has its own support built-in, some applications use their own MPI implementation which may require you to specify additional parameters (such as number of tasks) for a job to run in the desired way. For this reason, we recommend you always try to substitute the `mpirun`/`mpiexec` commands in your job scripts with `srun`.

`srun` is used to submit a job for real time execution, obtains an allocation first if necessary, and also serves as Slurm's native way of executing MPI jobs. For example, to launch a job on 2 nodes from a login node, you can run:

```console
$ srun -N 2 -p p01-arm -A OPEN-00-00 hostname
p01-arm02.cs.it4i.cz
p01-arm01.cs.it4i.cz
```

Similarly, you can use `srun` from within the allocation obtained via `salloc`, both utilizing either whole or part of the whole allocation:

```console
$ salloc -N 4 -p p01-arm -A OPEN-00-00
salloc: Granted job allocation 1551
salloc: Waiting for resource configuration
salloc: Nodes p01-arm[01-04] are ready for job
$ srun hostname
p01-arm01.cs.it4i.cz
p01-arm02.cs.it4i.cz
p01-arm03.cs.it4i.cz
p01-arm04.cs.it4i.cz
$ srun -N 2 hostname
p01-arm01.cs.it4i.cz
p01-arm02.cs.it4i.cz
```

## Job Dependency Submission

To submit dependent jobs in sequence, use the `-d`, or `--dependency` flag. `sbatch` also provides `--parsable` flag, which conveniently outputs only the job ID number, and the cluster name if present, and can thus be used to submit a large number of jobs dependent on each other in an automated way. For example, to submit 3 jobs, where the next one always depends on the previous one, we can use:

```console
$ first=$(sbatch --parsable job1.sh)
$ second=$(sbatch --parsable --dependency=afterok:${first} job2.sh)
$ sbatch --parsable --dependency=afterok:${second} job3.sh
1581
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1579   p01-arm     job1  opr0019 PD       0:00      8 (Priority)
              1580   p01-arm     job2  opr0019 PD       0:00      8 (Dependency)
              1581   p01-arm     job3  opr0019 PD       0:00      8 (Dependency)
$ scontrol show job 1580 | grep JobState
   JobState=PENDING Reason=Dependency Dependency=afterok:1579(unfulfilled)
$ scontrol show job 1581 | grep JobState
   JobState=PENDING Reason=Dependency Dependency=afterok:1580(unfulfilled)
```

Multiple job dependencies can be specified by using the colon (`:`) separator for job IDs, and comma (`,`) separator for different job exit statuses:

```console
$ sbatch --dependency=afterok:1585:1587,afternotok:1581
Submitted batch job 1596
$ scontrol show job 1596 | grep JobState
   JobState=PENDING Reason=Dependency Dependency=afterok:1585(unfulfilled),afterok:1587(unfulfilled),afternotok:1581(unfulfilled)
```

Job dependencies can also be used for submitting long running jobs for which the maximum allowed wall time of partition is insufficient. The only prerequisite is the program has a way of safely saving its progress before it actually reaches end (for example, by creating a STOPCAR file when using VASP). If the job is capable of this, we can submit a job multiple times in a loop, restarting it each time until it finishes:

```console
$ previous=$(sbatch --parsable jobstart.sh)
$ for id in $(seq 2 6)
> do
> submit=$(sbatch --parsable --dependency=afterok:${previous} restart.sh)
> previous=$submit
> done
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1582   p01-arm jobstart  opr0019 PD       0:00      8 (Priority)
              1583   p01-arm  restart  opr0019 PD       0:00      8 (Dependency)
              1584   p01-arm  restart  opr0019 PD       0:00      8 (Dependency)
              1585   p01-arm  restart  opr0019 PD       0:00      8 (Dependency)
              1586   p01-arm  restart  opr0019 PD       0:00      8 (Dependency)
              1587   p01-arm  restart  opr0019 PD       0:00      8 (Dependency)
$ for id in $(seq 1582 1587)
> do
> scontrol show job $id | grep JobState
> done
   JobState=PENDING Reason=Priority Dependency=(null)
   JobState=PENDING Reason=Dependency Dependency=afterok:1582(unfulfilled)
   JobState=PENDING Reason=Dependency Dependency=afterok:1583(unfulfilled)
   JobState=PENDING Reason=Dependency Dependency=afterok:1584(unfulfilled)
   JobState=PENDING Reason=Dependency Dependency=afterok:1585(unfulfilled)
   JobState=PENDING Reason=Dependency Dependency=afterok:1586(unfulfilled)
```

Here we have two different submission scripts; one for the first iteration, and one for restarts, since it is often necessary to use a slightly altered input file in order for the job to restart from saved state. As we can see, only one of the jobs will be eligible to run at a time, since the rest of them depend on the correct exit status (zero) of the previous one.

### Overview of Job Dependency Options

| Job Status Format | Explanation |
| :---------------: | :---------- |
| after:job_id[[+time][:jobid[+time]...]] | After the specified jobs start or are cancelled and 'time' in minutes from job start or  cancellation happens, this job can begin execution. If no 'time' is given then there is no delay after start or cancellation. |
| afterany:job_id[:jobid...] | This job can begin execution after the specified jobs have terminated. |
| afterburstbuffer:job_id[:jobid...] | This job can begin execution after the specified jobs have terminated and any  associated  burst buffer stage out operations have completed. |
| aftercorr:job_id[:jobid...] | A  task  of  this job array can begin execution after the corresponding task ID in the specified job has completed successfully (ran to completion with an exit code of zero). |
| afternotok:job_id[:jobid...] | This job can begin execution after the specified jobs  have  terminated  in  some  failed  state (non-zero exit code, node failure, timed out, etc.). |
| afterok:job_id[:jobid...] | This job can begin execution after the specified jobs have successfully executed (ran to completion with an exit code of zero). |
| singleton | This job can begin execution after any previously launched jobs sharing the same  job  name  and user  have  terminated.  In other words, only one job by that name and owned by that user can be running or suspended at any point in time.  In a federation, a singleton dependency must be fulfilled on all clusters unless `DependencyParameters=disable_remote_singleton` is used in `slurm.conf`. |

## Job Management

Apart from job submission and execution, Slurm also provides a number of commands dedicated to various job management purposes, such as viewing current state of the job queue, printing information about existing partitions and their configurations, showing job priorities and how they were arrived at, displaying accounting data and many other.

### Job Partition Information

To view information about available job partitions, use the `sinfo` command:

```console
$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
p00-arm      up 1-00:00:00      1   idle p00-arm01
p01-arm*     up 1-00:00:00      8   idle p01-arm[01-08]
p02-intel    up 1-00:00:00      2   idle p02-intel[01-02]
p03-amd      up 1-00:00:00      2   idle p03-amd[01-02]
p04-edge     up 1-00:00:00      1   idle p04-edge01
p05-synt     up 1-00:00:00      1   idle p05-synt01
```

Here we can see output of the `sinfo` command ran on the Complementary System. By default, it shows basic node and partition configurations.

To view partition summary information, use `sinfo -s`, or `sinfo --summarize`:

```console
$ sinfo -s
PARTITION AVAIL  TIMELIMIT   NODES(A/I/O/T) NODELIST
p00-arm      up 1-00:00:00          0/1/0/1 p00-arm01
p01-arm*     up 1-00:00:00          0/8/0/8 p01-arm[01-08]
p02-intel    up 1-00:00:00          0/2/0/2 p02-intel[01-02]
p03-amd      up 1-00:00:00          0/2/0/2 p03-amd[01-02]
p04-edge     up 1-00:00:00          0/1/0/1 p04-edge01
p05-synt     up 1-00:00:00          0/1/0/1 p05-synt01
```

This lists only a partition state summary with no dedicated column for partition state. Instead, it is summarized in the `NODES(A/I/O/T)` column, where the `A/I/O/T` stands for `allocated/idle/other/total`.

`sinfo` can also report more granular information, such detailed exact node-oriented information:

```console
$ sinfo -Nel
Mon Feb 13 10:06:54 2023
NODELIST     NODES PARTITION       STATE CPUS    S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON
p00-arm01        1   p00-arm        idle 64     64:1:1 260988        0      1 aarch64, none
p01-arm01        1  p01-arm*   allocated 48     48:1:1  32132        0      1 aarch64, none
p01-arm02        1  p01-arm*   allocated 48     48:1:1  32132        0      1 aarch64, none
p01-arm03        1  p01-arm*   allocated 48     48:1:1  32132        0      1 aarch64, none
p01-arm04        1  p01-arm*   allocated 48     48:1:1  32132        0      1 aarch64, none
p01-arm05        1  p01-arm*   allocated 48     48:1:1  32132        0      1 aarch64, none
p01-arm06        1  p01-arm*   allocated 48     48:1:1  32132        0      1 aarch64, none
p01-arm07        1  p01-arm*        idle 48     48:1:1  32132        0      1 aarch64, none
p01-arm08        1  p01-arm*        idle 48     48:1:1  32132        0      1 aarch64, none
p02-intel01      1 p02-intel        idle 64     64:1:1 257252        0      1 x86_64,i none
p02-intel02      1 p02-intel        idle 64     64:1:1 257252        0      1 x86_64,i none
p03-amd01        1   p03-amd        idle 64     64:1:1 257270        0      1 x86_64,a none
p03-amd02        1   p03-amd        idle 64     64:1:1 257270        0      1 x86_64,a none
p04-edge01       1  p04-edge        idle 16     16:1:1 128433        0      1 x86_64,i none
p05-synt01       1  p05-synt        idle 8       8:1:1 128303        0      1 x86_64,a none
```

For more information about the `sinfo` command, its flags, and formatting options, see the manual, either by using the `man sinfo` command or [online][e].

### Job Queue Information

To view information about queued jobs, use the `squeue` command:

```console
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1556   p01-arm interact  opr0019  R       1:07      8 p01-arm[01-08]
              1558 p02-intel interact easybuil CD       0:21      2 p02-intel[01-02]
              1557   p03-amd interact  opr0019  R       0:57      1 p03-amd01
```

By default, this shows the job ID, partition, name of the job, job owner's username, job state, how long has the job been already running, number of allocated nodes, and a list of allocated nodes.

To view jobs only belonging to a particular user, you can either use `--user=<username>`, or `--me`, which serves as an alias for `--user=$USER`, to shows only your jobs:

```console
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1559   p01-arm interact  opr0019  R       3:37      8 p01-arm[01-08]
              1560 p02-intel interact easybuil  R       0:05      2 p02-intel[01-02]
              1557   p03-amd interact  opr0019  R      10:22      1 p03-amd01
$ squeue --me
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1559   p01-arm interact  opr0019  R       4:04      8 p01-arm[01-08]
              1557   p03-amd interact  opr0019  R      10:49      1 p03-amd01
```

`squeue` also allows for printing information about specific jobs using the `--jobs` flag:

```console
$ squeue --jobs 1557,1560
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1560 p02-intel interact easybuil  R       2:09      2 p02-intel[01-02]
              1557   p03-amd interact  opr0019  R      12:26      1 p03-amd01
```

For more information about the `squeue` command, its flags, and formatting options, see the manual, either by using the `man squeue` command or [online][f].

#### Job State Codes Overview

| Code | Job State     | Explanation                                                                                                                                                    |
| :--: | :------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| BF   | BOOT_FAIL     | Job terminated due to launch failure, typically due to a hardware failure (e.g. unable to boot the node or block and the job can not be requeued).             |
| CA   | CANCELLED     | Job was explicitly cancelled by the user or system administrator.  The job may or may not have been initiated.                                                 |
| CD   | COMPLETED     | Job has terminated all processes on all nodes with an exit code of zero.                                                                                       |
| CF   | CONFIGURING   | Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).                                                             |
| CG   |COMPLETING     | Job is in the process of completing. Some processes on some nodes may still be active.                                                                         |
| DL   | DEADLINE      | Job terminated on deadline.                                                                                                                                    |
| F    | FAILED        | Job terminated with non-zero exit code or other failure condition.                                                                                             |
| NF   | NODE_FAIL     | Job terminated due to failure of one or more allocated nodes.                                                                                                  |
| OOM  | OUT_OF_MEMORY | Job experienced out of memory error.                                                                                                                           |
| PD   | PENDING       | Job is awaiting resource allocation.                                                                                                                           |
| PR   | PREEMPTED     | Job terminated due to preemption.                                                                                                                              |
| R    | RUNNING       | Job currently has an allocation.                                                                                                                               |
| RD   | RESV_DEL_HOLD | Job is being held after requested reservation was deleted.                                                                                                     |
| RF   | REQUEUE_FED   | Job is being requeued by a federation.                                                                                                                         |
| RH   | REQUEUE_HOLD  | Held job is being requeued.                                                                                                                                    |
| RQ   | REQUEUED      | Completing job is being requeued.                                                                                                                              |
| RS   | RESIZING      | Job is about to change size.                                                                                                                                   |
| RV   | REVOKED       | Sibling was removed from cluster due to other cluster starting the job.                                                                                        |
| SI   | SIGNALING     | Job is being signaled.                                                                                                                                         |
| SE   | SPECIAL_EXIT  | The job was requeued in a special state. This state can be set by users, typically in EpilogSlurmctld, if the job has terminated with a particular exit value. |
| SO   | STAGE_OUT     | Job is staging out files.                                                                                                                                      |
| ST   | STOPPED       | Job has an allocation, but execution has been stopped with SIGSTOP signal.  CPUS have been retained by this job.                                               |
| S    | SUSPENDED     | Job has an allocation, but execution has been suspended and CPUs have been released for other jobs.                                                            |
| TO   | TIMEOUT       | Job terminated upon reaching its time limit.                                                                                                                   |

### Job Cancelation

`scancel` is used to send signals or cancel jobs. For example, to cancel a specific job, you would run:

```console
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1563   p01-arm interact  opr0019 PD       0:00      3 (Resources)
              1562   p01-arm interact  opr0019  R       0:20      8 p01-arm[01-08]
              1561   p03-amd interact  opr0019  R       0:25      1 p03-amd01
$ scancel 1562
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1562   p01-arm interact  opr0019 CA       0:57      8 p01-arm[01-08]
              1563   p01-arm interact  opr0019  R       0:03      3 p01-arm[01-03]
              1561   p03-amd interact  opr0019  R       1:06      1 p03-amd01
```

To cancel multiple jobs, simply list all of their job IDs:

```console
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1562   p01-arm interact  opr0019 CA       0:57      8 p01-arm[01-08]
              1564   p01-arm interact  opr0019 PD       0:00      8 (Resources)
              1563   p01-arm interact  opr0019  R       3:31      3 p01-arm[01-03]
              1561   p03-amd interact  opr0019  R       4:34      1 p03-amd01
              1565   p03-amd interact  opr0019  R       0:07      1 p03-amd01
$ scancel 1562 1563 1561 1565 1564
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1562   p01-arm interact  opr0019 CA       0:57      8 p01-arm[01-08]
              1563   p01-arm interact  opr0019 CA       4:01      3 p01-arm[01-03]
              1564   p01-arm interact  opr0019 CA       0:00      8
              1561   p03-amd interact  opr0019 CA       5:04      1 p03-amd01
              1565   p03-amd interact  opr0019 CA       0:37      1 p03-amd01
```

Slurm also allows for canceling only jobs which fulfill certain criteria, for example, the partition to which they have been submitted, or their job state (or a mixture of both):

```console
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1569   p01-arm interact  opr0019 PD       0:00      3 (Resources)
              1571   p01-arm interact  opr0019 PD       0:00      3 (Priority)
              1568   p01-arm interact  opr0019  R       1:52      8 p01-arm[01-08]
              1566   p03-amd interact  opr0019  R       1:55      1 p03-amd01
              1567   p03-amd interact  opr0019  R       1:53      1 p03-amd01
              1570   p03-amd interact  opr0019  R       0:26      1 p03-amd01
$ scancel --partition p03-amd
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1569   p01-arm interact  opr0019 PD       0:00      3 (Resources)
              1571   p01-arm interact  opr0019 PD       0:00      3 (Priority)
              1568   p01-arm interact  opr0019  R       3:08      8 p01-arm[01-08]
              1566   p03-amd interact  opr0019 CA       2:42      1 p03-amd01
              1567   p03-amd interact  opr0019 CA       2:40      1 p03-amd01
              1570   p03-amd interact  opr0019 CA       1:13      1 p03-amd01
$ scancel --partition p01-arm --state R
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1568   p01-arm interact  opr0019 CA       4:29      8 p01-arm[01-08]
              1569   p01-arm interact  opr0019  R       0:07      3 p01-arm[01-03]
              1571   p01-arm interact  opr0019  R       0:07      3 p01-arm[04-06]
              1566   p03-amd interact  opr0019 CA       2:42      1 p03-amd01
              1567   p03-amd interact  opr0019 CA       2:40      1 p03-amd01
              1570   p03-amd interact  opr0019 CA       1:13      1 p03-amd01
```

For more information about the `scancel` command, its flags, and formatting options, see the manual, either by using the `man scancel` command or [online][g].

### Accessing Detailed Job Information and Advanced Job Options

!!! Note
    Please note that most of the `scontrol` functionality is reserved for system administrator's use only.

`scontrol` is a program used for Slurm configuration, and as such can be used for accessing detailed job information and advanced functions, such as holding, releasing, requeueing, and suspending jobs.

To view detailed job information, you can use `scontrol show job <job_id>`, for example:

```console
$ scontrol show job 1571
   UserId=opr0019(5856) GroupId=opr0019(6432) MCS_label=N/A
   Priority=4294901692 Nice=0 Account=easybuild QOS=normal
   JobState=RUNNING Reason=None Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0
   DerivedExitCode=0:0
   RunTime=00:59:02 TimeLimit=02:00:00 TimeMin=N/A
   SubmitTime=2023-02-13T12:47:01 EligibleTime=2023-02-13T12:47:01
   AccrueTime=2023-02-13T12:47:01
   StartTime=2023-02-13T12:49:51 EndTime=2023-02-13T14:49:51 Deadline=N/A
   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2023-02-13T12:49:51 Scheduler=Main
   Partition=p01-arm AllocNode:Sid=login:1122068
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=p01-arm[04-06]
   BatchHost=p01-arm04
   NumNodes=3 NumCPUs=144 NumTasks=3 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   TRES=cpu=144,node=3,billing=144
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   JOB_GRES=(null)
     Nodes=p01-arm[04-06] CPU_IDs=0-47 Mem=0 GRES=
   MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
   Command=(null)
   WorkDir=/home/opr0019
   Power=
```

!!! Note
    `sbatch`, `salloc`, and `srun` both support the `-H`, or `--hold` flag to submit a job directly in a held state. However, they can only be released via `scontrol release <job_id>`.

Sometimes you may want to temporarily prevent your job from running. For this reason, you can tell Slurm to hold your job; this will temporarily change its job priority to zero. Once you are sure you want your jobs to execute again, you can tell Slurm to release them back to the queue:

```console
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1572   p01-arm interact  opr0019 PD       0:00      8 (Resources)
              1576   p01-arm interact  opr0019 PD       0:00      8 (Priority)
              1569   p01-arm interact  opr0019  R    1:14:11      3 p01-arm[01-03]
              1571   p01-arm interact  opr0019  R    1:14:11      3 p01-arm[04-06]
$ scontrol hold 1572 1576
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1576   p01-arm interact  opr0019 PD       0:00      8 (JobHeldUser)
              1572   p01-arm interact  opr0019 PD       0:00      8 (JobHeldUser)
              1569   p01-arm interact  opr0019  R    1:14:32      3 p01-arm[01-03]
              1571   p01-arm interact  opr0019  R    1:14:32      3 p01-arm[04-06]
$ scontrol release 1572 1576
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1572   p01-arm interact  opr0019 PD       0:00      8 (Resources)
              1576   p01-arm interact  opr0019 PD       0:00      8 (Priority)
              1569   p01-arm interact  opr0019  R    1:14:39      3 p01-arm[01-03]
              1571   p01-arm interact  opr0019  R    1:14:39      3 p01-arm[04-06]
```

`scontrol` also offers an interactive mode, where you can run commands in quick succession:

```console
$ scontrol
scontrol: hold 1572
scontrol: release 1572
scontrol: exit
$
```

For more information about the `scontrol` command, its flags and formatting options, see manual, either by using the `man scontrol` command or [online][h].

## Capacity Computing

Job arrays offer a mechanism for submitting and managing collections of similar jobs quicky and easily by specifying an additional `-a`, or `--array=<indexes>` parameter. Jobs have the same initial options (such as size, time limit, etc.), but also have several additional environment variables set:

| Variable                 | Value                                           |
| :----------------------- | :---------------------------------------------- |
| `SLURM_ARRAY_JOB_ID`     | Equals to the first job ID of the array.        |
| `SLURM_ARRAY_TASK_ID`    | Equals to the job array index value.            |
| `SLURM_ARRAY_TASK_COUNT` | Equals to the number of tasks in the job array. |
| `SLURM_ARRAY_TASK_MAX`   | Equals to the highest job array index value.    |
| `SLURM_ARRAY_TASK_MIN`   | Equals to the lowest job array index value.     |
| `SLURM_ARRAY_TASK_STEP`  | Equals to the step size of task IDs.            |

Consider following batch script, `batchscript.sh`:

```shell
#!/bin/sh
#SBATCH -A OPEN-00-00
#SBATCH -p p01-arm
#SBATCH -N 1

set | grep -e "^SLURM_JOB_ID" -e "^SLURM_ARRAY"

exit
```

We can use it to submit 3 jobs in an array

```shell
sbatch --array=1-3 batchscript.sh
```

which would result in 3 output files, by default called `slurm-${SLURM_ARRAY_JOB_ID}_${SLURM_ARRAY_TASK_ID}.out`

```console
$ ls -l slurm*.out
-rw-rw-r-- 1 opr0019 opr0019 159 Feb 17 14:47 slurm-1632_1.out
-rw-rw-r-- 1 opr0019 opr0019 159 Feb 17 14:47 slurm-1632_2.out
-rw-rw-r-- 1 opr0019 opr0019 159 Feb 17 14:47 slurm-1632_3.out
```

with the following contents:

```console
$ cat slurm-1632_1.out
SLURM_ARRAY_JOB_ID=1632
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_ID=1
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1
SLURM_ARRAY_TASK_STEP=1
SLURM_JOB_ID=1633

$ cat slurm-1632_2.out
SLURM_ARRAY_JOB_ID=1632
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_ID=2
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1
SLURM_ARRAY_TASK_STEP=1
SLURM_JOB_ID=1634

$ cat slurm-1632_3.out
SLURM_ARRAY_JOB_ID=1632
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_ID=3
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1
SLURM_ARRAY_TASK_STEP=1
SLURM_JOB_ID=1632
```

For more information about job arrays, see [Job Array Support][i] section of the official Slurm documentation.

[a]: https://slurm.schedmd.com/
[b]: http://slurmlearning.deic.dk/
[c]: https://slurm.schedmd.com/documentation.html
[d]: https://slurm.schedmd.com/man_index.html
[e]: https://slurm.schedmd.com/sinfo.html
[f]: https://slurm.schedmd.com/squeue.html
[g]: https://slurm.schedmd.com/scancel.html
[h]: https://slurm.schedmd.com/scontrol.html
[i]: https://slurm.schedmd.com/job_array.html

[1]: #quick-overview-of-common-batch-job-options