From ee018061dff645287e443800dcf827ad8fe8959b Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Roman=20Sl=C3=ADva?= <roman.sliva@vsb.cz>
Date: Thu, 20 Jul 2023 13:28:05 +0200
Subject: [PATCH] slurm-job-submission-and-execution.md - Back to basics
---
.../slurm-job-submission-and-execution.md | 743 ++----------------
1 file changed, 76 insertions(+), 667 deletions(-)
diff --git a/docs.it4i/general/slurm-job-submission-and-execution.md b/docs.it4i/general/slurm-job-submission-and-execution.md
index d9de1de37..3d8093a16 100644
--- a/docs.it4i/general/slurm-job-submission-and-execution.md
+++ b/docs.it4i/general/slurm-job-submission-and-execution.md
@@ -1,401 +1,12 @@
# Slurm Job Submission and Execution
-!!! Important
- Slurm scheduler is currently in use on Complementary System only.
- Starting July 19. 9AM, Slurm will be in use on the Barbora cluster also.
+## Introduction
-## Slurm Introduction
+[Slurm][1] workload manager is used to allocate and access Barbora cluster and Complementary systems resources.
-[Slurm][a] is a workload manager which facilitates access to cluster resources, and manages scheduling, starting, and executing jobs, and job monitoring. We highly encourage you have a look at the [interactive Slurm tutorial][b].
+## Getting Partitions Information
-A `man` page exists for all Slurm commands, as well as `--help` command option, which provides a brief summary of options. Slurm [documentation][c] and [man pages][d] are also available online.
-
-### Quick Overview of Common Commands
-
-| Command | Explanation |
-| :------: | :-------------------------------------------------------------------------------------------------------------------------- |
-| sinfo | View information about nodes and partitions. |
-| squeue | View information about jobs located in the scheduling queue. |
-| sacct | View information about jobs from Slurm accounting database. |
-| scontrol | View or modify jobs, nodes, partitions, reservations, and other Slurm objects. |
-| |
-| sbatch | Submit a batch job to Slurm. |
-| salloc | Run an interactive job. |
-| srun | Run parallel job. |
-| scancel | Cancel job. |
-
-### Job Submission Options
-
-Slurm provides three different commands for job submission: `salloc`, `sbatch`, and `srun`. Each of those serves a slightly different purpose;
-`salloc` is used to run interactive job - behaviour of this command on IT4Innovation's clusters might differ from behaviour at other clusters, it is alternative to PBS qsub -I command.
-`sbatch` is used to submit a batch script to Slurm, it is alternative to PBS qsub command used without -I option.
-`srun` behaviour of the command depends on whether it was run from inside of the job or from outside, from inside of the job command serves as Slurm's native way of executing MPI-enabled applications, from outside of the job command obtains appropriate allocation first and then run a specified command in parallel.
-
-## Batch Mode
-
-`sbatch` command can be used to submit a script for later execution. A batch script usually consists of three parts:
-
-* interpreter used for code execution, denoted by shebang sign (`#!`) followed by an absolute path to the interpreter
-* job options, denoted by a hash sign (`#`), a keyword for scheduler to pick upon, and submission options
-* job instructions; these contain everything an interpreter should do once the job has been initialized, including scheduler-independent environment setup
-
-Your most commonly used interpreter will be `bash`, or some other shell command language interpreter, such as `dash`, `zsh`, etc. You would then define your interpreter on the first line of your batch script like this:
-
-```shell
-#!/usr/bin/bash
-```
-
-To define Slurm job options within the batch script, use `SBATCH` keyword followed by your job option. Multiple `SBATCH` statements can be specified:
-
-```shell
-#SBATCH -A OPEN-00-00
-#SBATCH -p qcpu
-#SBATCH -N 4
-#SBATCH --ntasks-per-node 36
-```
-
-Here we asked for four nodes and for 36 tasks to be executed per node on partition qcpu using OPEN-00-00's project resources.
-
-Job instructions should contain everything you'd like your job to do; that is, every single command the job is supposed to execute:
-
-```shell
-ml OpenMPI/4.1.4-GCC-11.3.0
-
-srun hostname | uniq -c
-```
-
-Combined together, the previous examples make up a following script:
-
-```shell
-#!/usr/bin/bash
-#SBATCH -A OPEN-00-00
-#SBATCH -N 4
-#SBATCH --ntasks-per-node 36
-
-ml OpenMPI/4.1.4-GCC-11.3.0
-
-srun hostname | uniq -c
-```
-
-And by submitting it from the login node of Barbora cluster
-
-```console
-$ sbatch ./script.sh
-Submitted batch job 1511
-```
-
-we get an output file with the following contents:
-
-```console
-$ cat slurm-1511.out
- 36 cn1.barbora.it4i.cz
- 36 cn2.barbora.it4i.cz
- 36 cn3.barbora.it4i.cz
- 36 cn4.barbora.it4i.cz
-```
-
-### Quick Overview of Common Batch Job Options
-
-| Job Option | Specification |
-| :---------------: | :-------------------------------------------------- |
-| Job Name | `-J`, `--job-name=<name>` |
-| Account | `-A`, `--account=<account>` |
-| Queue / Partition | `-p`, `--partition=<partition_names>` |
-| Node Count | `-N`, `--nodes=<minnodes>[-maxnodes]` |
-| Total Task Count | `-n`, `--ntasks=<number>` |
-| Tasks Per Node\* | `--ntasks-per-node=<ntasks>` |
-| CPUs Per Task\* | `-c`, `--cpus-per-task=<ncpus>` |
-| Wall Clock Limit | `-t`, `-t <time>` |
-| Copy Environment | `--export={[ALL,]<environment_variables>|ALL|NONE}` |
-| Job Dependency | `-d`, `--dependency=<dependency_list>` |
-| Job Arrays | `-a`, `--array=<indexes>` |
-
-* When the `--ntasks-per-node` option is used in conjunction with `--ntasks`, it is treated as a maximum count of tasks per node instead. For more information, see the `man` page.
-
-* Beginning with 22.05, `srun` will not inherit the `--cpus-per-task` value requested by `salloc` or `sbatch`. It must be requested again with the call to `srun` or set with the `SRUN_CPUS_PER_TASK` environment variable if desired for the task(s).
-
-#### Job Environment Variables
-
-Slurm provides useful information to the job via environment variables. Environment variables are available on all nodes allocated to job when accessed via Slurm supported means (srun, compatible mpirun).
-
-See all Slurm variables
-
-```
-set | grep ^SLURM
-```
-
-### Useful Variables
-
-| variable name | description | example |
-| ------ | ------ | ------ |
-| SLURM_JOBID | job id of the executing job| 593 |
-| SLURM_JOB_NODELIST | nodes allocated to the job | p03-amd[01-02] |
-| SLURM_JOB_NUM_NODES | number of nodes allocated to the job | 2 |
-| SLURM_STEP_NODELIST | nodes allocated to the job step | p03-amd01 |
-| SLURM_STEP_NUM_NODES | number of nodes allocated to the job step | 1 |
-| SLURM_JOB_PARTITION | name of the partition | p03-amd |
-| SLURM_SUBMIT_DIR | submit directory | /scratch/project/open-xx-yy/work |
-
-See [Slurm srun documentation][2] for details.
-
-Get job nodelist
-
-```
-$ echo $SLURM_JOB_NODELIST
-cn[10-11]
-```
-
-Expand nodelist to list of nodes.
-
-```
-$ scontrol show hostnames $SLURM_JOB_NODELIST
-cn10
-cn11
-```
-
-#### Common Job Workflow
-
-!!! Note
- Production jobs must use the `/scratch` directory for I/O
-
-The recommended way to run production jobs is to change to the `/scratch` directory early in the jobscript, copy all inputs to `/scratch`, execute your calculations, and copy outputs back to the `/home` directory. While it is possible to prepare and submit your jobs from the `/scratch` itself, note that it is meant only for temporary data storage during job execution; **files older than 90 days may be automatically deleted**.
-
-```shell
-#!/bin/bash
-#SBATCH -J job_example
-#SBATCH -A OPEN-00-00
-#SBATCH -p qcpu
-#SBATCH -n 4
-
-cd $SLURM_SUBMIT_DIR
-
-SCRDIR="/scratch/project/open-00-00/${USER}/myjob"
-mkdir -p "${SCRDIR}"
-
-# change to scratch directory, exit on failure
-cd "${SCRDIR}" || exit 1
-
-# copy input files to scratch
-cp "${SLURM_SUBMIT_DIR}/input" .
-cp "${SLURM_SUBMIT_DIR}/mympiprog.x" .
-
-# load required module(s)
-# (Always specify the module's name and version in your script;
-# see https://docs.it4i.cz/software/modules/lmod/#loading-modules.)
-ml OpenMPI/4.1.4-GCC-11.3.0
-
-# execute the calculation
-srun ./mympiprog.x
-
-# copy output file to home
-cp output "${SLURM_SUBMIT_DIR}/"
-
-exit
-```
-
-In this example, a directory in `/home` holds the input file `input` and the `mympiprog.x` executable. We create the `myjob` directory on the `/scratch` filesystem, copy input and executable files from the `/home` directory where the `sbatch` was invoked (`$SLURM_SUBMIT_DIR`) to `/scratch`, execute the MPI program `mympiprog.x`, and copy the `output` file back to the `/home` directory. `mympiprog.x` is run with 4 tasks, allocated according to Slurm configuration.
-
-#### Selecting the Number of Nodes
-
-Some jobs may run significantly slower when the tasks are spread too thin; in that case, it may be beneficial to specify the number of nodes:
-
-```shell
-#!/bin/bash
-#SBATCH -J job_example
-#SBATCH -A OPEN-00-00
-#SBATCH -p p01-arm
-#SBATCH -n 4
-#SBATCH -N 2
-
-srun hostname | sort | uniq -c
-
-exit
-```
-
-The `-N`, or `--nodes` option takes two arguments: `minnodes`, minimum number of nodes to allocate, and `maxnodes`, maximum number of nodes to allocate. When only one number is specified, it means to allocate exectly the specified number of nodes. Output of the above script may then look something like this:
-
-```console
-$ cat slurm-1642.out
- 2 p01-arm01.cs.it4i.cz
- 2 p01-arm02.cs.it4i.cz
-```
-
-On the other hand, you may also want to allow Slurm some flexibility in scheduling your job, which may result in earlier job execution. Taking the last jobscript as an example, Slurm may not be able to satisfy a request for 4 tasks on 2 nodes. In that case, the following jobscript
-
-```shell
-#!/bin/bash
-#SBATCH -J job_example
-#SBATCH -A OPEN-00-00
-#SBATCH -p p01-arm
-#SBATCH -n 4
-#SBATCH -N 2-4
-
-srun hostname | sort | uniq -c
-
-exit
-```
-
-may run earlier than jobs asking strictly for 4 tasks on 2 nodes, because the number of required slots may be easier to satisfy by simply spreading them out across more nodes:
-
-```console
-$ cat slurm-1643.out
- 1 p01-arm01.cs.it4i.cz
- 1 p01-arm02.cs.it4i.cz
- 1 p01-arm03.cs.it4i.cz
- 1 p01-arm04.cs.it4i.cz
-```
-
-## Interactive Mode
-
-Sometimes you may want to run your job interactively, for example for debugging, running your commands one by one from the command line. Slurm provides two diferrent commands for these cases, each with a slightly different purpose: `salloc` and `srun`. Both of these take [options similar to `sbatch`][1].
-
-### Resource Allocation
-
-`salloc` requests an allocation, and once obtained, starts a shell on one of the nodes (master node). For example, you can allocate 2 nodes for 10 minutes with the following command:
-
-```console
-user@login$ salloc -N 2 -A OPEN-00-00 -p p01-arm -t 10
-salloc: Granted job allocation 1539
-salloc: Waiting for resource configuration
-salloc: Nodes p01-arm[01-02] are ready for job
-
-user@p01-arm01$
-```
-
-Here we can see that Slurm started a shell session on node `p01-arm01`. To start a parallel execution on allocated nodes, you can run:
-
-```console
-$ srun hostname
-p01-arm01.cs.it4i.cz
-p01-arm02.cs.it4i.cz
-```
-
-To finish the job, you can either use the `exit` keyword, or Ctrl+D (`^D`) control sequence:
-
-```console
-$ exit
-salloc: Relinquishing job allocation 1539
-```
-
-If your job exceeds either the specified time limit or the maximum possible time limit for the selected partition, you will instead be booted out by the scheduler with the following message:
-
-```console
-$ salloc: Job 1544 has exceeded its time limit and its allocation has been revoked.
-srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
-slurmstepd: error: *** STEP 1544.interactive ON p01-arm01 CANCELLED AT 2023-02-07T15:18:07 DUE TO TIME LIMIT ***
-exit
-$
-```
-
-### Job Submission
-
-!!! Note
- `srun` functions as Slurm's native way of executing MPI-aware applications, with several benefits over `mpirun`/`mpiexec`; it knows the exact machine configuration, job allocation, and Slurm's job environment variables. While our OpenMPI modules are compiled with Slurm support, and Intel MPI has its own support built-in, some applications use their own MPI implementation which may require you to specify additional parameters (such as number of tasks) for a job to run in the desired way. For this reason, we recommend you always try to substitute the `mpirun`/`mpiexec` commands in your job scripts with `srun`.
-
-`srun` is used to submit a job for real time execution, obtains an allocation first if necessary, and also serves as Slurm's native way of executing MPI jobs. For example, to launch a job on 2 nodes from a login node, you can run:
-
-```console
-$ srun -N 2 -p p01-arm -A OPEN-00-00 hostname
-p01-arm02.cs.it4i.cz
-p01-arm01.cs.it4i.cz
-```
-
-Similarly, you can use `srun` from within the allocation obtained via `salloc`, both utilizing either whole or part of the whole allocation:
-
-```console
-$ salloc -N 4 -p p01-arm -A OPEN-00-00
-salloc: Granted job allocation 1551
-salloc: Waiting for resource configuration
-salloc: Nodes p01-arm[01-04] are ready for job
-$ srun hostname
-p01-arm01.cs.it4i.cz
-p01-arm02.cs.it4i.cz
-p01-arm03.cs.it4i.cz
-p01-arm04.cs.it4i.cz
-$ srun -N 2 hostname
-p01-arm01.cs.it4i.cz
-p01-arm02.cs.it4i.cz
-```
-
-## Job Dependency Submission
-
-To submit dependent jobs in sequence, use the `-d`, or `--dependency` flag. `sbatch` also provides `--parsable` flag, which conveniently outputs only the job ID number, and the cluster name if present, and can thus be used to submit a large number of jobs dependent on each other in an automated way. For example, to submit 3 jobs, where the next one always depends on the previous one, we can use:
-
-```console
-$ first=$(sbatch --parsable job1.sh)
-$ second=$(sbatch --parsable --dependency=afterok:${first} job2.sh)
-$ sbatch --parsable --dependency=afterok:${second} job3.sh
-1581
-$ squeue
- JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
- 1579 p01-arm job1 user017 PD 0:00 8 (Priority)
- 1580 p01-arm job2 user017 PD 0:00 8 (Dependency)
- 1581 p01-arm job3 user017 PD 0:00 8 (Dependency)
-$ scontrol show job 1580 | grep JobState
- JobState=PENDING Reason=Dependency Dependency=afterok:1579(unfulfilled)
-$ scontrol show job 1581 | grep JobState
- JobState=PENDING Reason=Dependency Dependency=afterok:1580(unfulfilled)
-```
-
-Multiple job dependencies can be specified by using the colon (`:`) separator for job IDs, and comma (`,`) separator for different job exit statuses:
-
-```console
-$ sbatch --dependency=afterok:1585:1587,afternotok:1581
-Submitted batch job 1596
-$ scontrol show job 1596 | grep JobState
- JobState=PENDING Reason=Dependency Dependency=afterok:1585(unfulfilled),afterok:1587(unfulfilled),afternotok:1581(unfulfilled)
-```
-
-Job dependencies can also be used for submitting long running jobs for which the maximum allowed wall time of partition is insufficient. The only prerequisite is the program has a way of safely saving its progress before it actually reaches end (for example, by creating a STOPCAR file when using VASP). If the job is capable of this, we can submit a job multiple times in a loop, restarting it each time until it finishes:
-
-```console
-$ previous=$(sbatch --parsable jobstart.sh)
-$ for id in $(seq 2 6)
-> do
-> submit=$(sbatch --parsable --dependency=afterok:${previous} restart.sh)
-> previous=$submit
-> done
-$ squeue
- JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
- 1582 p01-arm jobstart user017 PD 0:00 8 (Priority)
- 1583 p01-arm restart user017 PD 0:00 8 (Dependency)
- 1584 p01-arm restart user017 PD 0:00 8 (Dependency)
- 1585 p01-arm restart user017 PD 0:00 8 (Dependency)
- 1586 p01-arm restart user017 PD 0:00 8 (Dependency)
- 1587 p01-arm restart user017 PD 0:00 8 (Dependency)
-$ for id in $(seq 1582 1587)
-> do
-> scontrol show job $id | grep JobState
-> done
- JobState=PENDING Reason=Priority Dependency=(null)
- JobState=PENDING Reason=Dependency Dependency=afterok:1582(unfulfilled)
- JobState=PENDING Reason=Dependency Dependency=afterok:1583(unfulfilled)
- JobState=PENDING Reason=Dependency Dependency=afterok:1584(unfulfilled)
- JobState=PENDING Reason=Dependency Dependency=afterok:1585(unfulfilled)
- JobState=PENDING Reason=Dependency Dependency=afterok:1586(unfulfilled)
-```
-
-Here we have two different submission scripts; one for the first iteration, and one for restarts, since it is often necessary to use a slightly altered input file in order for the job to restart from saved state. As we can see, only one of the jobs will be eligible to run at a time, since the rest of them depend on the correct exit status (zero) of the previous one.
-
-### Overview of Job Dependency Options
-
-| Job Status Format | Explanation |
-| :---------------: | :---------- |
-| after:job_id[[+time][:jobid[+time]...]] | After the specified jobs start or are cancelled and 'time' in minutes from job start or cancellation happens, this job can begin execution. If no 'time' is given then there is no delay after start or cancellation. |
-| afterany:job_id[:jobid...] | This job can begin execution after the specified jobs have terminated. |
-| afterburstbuffer:job_id[:jobid...] | This job can begin execution after the specified jobs have terminated and any associated burst buffer stage out operations have completed. |
-| aftercorr:job_id[:jobid...] | A task of this job array can begin execution after the corresponding task ID in the specified job has completed successfully (ran to completion with an exit code of zero). |
-| afternotok:job_id[:jobid...] | This job can begin execution after the specified jobs have terminated in some failed state (non-zero exit code, node failure, timed out, etc.). |
-| afterok:job_id[:jobid...] | This job can begin execution after the specified jobs have successfully executed (ran to completion with an exit code of zero). |
-| singleton | This job can begin execution after any previously launched jobs sharing the same job name and user have terminated. In other words, only one job by that name and owned by that user can be running or suspended at any point in time. In a federation, a singleton dependency must be fulfilled on all clusters unless `DependencyParameters=disable_remote_singleton` is used in `slurm.conf`. |
-
-## Job Management
-
-Apart from job submission and execution, Slurm also provides a number of commands dedicated to various job management purposes, such as viewing current state of the job queue, printing information about existing partitions and their configurations, showing job priorities and how they were arrived at, displaying accounting data and many other.
-
-### Job Partition Information
-
-To view information about job partitions and nodes, use the `sinfo` command. Useful option for quick overview of cluster is `-s`, or `--summarize`:
+Display partitions/queues
```console
$ sinfo -s
@@ -415,333 +26,131 @@ qfat up 2-00:00:00 0/1/0/1 cn201
qdgx up 2-00:00:00 0/1/0/1 cn202
qviz up 8:00:00 0/2/0/2 vizserv[1-2]
```
-Here we can see output of the `sinfo` command ran on Barbora cluster.
-`NODES(A/I/O/T)` column sumarizes node count per state, where the `A/I/O/T` stands for `allocated/idle/other/total`.
+## Getting Job Information
-`sinfo` can also report more granular information, such detailed exact node-oriented information:
+Show jobs
```console
-$ sinfo -Nel
-Mon Feb 13 10:06:54 2023
-NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON
-p00-arm01 1 p00-arm idle 64 64:1:1 260988 0 1 aarch64, none
-p01-arm01 1 p01-arm* allocated 48 48:1:1 32132 0 1 aarch64, none
-p01-arm02 1 p01-arm* allocated 48 48:1:1 32132 0 1 aarch64, none
-p01-arm03 1 p01-arm* allocated 48 48:1:1 32132 0 1 aarch64, none
-p01-arm04 1 p01-arm* allocated 48 48:1:1 32132 0 1 aarch64, none
-p01-arm05 1 p01-arm* allocated 48 48:1:1 32132 0 1 aarch64, none
-p01-arm06 1 p01-arm* allocated 48 48:1:1 32132 0 1 aarch64, none
-p01-arm07 1 p01-arm* idle 48 48:1:1 32132 0 1 aarch64, none
-p01-arm08 1 p01-arm* idle 48 48:1:1 32132 0 1 aarch64, none
-p02-intel01 1 p02-intel idle 64 64:1:1 257252 0 1 x86_64,i none
-p02-intel02 1 p02-intel idle 64 64:1:1 257252 0 1 x86_64,i none
-p03-amd01 1 p03-amd idle 64 64:1:1 257270 0 1 x86_64,a none
-p03-amd02 1 p03-amd idle 64 64:1:1 257270 0 1 x86_64,a none
-p04-edge01 1 p04-edge idle 16 16:1:1 128433 0 1 x86_64,i none
-p05-synt01 1 p05-synt idle 8 8:1:1 128303 0 1 x86_64,a none
+$ squeue --me
+ JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
+ 104 qcpu interact user R 1:48 2 cn[101-102]
```
-For more information about the `sinfo` command, its flags, and formatting options, see the manual, either by using the `man sinfo` command or [online][e].
-
-### Job Queue Information
-
-To view information about queued jobs, use the `squeue` command:
+Show job details for specific job
```console
-$ squeue
- JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
- 1556 p01-arm interact user017 R 1:07 8 p01-arm[01-08]
- 1558 p02-intel interact user018 CD 0:21 2 p02-intel[01-02]
- 1557 p03-amd interact user017 R 0:57 1 p03-amd01
+$ scontrol -d show job JOBID
```
-By default, this shows the job ID, partition, name of the job, job owner's username, job state, how long has the job been already running, number of allocated nodes, and a list of allocated nodes.
-
-To view jobs only belonging to a particular user, you can either use `--user=<username>`, or `--me`, which serves as an alias for `--user=$USER`, to shows only your jobs:
+Show job details for executing job from job session
```console
-$ squeue --me
- JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
- 1559 p01-arm interact user017 R 4:04 8 p01-arm[01-08]
- 1557 p03-amd interact user017 R 10:49 1 p03-amd01
+$ scontrol -d show job $SLURM_JOBID
```
-`squeue` also allows for printing information about specific jobs using the `--jobs` flag:
+## Running Interactive Jobs
+
+Run interactive job
```console
-$ squeue --jobs 1557,1560
- JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
- 1560 p02-intel interact user018 R 2:09 2 p02-intel[01-02]
- 1557 p03-amd interact user017 R 12:26 1 p03-amd01
+ $ salloc -A PROJECT-ID -p qcpu
```
-| `squeue -A $project`| List project's jobs. |
-| `squeue -t RUNNING` | List running jobs |
-| `squeue -t PENDING` | List pending jobs |
-
-For more information about the `squeue` command, its flags, and formatting options, see the manual, either by using the `man squeue` command or [online][f].
-
-#### Job State Codes Overview
-
-| Code | Job State | Explanation |
-| :--: | :------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| BF | BOOT_FAIL | Job terminated due to launch failure, typically due to a hardware failure (e.g. unable to boot the node or block and the job can not be requeued). |
-| CA | CANCELLED | Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated. |
-| CD | COMPLETED | Job has terminated all processes on all nodes with an exit code of zero. |
-| CF | CONFIGURING | Job has been allocated resources, but are waiting for them to become ready for use (e.g. booting). |
-| CG |COMPLETING | Job is in the process of completing. Some processes on some nodes may still be active. |
-| DL | DEADLINE | Job terminated on deadline. |
-| F | FAILED | Job terminated with non-zero exit code or other failure condition. |
-| NF | NODE_FAIL | Job terminated due to failure of one or more allocated nodes. |
-| OOM | OUT_OF_MEMORY | Job experienced out of memory error. |
-| PD | PENDING | Job is awaiting resource allocation. |
-| PR | PREEMPTED | Job terminated due to preemption. |
-| R | RUNNING | Job currently has an allocation. |
-| RD | RESV_DEL_HOLD | Job is being held after requested reservation was deleted. |
-| RF | REQUEUE_FED | Job is being requeued by a federation. |
-| RH | REQUEUE_HOLD | Held job is being requeued. |
-| RQ | REQUEUED | Completing job is being requeued. |
-| RS | RESIZING | Job is about to change size. |
-| RV | REVOKED | Sibling was removed from cluster due to other cluster starting the job. |
-| SI | SIGNALING | Job is being signaled. |
-| SE | SPECIAL_EXIT | The job was requeued in a special state. This state can be set by users, typically in EpilogSlurmctld, if the job has terminated with a particular exit value. |
-| SO | STAGE_OUT | Job is staging out files. |
-| ST | STOPPED | Job has an allocation, but execution has been stopped with SIGSTOP signal. CPUS have been retained by this job. |
-| S | SUSPENDED | Job has an allocation, but execution has been suspended and CPUs have been released for other jobs. |
-| TO | TIMEOUT | Job terminated upon reaching its time limit. |
-
-### Job Cancelation
-
-`scancel` is used to send signals or cancel jobs. For example, to cancel a specific job, you would run:
+Run interactive job, with X11 forwarding
```console
-$ squeue
- JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
- 1563 p01-arm interact user017 PD 0:00 3 (Resources)
- 1562 p01-arm interact user017 R 0:20 8 p01-arm[01-08]
- 1561 p03-amd interact user017 R 0:25 1 p03-amd01
-$ scancel 1562
-$ squeue
- JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
- 1562 p01-arm interact user017 CA 0:57 8 p01-arm[01-08]
- 1563 p01-arm interact user017 R 0:03 3 p01-arm[01-03]
- 1561 p03-amd interact user017 R 1:06 1 p03-amd01
+ $ salloc -A PROJECT-ID -p qcpu --x11
```
-To cancel multiple jobs, simply list all of their job IDs:
+!!! warning
+ Do not use `srun` for initiating interactive jobs, subsequent `srun`, `mpirun` invocations would block forever.
-```console
-$ squeue
- JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
- 1562 p01-arm interact user017 CA 0:57 8 p01-arm[01-08]
- 1564 p01-arm interact user017 PD 0:00 8 (Resources)
- 1563 p01-arm interact user017 R 3:31 3 p01-arm[01-03]
- 1561 p03-amd interact user017 R 4:34 1 p03-amd01
- 1565 p03-amd interact user017 R 0:07 1 p03-amd01
-$ scancel 1562 1563 1561 1565 1564
-$ squeue
- JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
- 1562 p01-arm interact user017 CA 0:57 8 p01-arm[01-08]
- 1563 p01-arm interact user017 CA 4:01 3 p01-arm[01-03]
- 1564 p01-arm interact user017 CA 0:00 8
- 1561 p03-amd interact user017 CA 5:04 1 p03-amd01
- 1565 p03-amd interact user017 CA 0:37 1 p03-amd01
-```
+## Running Batch Jobs
-Slurm also allows for canceling only jobs which fulfill certain criteria, for example, the partition to which they have been submitted, or their job state (or a mixture of both):
+Run batch job
```console
-$ squeue
- JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
- 1569 p01-arm interact user017 PD 0:00 3 (Resources)
- 1571 p01-arm interact user017 PD 0:00 3 (Priority)
- 1568 p01-arm interact user017 R 1:52 8 p01-arm[01-08]
- 1566 p03-amd interact user017 R 1:55 1 p03-amd01
- 1567 p03-amd interact user017 R 1:53 1 p03-amd01
- 1570 p03-amd interact user017 R 0:26 1 p03-amd01
-$ scancel --partition p03-amd
-$ squeue
- JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
- 1569 p01-arm interact user017 PD 0:00 3 (Resources)
- 1571 p01-arm interact user017 PD 0:00 3 (Priority)
- 1568 p01-arm interact user017 R 3:08 8 p01-arm[01-08]
- 1566 p03-amd interact user017 CA 2:42 1 p03-amd01
- 1567 p03-amd interact user017 CA 2:40 1 p03-amd01
- 1570 p03-amd interact user017 CA 1:13 1 p03-amd01
-$ scancel --partition p01-arm --state R
-$ squeue
- JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
- 1568 p01-arm interact user017 CA 4:29 8 p01-arm[01-08]
- 1569 p01-arm interact user017 R 0:07 3 p01-arm[01-03]
- 1571 p01-arm interact user017 R 0:07 3 p01-arm[04-06]
- 1566 p03-amd interact user017 CA 2:42 1 p03-amd01
- 1567 p03-amd interact user017 CA 2:40 1 p03-amd01
- 1570 p03-amd interact user017 CA 1:13 1 p03-amd01
+ $ sbatch ./script.sh
```
-For more information about the `scancel` command, its flags, and formatting options, see the manual, either by using the `man scancel` command or [online][g].
+```shell
+#!/usr/bin/bash
+#SBATCH -J MyJobName
+#SBATCH -A OPEN-00-00
+#SBATCH -N 4
+#SBATCH --ntasks-per-node 36
+#SBATCH -p qcpu
+#SBATCH -t 12:00:00
-### Accessing Detailed Job Information and Advanced Job Options
+ml OpenMPI/4.1.4-GCC-11.3.0
-!!! Note
- Please note that most of the `scontrol` functionality is reserved for system administrator's use only.
+srun hostname | uniq -c
+`
-`scontrol` is a program used for Slurm configuration, and as such can be used for accessing detailed job information and advanced functions, such as holding, releasing, requeueing, and suspending jobs.
+Useful command options (salloc, sbatch, srun)
-To view detailed job information, you can use `scontrol show job <job_id>`, for example:
+* -N, --nodes
+* --tasks-per-node
+* -n, --ntasks
+* -c, --cpus-per-task
-```console
-$ scontrol show job 1571
- UserId=user017(5856) GroupId=user017(6432) MCS_label=N/A
- Priority=4294901692 Nice=0 Account=user018 QOS=normal
- JobState=RUNNING Reason=None Dependency=(null)
- Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0
- DerivedExitCode=0:0
- RunTime=00:59:02 TimeLimit=02:00:00 TimeMin=N/A
- SubmitTime=2023-02-13T12:47:01 EligibleTime=2023-02-13T12:47:01
- AccrueTime=2023-02-13T12:47:01
- StartTime=2023-02-13T12:49:51 EndTime=2023-02-13T14:49:51 Deadline=N/A
- SuspendTime=None SecsPreSuspend=0 LastSchedEval=2023-02-13T12:49:51 Scheduler=Main
- Partition=p01-arm AllocNode:Sid=login:1122068
- ReqNodeList=(null) ExcNodeList=(null)
- NodeList=p01-arm[04-06]
- BatchHost=p01-arm04
- NumNodes=3 NumCPUs=144 NumTasks=3 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
- TRES=cpu=144,node=3,billing=144
- Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
- JOB_GRES=(null)
- Nodes=p01-arm[04-06] CPU_IDs=0-47 Mem=0 GRES=
- MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
- Features=(null) DelayBoot=00:00:00
- OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
- Command=(null)
- WorkDir=/home/user017
- Power=
-```
+## Slurm Job Environment Variables
-!!! Note
- `sbatch`, `salloc`, and `srun` both support the `-H`, or `--hold` flag to submit a job directly in a held state. However, they can only be released via `scontrol release <job_id>`.
+Slurm provides useful information to the job via environment variables. Environment variables are available on all nodes allocated to job when accessed via Slurm supported means (srun, compatible mpirun).
-Sometimes you may want to temporarily prevent your job from running. For this reason, you can tell Slurm to hold your job; this will temporarily change its job priority to zero. Once you are sure you want your jobs to execute again, you can tell Slurm to release them back to the queue:
+See all Slurm variables
-```console
-$ squeue
- JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
- 1572 p01-arm interact user017 PD 0:00 8 (Resources)
- 1576 p01-arm interact user017 PD 0:00 8 (Priority)
- 1569 p01-arm interact user017 R 1:14:11 3 p01-arm[01-03]
- 1571 p01-arm interact user017 R 1:14:11 3 p01-arm[04-06]
-$ scontrol hold 1572 1576
-$ squeue
- JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
- 1576 p01-arm interact user017 PD 0:00 8 (JobHeldUser)
- 1572 p01-arm interact user017 PD 0:00 8 (JobHeldUser)
- 1569 p01-arm interact user017 R 1:14:32 3 p01-arm[01-03]
- 1571 p01-arm interact user017 R 1:14:32 3 p01-arm[04-06]
-$ scontrol release 1572 1576
-$ squeue
- JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
- 1572 p01-arm interact user017 PD 0:00 8 (Resources)
- 1576 p01-arm interact user017 PD 0:00 8 (Priority)
- 1569 p01-arm interact user017 R 1:14:39 3 p01-arm[01-03]
- 1571 p01-arm interact user017 R 1:14:39 3 p01-arm[04-06]
```
-
-`scontrol` also offers an interactive mode, where you can run commands in quick succession:
-
-```console
-$ scontrol
-scontrol: hold 1572
-scontrol: release 1572
-scontrol: exit
-$
+set | grep ^SLURM
```
-For more information about the `scontrol` command, its flags and formatting options, see manual, either by using the `man scontrol` command or [online][h].
-
-## Capacity Computing
-
-Job arrays offer a mechanism for submitting and managing collections of similar jobs quicky and easily by specifying an additional `-a`, or `--array=<indexes>` parameter. Jobs have the same initial options (such as size, time limit, etc.), but also have several additional environment variables set:
-
-| Variable | Value |
-| :----------------------- | :---------------------------------------------- |
-| `SLURM_ARRAY_JOB_ID` | Equals to the first job ID of the array. |
-| `SLURM_ARRAY_TASK_ID` | Equals to the job array index value. |
-| `SLURM_ARRAY_TASK_COUNT` | Equals to the number of tasks in the job array. |
-| `SLURM_ARRAY_TASK_MAX` | Equals to the highest job array index value. |
-| `SLURM_ARRAY_TASK_MIN` | Equals to the lowest job array index value. |
-| `SLURM_ARRAY_TASK_STEP` | Equals to the step size of task IDs. |
+### Useful Variables
-Consider following batch script, `script.sh`:
+| variable name | description | example |
+| ------ | ------ | ------ |
+| SLURM_JOBID | job id of the executing job| 593 |
+| SLURM_JOB_NODELIST | nodes allocated to the job | cn[101-102] |
+| SLURM_JOB_NUM_NODES | number of nodes allocated to the job | 2 |
+| SLURM_STEP_NODELIST | nodes allocated to the job step | cn101 |
+| SLURM_STEP_NUM_NODES | number of nodes allocated to the job step | 1 |
+| SLURM_JOB_PARTITION | name of the partition | qcpu |
+| SLURM_SUBMIT_DIR | submit directory | /scratch/project/open-xx-yy/work |
-```shell
-#!/bin/sh
-#SBATCH -A OPEN-00-00
-#SBATCH -p p01-arm
-#SBATCH -N 1
+See [Slurm srun documentation][2] for details.
-set | grep -e "^SLURM_JOB_ID" -e "^SLURM_ARRAY"
+Get job nodelist
-exit
+```
+$ echo $SLURM_JOB_NODELIST
+cn[101-102]
```
-We can use it to submit 3 jobs in an array
+Expand nodelist to list of nodes.
-```shell
-sbatch --array=1-3 script.sh
+```
+$ scontrol show hostnames $SLURM_JOB_NODELIST
+cn101
+cn102
```
-which would result in 3 output files, by default called `slurm-${SLURM_ARRAY_JOB_ID}_${SLURM_ARRAY_TASK_ID}.out`
+## Modifying Jobs
-```console
-$ ls -l slurm*.out
--rw-rw-r-- 1 user017 user017 159 Feb 17 14:47 slurm-1632_1.out
--rw-rw-r-- 1 user017 user017 159 Feb 17 14:47 slurm-1632_2.out
--rw-rw-r-- 1 user017 user017 159 Feb 17 14:47 slurm-1632_3.out
+```
+$ scontrol update JobId=JOBID ATTR=VALUE
```
-with the following contents:
+for example
-```console
-$ cat slurm-1632_1.out
-SLURM_ARRAY_JOB_ID=1632
-SLURM_ARRAY_TASK_COUNT=3
-SLURM_ARRAY_TASK_ID=1
-SLURM_ARRAY_TASK_MAX=3
-SLURM_ARRAY_TASK_MIN=1
-SLURM_ARRAY_TASK_STEP=1
-SLURM_JOB_ID=1633
-
-$ cat slurm-1632_2.out
-SLURM_ARRAY_JOB_ID=1632
-SLURM_ARRAY_TASK_COUNT=3
-SLURM_ARRAY_TASK_ID=2
-SLURM_ARRAY_TASK_MAX=3
-SLURM_ARRAY_TASK_MIN=1
-SLURM_ARRAY_TASK_STEP=1
-SLURM_JOB_ID=1634
-
-$ cat slurm-1632_3.out
-SLURM_ARRAY_JOB_ID=1632
-SLURM_ARRAY_TASK_COUNT=3
-SLURM_ARRAY_TASK_ID=3
-SLURM_ARRAY_TASK_MAX=3
-SLURM_ARRAY_TASK_MIN=1
-SLURM_ARRAY_TASK_STEP=1
-SLURM_JOB_ID=1632
+```
+$ scontrol update JobId=JOBID Comment='The best job ever'
```
-For more information about job arrays, see [Job Array Support][i] section of the official Slurm documentation.
+## Deleting Jobs
-[a]: https://slurm.schedmd.com/
-[b]: http://slurmlearning.deic.dk/
-[c]: https://slurm.schedmd.com/documentation.html
-[d]: https://slurm.schedmd.com/man_index.html
-[e]: https://slurm.schedmd.com/sinfo.html
-[f]: https://slurm.schedmd.com/squeue.html
-[g]: https://slurm.schedmd.com/scancel.html
-[h]: https://slurm.schedmd.com/scontrol.html
-[i]: https://slurm.schedmd.com/job_array.html
+```
+$ scancel JOBID
+```
-[1]: #quick-overview-of-common-batch-job-options
+[1]: https://slurm.schedmd.com/
+[2]: https://slurm.schedmd.com/srun.html#SECTION_OUTPUT-ENVIRONMENT-VARIABLES
--
GitLab