Slurm provides usefull information to the job via environment variables. Environment variables are available on all nodes allocated to job when accessed via Slurm supported means (srun, compatible mpirun).
Slurm provides useful information to the job via environment variables. Environment variables are available on all nodes allocated to job when accessed via Slurm supported means (srun, compatible mpirun).
See all Slurm variables
See all Slurm variables
```
```
set | grep ^SLURM
set | grep ^SLURM
```
```
...
@@ -96,12 +97,14 @@ set | grep ^SLURM
...
@@ -96,12 +97,14 @@ set | grep ^SLURM
See [Slurm srun documentation][2] for details.
See [Slurm srun documentation][2] for details.
Get job nodelist
Get job nodelist
```
```
$ echo $SLURM_JOB_NODELIST
$ echo $SLURM_JOB_NODELIST
p03-amd[01-02]
p03-amd[01-02]
```
```
Expand nodelist to list of nodes.
Expand nodelist to list of nodes.
```
```
$ scontrol show hostnames $SLURM_JOB_NODELIST
$ scontrol show hostnames $SLURM_JOB_NODELIST
p03-amd01
p03-amd01
...
@@ -139,7 +142,7 @@ $ scancel JOBID
...
@@ -139,7 +142,7 @@ $ scancel JOBID
Use `-t`, `--time` option to specify job run time limit. Default job time limit is 2 hours, maximum job time limit is 24 hours.
Use `-t`, `--time` option to specify job run time limit. Default job time limit is 2 hours, maximum job time limit is 24 hours.
Slurm supports the ability to define and schedule arbitrary resources - Generic RESources (GRES) in Slurm's terminology. We use GRES for scheduling/allocating GPUs and FPGAs.
Slurm supports the ability to define and schedule arbitrary resources - Generic RESources (GRES) in Slurm's terminology. We use GRES for scheduling/allocating GPUs and FPGAs.
!!! warning
!!! warning
Use only allocated GPUs and FPGAs. Resource separation is not enforced. If you use non-allocated resources, you can observe strange behaviour and get into troubles.
Use only allocated GPUs and FPGAs. Resource separation is not enforced. If you use non-allocated resources, you can observe strange behavior and get into troubles.
IDX in the GRES attribute specifies index/indexes of FPGA(s) (or GPUs) allocated to the job on the node. In the given example - allocated resources are fpga:xilinx_alveo_u250:1(IDX:0), we should use FPGA with index/number 0 on node p03-amd01.
IDX in the GRES attribute specifies index/indexes of FPGA(s) (or GPUs) allocated to the job on the node. In the given example - allocated resources are `fpga:xilinx_alveo_u250:1(IDX:0)`, we should use FPGA with index/number 0 on node p03-amd01.