diff --git a/docs.it4i/general/job-arrays.md b/docs.it4i/general/job-arrays.md index 75ccbc696791f98b300c134f8d26ba032b0cd587..5d03ba2e560a3439182529fc5fe16801ae96738c 100644 --- a/docs.it4i/general/job-arrays.md +++ b/docs.it4i/general/job-arrays.md @@ -6,7 +6,7 @@ A job array is a compact representation of many jobs called tasks. Tasks share t * job Identifiers of tasks only differ by their indices * the state of tasks can differ -All tasks within a job array have the same scheduling priority and schedule as independent jobs. An entire job array is submitted through a single `sbatch` command and may be managed by `qdel`, `qalter`, `qhold`, `qrls`, and `qsig` commands as a single job. +All tasks within a job array have the same scheduling priority and schedule as independent jobs. An entire job array is submitted through a single `sbatch` command and may be managed by `squeue`, `scancel` and `scontrol` commands as a single job. ## Shared Jobscript @@ -14,9 +14,10 @@ All tasks in a job array use the very same single jobscript. Each task runs its Example: -Assume we have 900 input files with the name of each beginning with "file" (e.g. file001, ..., file900). Assume we would like to use each of these input files with myprog.x program executable, each as a separate job. +Assume we have 900 input files with the name of each beginning with "file" (e.g. file001, ..., file900). Assume we would like to use each of these input files with myprog.x program executable, +each as a separate, single node job running 128 threats. -First, we create a tasklist file (or subjobs list), listing all tasks (subjobs) - all input files in our example: +First, we create a `tasklist` file, listing all tasks - all input files in our example: ```console $ find . -name 'file*' > tasklist @@ -26,64 +27,55 @@ Then we create a jobscript: ```bash #!/bin/bash -#PBS -A OPEN-00-00 -#PBS -q qprod -#PBS -l select=1,walltime=02:00:00 +#SBATCH -p qcpu +#SBATCH -A SERVICE +#SBATCH --nodes 1 --ntasks-per-node 1 --cpus-per-task 128 +#SBATCH -t 02:00:00 +#SBATCH -o /dev/null # change to scratch directory -SCRDIR=/scratch/project/${PBS_ACCOUNT,,}/${USER}/${PBS_JOBID} +SCRDIR=/scratch/project/$SLURM_JOB_ACCOUNT/$SLURM_JOB_USER/$SLURM_JOB_ID mkdir -p $SCRDIR cd $SCRDIR || exit # get individual tasks from tasklist with index from PBS JOB ARRAY -TASK=$(sed -n "${PBS_ARRAY_INDEX}p" $PBS_O_WORKDIR/tasklist) +TASK=$(sed -n "${SLURM_ARRAY_TASK_ID}p" $SLURM_SUBMIT_DIR/tasklist) # copy input file and executable to scratch -cp $PBS_O_WORKDIR/$TASK input -cp $PBS_O_WORKDIR/myprog.x . +cp $SLURM_SUBMIT_DIR/$TASK input +cp $SLURM_SUBMIT_DIR/myprog.x . # execute the calculation ./myprog.x < input > output # copy output file to submit directory -cp output $PBS_O_WORKDIR/$TASK.out +cp output $SLURM_SUBMIT_DIR/$TASK.out ``` -In this example, the submit directory contains the 900 input files, the myprog.x executable, and the jobscript file. As an input for each run, we take the filename of the input file from the created tasklist file. We copy the input file to the local scratch memory `/lscratch/$PBS_JOBID`, execute the myprog.x and copy the output file back to the submit directory, under the `$TASK.out` name. The myprog.x executable runs on one node only and must use threads to run in parallel. Be aware, that if the myprog.x **is not multithreaded**, then all the **jobs are run as single-thread programs in a sequential manner**. Due to the allocation of the whole node, the accounted time is equal to the usage of the whole node, while using only 1/16 of the node. - -If running a huge number of parallel multicore (in means of multinode multithread, e.g. MPI enabled) jobs is needed, then a job array approach should be used. The main difference, as compared to the previous examples using one node, is that the local scratch memory should not be used (as it is not shared between nodes) and MPI or other techniques for parallel multinode processing has to be used properly. +In this example, the submit directory contains the 900 input files, the myprog.x executable, +and the jobscript file. As an input for each run, we take the filename of the input file from the created +tasklist file. We copy the input file to a scratch directory `/scratch/project/$SLURM_JOB_ACCOUNT/$SLURM_JOB_USER/$SLURM_JOB_ID`, +execute the myprog.x and copy the output file back to the submit directory, under the `$TASK.out` name. The myprog.x executable runs on one node only and must use threads to run in parallel. +Be aware, that if the myprog.x **is not multithreaded or multi-process (MPI)**, then all the **jobs are run as single-thread programs, wasting node resources**. ## Submiting Job Array -To submit the job array, use the `qsub -J` command. The 900 jobs of the [example above][3] may be submitted like this: - -```console -$ qsub -N JOBNAME -J 1-900 jobscript -506493[].isrv5 -``` - -In this example, we submit a job array of 900 subjobs. Each subjob will run on one full node and is assumed to take less than 2 hours (note the #PBS directives in the beginning of the jobscript file, do not forget to set your valid PROJECT_ID and desired queue). - -Sometimes for testing purposes, you may need to submit a one-element only array. This is not allowed by PBSPro, but there is a workaround: +To submit the job array, use the `sbatch --array` command. The 900 jobs of the [example above][3] may be submitted like this: ```console -$ qsub -N JOBNAME -J 9-10:2 jobscript +$ sbatch -J JOBNAME --array 1-900 ./jobscript ``` -This will only choose the lower index (9 in this example) for submitting/running your job. +In this example, we submit a job array of 900 tasks. Each task will run on one full node and is assumed to take less than 2 hours (note the #SBATCH directives in the beginning of the jobscript file, do not forget to set your valid PROJECT_ID and desired queue). ## Managing Job Array -Check status of the job array using the `qstat` command. +Check status of the job array using the `squeue --me` command, alternatively `squeue --me --array`. ```console -$ qstat -a 12345[].dm2 - -dm2: - Req'd Req'd Elap -Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time ---------------- -------- -- |---|---| ------ --- --- ------ ----- - ----- -12345[].dm2 user2 qprod xx 13516 1 16 -- 00:50 B 00:02 +$ squeue --me --long + JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON) +2499924_[5-101] qcpu myarray jansik PENDING 0:00 1:00 1 (Resources) ``` When the status is B, it means that some subjobs are already running.