capacity-computing.md

Capacity computing

Introduction

In many cases, it is useful to submit huge (>100+) number of computational jobs into the PBS queue system. Huge number of (small) jobs is one of the most effective ways to execute embarrassingly parallel calculations, achieving best runtime, throughput and computer utilization.

However, executing huge number of jobs via the PBS queue may strain the system. This strain may result in slow response to commands, inefficient scheduling and overall degradation of performance and user experience, for all users. For this reason, the number of jobs is limited to 100 per user, 1000 per job array

!!! Note "Note" Please follow one of the procedures below, in case you wish to schedule more than 100 jobs at a time.

• Use Job arrays when running huge number of multithread (bound to one node only) or multinode (multithread across several nodes) jobs
• Use GNU parallel when running single core jobs
• CombineGNU parallel with Job arrays when running huge number of single core jobs

Policy

1. A user is allowed to submit at most 100 jobs. Each job may be a job array.
2. The array size is at most 1000 subjobs.

Job arrays

!!! Note "Note" Huge number of jobs may be easily submitted and managed as a job array.

A job array is a compact representation of many jobs, called subjobs. The subjobs share the same job script, and have the same values for all attributes and resources, with the following exceptions:

• each subjob has a unique index, $PBS_ARRAY_INDEX
• job Identifiers of subjobs only differ by their indices
• the state of subjobs can differ (R,Q,...etc.)

All subjobs within a job array have the same scheduling priority and schedule as independent jobs. Entire job array is submitted through a single qsub command and may be managed by qdel, qalter, qhold, qrls and qsig commands as a single job.

Shared jobscript

All subjobs in job array use the very same, single jobscript. Each subjob runs its own instance of the jobscript. The instances execute different work controlled by $PBS_ARRAY_INDEX variable.

Example:

Assume we have 900 input files with name beginning with "file" (e. g. file001, ..., file900). Assume we would like to use each of these input files with program executable myprog.x, each as a separate job.

First, we create a tasklist file (or subjobs list), listing all tasks (subjobs) - all input files in our example:

$ find . -name 'file*' > tasklist

Then we create jobscript: