job-priority.md

# Job Scheduling

## Job Priority

The scheduler gives each job a priority and then uses this job priority to select which job(s) to run.

Job priority is determined by these job properties (in order of importance):

1. queue priority
1. fair-share priority
1. job age/eligible time

### Queue Priority

Queue priority is the priority of the queue in which the job is waiting prior to execution.

Queue priority has the biggest impact on job priority. The priority of jobs in higher priority queues is always greater than the priority of jobs in lower priority queues. Other properties of jobs used for determining the job priority (fair-share priority, eligible time) cannot compete with queue priority.

Queue priorities can be seen [here][a].

### Fair-Share Priority

Fair-share priority is calculated based on recent usage of resources. Fair-share priority is calculated per project, i.e. all members of a project share the same fair-share priority. Projects with higher recent usage have a lower fair-share priority than projects with lower or no recent usage.

Fair-share priority is used for ranking jobs with equal queue priority.

Usage decays, halving at intervals of 7 days.

### Job Age/Eligible Time

The job age factor represents the length of time a job has been sitting in the queue and eligible to run.

Job age has the least impact on priority.

### Formula

Job priority is calculated as:

---8<--- "job_sort_formula.md"

### Job Backfilling

The scheduler uses job backfilling.

Backfilling means fitting smaller jobs around the higher-priority jobs that the scheduler is going to run next, in such a way that the higher-priority jobs are not delayed. Backfilling allows us to keep resources from becoming idle when the top job (the job with the highest priority) cannot run.

The scheduler makes a list of jobs to run in order of priority. The scheduler looks for smaller jobs that can fit into the usage gaps around the highest-priority jobs in the list. The scheduler looks in the prioritized list of jobs and chooses the highest-priority smaller jobs that fit. Filler jobs are run only if they will not delay the start time of top jobs.

This means that jobs with lower priority can be run before jobs with higher priority.

!!! note
    It is **very beneficial to specify the timelimit** when submitting jobs.

Specifying more accurate timelimit enables better scheduling, better times, and better resource usage. Jobs with suitable (small) timelimit can be backfilled - and overtake job(s) with a higher priority.

---8<--- "mathjax.md"

## Technical Details

Priorities are set using Slurm's [Multifactor Priority Plugin][1]. Current settings are as follows:

```
$ grep ^Priority /etc/slurm/slurm.conf
PriorityFlags=DEPTH_OBLIVIOUS
PriorityType=priority/multifactor
PriorityDecayHalfLife=7-0
PriorityMaxAge=14-0
PriorityWeightAge=100000
PriorityWeightFairshare=10000000
PriorityWeightPartition=1000000000
```

## Inspecting Job Priority

One can inspect job priority using `sprio` command. Job priority is in the field PRIORITY and it is comprised of PARTITION, FAIRSHARE and AGE priorities.

```
$ sprio -l -j 894782
          JOBID PARTITION     USER  ACCOUNT   PRIORITY       SITE        AGE      ASSOC  FAIRSHARE    JOBSIZE  PARTITION    QOSNAME        QOS        NICE                 TRES
         894782 qgpu         user1  service  300026688          0         17          0      26671          0  300000000     normal          0           0
```

[1]: https://slurm.schedmd.com/priority_multifactor.html

[a]: https://extranet.it4i.cz/rsweb/karolina/queues