Skip to content
Snippets Groups Projects
job-priority.md 3.58 KiB
Newer Older
Lukáš Krupčík's avatar
Lukáš Krupčík committed
Job scheduling
Lukáš Krupčík's avatar
Lukáš Krupčík committed
==============

Job execution priority
----------------------

Lukáš Krupčík's avatar
Lukáš Krupčík committed
Scheduler gives each job an execution priority and then uses this job execution priority to select which job(s) to run.
Lukáš Krupčík's avatar
Lukáš Krupčík committed

Lukáš Krupčík's avatar
Lukáš Krupčík committed
Job execution priority on Anselm is determined by these job properties (in order of importance):
Lukáš Krupčík's avatar
Lukáš Krupčík committed

1.  queue priority
2.  fairshare priority
3.  eligible time

### Queue priority

Lukáš Krupčík's avatar
Lukáš Krupčík committed
Queue priority is priority of queue where job is queued before execution.
Lukáš Krupčík's avatar
Lukáš Krupčík committed

Lukáš Krupčík's avatar
Lukáš Krupčík committed
Queue priority has the biggest impact on job execution priority. Execution priority of jobs in higher priority queues is always greater than execution priority of jobs in lower priority queues. Other properties of job used for determining job execution priority (fairshare priority, eligible time) cannot compete with queue priority.
Lukáš Krupčík's avatar
Lukáš Krupčík committed

Queue priorities can be seen at <https://extranet.it4i.cz/anselm/queues>![external](../../img/external.png)
Lukáš Krupčík's avatar
Lukáš Krupčík committed

### Fairshare priority

Lukáš Krupčík's avatar
Lukáš Krupčík committed
Fairshare priority is priority calculated on recent usage of resources. Fairshare priority is calculated per project, all members of project share same fairshare priority. Projects with higher recent usage have lower fairshare priority than projects with lower or none recent usage.
Lukáš Krupčík's avatar
Lukáš Krupčík committed

Fairshare priority is used for ranking jobs with equal queue priority.

Fairshare priority is calculated as

Lukáš Krupčík's avatar
Lukáš Krupčík committed
![](../../img/fairshare_formula.png)
Lukáš Krupčík's avatar
Lukáš Krupčík committed

Lukáš Krupčík's avatar
Lukáš Krupčík committed
where MAX_FAIRSHARE has value 1E6, usage~Project~ is cumulated usage by all members of selected project, usage~Total~ is total usage by all users, by all projects.
Lukáš Krupčík's avatar
Lukáš Krupčík committed

Lukáš Krupčík's avatar
Lukáš Krupčík committed
Usage counts allocated corehours (ncpus*walltime). Usage is decayed, or cut in half periodically, at the interval 168 hours (one week). Jobs queued in queue qexp are not calculated to project's usage.
Lukáš Krupčík's avatar
Lukáš Krupčík committed

>Calculated usage and fairshare priority can be seen at <https://extranet.it4i.cz/anselm/projects>.![external](../../img/external.png)
Lukáš Krupčík's avatar
Lukáš Krupčík committed

Lukáš Krupčík's avatar
Lukáš Krupčík committed
Calculated fairshare priority can be also seen as Resource_List.fairshare attribute of a job.
Lukáš Krupčík's avatar
Lukáš Krupčík committed

###Eligible time

Lukáš Krupčík's avatar
Lukáš Krupčík committed
Eligible time is amount (in seconds) of eligible time job accrued while waiting to run. Jobs with higher eligible time gains higher priority.
Lukáš Krupčík's avatar
Lukáš Krupčík committed

Lukáš Krupčík's avatar
Lukáš Krupčík committed
Eligible time has the least impact on execution priority. Eligible time is used for sorting jobs with equal queue priority and fairshare priority. It is very, very difficult for eligible time to compete with fairshare priority.
Lukáš Krupčík's avatar
Lukáš Krupčík committed

Lukáš Krupčík's avatar
Lukáš Krupčík committed
Eligible time can be seen as eligible_time attribute of job.
Lukáš Krupčík's avatar
Lukáš Krupčík committed

### Formula

Job execution priority (job sort formula) is calculated as:

Lukáš Krupčík's avatar
Lukáš Krupčík committed
![](../../img/job_sort_formula.png)
Lukáš Krupčík's avatar
Lukáš Krupčík committed

### Job backfilling

Anselm cluster uses job backfilling.

Lukáš Krupčík's avatar
Lukáš Krupčík committed
Backfilling means fitting smaller jobs around the higher-priority jobs that the scheduler is going to run next, in such a way that the higher-priority jobs are not delayed. Backfilling allows us to keep resources from becoming idle when the top job (job with the highest execution priority) cannot run.
Lukáš Krupčík's avatar
Lukáš Krupčík committed

Lukáš Krupčík's avatar
Lukáš Krupčík committed
The scheduler makes a list of jobs to run in order of execution priority. Scheduler looks for smaller jobs that can fit into the usage gaps around the highest-priority jobs in the list. The scheduler looks in the prioritized list of jobs and chooses the highest-priority smaller jobs that fit. Filler jobs are run only if they will not delay the start time of top jobs.
Lukáš Krupčík's avatar
Lukáš Krupčík committed

Lukáš Krupčík's avatar
Lukáš Krupčík committed
It means, that jobs with lower execution priority can be run before jobs with higher execution priority.
Lukáš Krupčík's avatar
Lukáš Krupčík committed

!!! Note "Note"
	It is **very beneficial to specify the walltime** when submitting jobs.
Lukáš Krupčík's avatar
Lukáš Krupčík committed

Lukáš Krupčík's avatar
Lukáš Krupčík committed
Specifying more accurate walltime enables better schedulling, better execution times and better resource usage. Jobs with suitable (small) walltime could be backfilled - and overtake job(s) with higher priority.