Skip to content
Snippets Groups Projects
resources-allocation-policy.md 8.57 KiB
Newer Older
# Resources Allocation Policy
David Hrbáč's avatar
David Hrbáč committed
## Job Queue Policies
Lukáš Krupčík's avatar
Lukáš Krupčík committed

The resources are allocated to the job in a fair-share fashion, subject to constraints set by the queue and resources available to the Project. The Fair-share at Anselm ensures that individual users may consume approximately equal amount of resources per week. Detailed information in the [Job scheduling](job-priority/) section. The resources are accessible via several queues for queueing the jobs. The queues provide prioritized and exclusive access to the computational resources. Following table provides the queue partitioning overview:
Lukáš Krupčík's avatar
Lukáš Krupčík committed

David Hrbáč's avatar
David Hrbáč committed
!!! note
David Hrbáč's avatar
David Hrbáč committed
    Check the queue status at <https://extranet.it4i.cz/anselm/>
David Hrbáč's avatar
David Hrbáč committed
| queue               | active project | project resources | nodes                                                | min ncpus | priority | authorization | walltime |
Lukáš Krupčík's avatar
Lukáš Krupčík committed
| ------------------- | -------------- | ----------------- | ---------------------------------------------------- | --------- | -------- | ------------- | -------- |
David Hrbáč's avatar
David Hrbáč committed
| qexp                | no             | none required     | 2 reserved, 31 totalincluding MIC, GPU and FAT nodes | 1         | 150      | no            | 1 h      |
| qprod               | yes            | 0                 | 178 nodes w/o accelerator                            | 16        | 0        | no            | 24/48 h  |
| qlong               | yes            | 0                 | 60 nodes w/o accelerator                             | 16        | 0        | no            | 72/144 h |
| qnvidia, qmic       | yes            | 0                 | 23 nvidia nodes, 4 mic nodes                         | 16        | 200      | yes           | 24/48 h  |
| qfat                | yes            | 0                 | 2 fat nodes                                          | 16        | 200      | yes           | 24/144 h |
David Hrbáč's avatar
David Hrbáč committed
| qfree               | yes            | none required     | 178 w/o accelerator                                  | 16        | -1024    | no            | 12 h     |
Lukáš Krupčík's avatar
Lukáš Krupčík committed

David Hrbáč's avatar
David Hrbáč committed
!!! note
Lukáš Krupčík's avatar
Lukáš Krupčík committed
 **The qfree queue is not free of charge**. [Normal accounting](#resources-accounting-policy) applies. However, it allows for utilization of free resources, once a Project exhausted all its allocated computational resources. This does not apply for Directors Discreation's projects (DD projects) by default. Usage of qfree after exhaustion of DD projects computational resources is allowed after request for this queue.
Lukáš Krupčík's avatar
Lukáš Krupčík committed

Lukáš Krupčík's avatar
Lukáš Krupčík committed
**The qexp queue is equipped with the nodes not having the very same CPU clock speed.** Should you need the very same CPU speed, you have to select the proper nodes during the PSB job submission.
Lukáš Krupčík's avatar
Lukáš Krupčík committed

Lukáš Krupčík's avatar
Lukáš Krupčík committed
* **qexp**, the Express queue: This queue is dedicated for testing and running very small jobs. It is not required to specify a project to enter the qexp. There are 2 nodes always reserved for this queue (w/o accelerator), maximum 8 nodes are available via the qexp for a particular user, from a pool of nodes containing Nvidia accelerated nodes (cn181-203), MIC accelerated nodes (cn204-207) and Fat nodes with 512GB RAM (cn208-209). This enables to test and tune also accelerated code or code with higher RAM requirements. The nodes may be allocated on per core basis. No special authorization is required to use it. The maximum runtime in qexp is 1 hour.
* **qprod**, the Production queue: This queue is intended for normal production runs. It is required that active project with nonzero remaining resources is specified to enter the qprod. All nodes may be accessed via the qprod queue, except the reserved ones. 178 nodes without accelerator are included. Full nodes, 16 cores per node are allocated. The queue runs with medium priority and no special authorization is required to use it. The maximum runtime in qprod is 48 hours.
* **qlong**, the Long queue: This queue is intended for long production runs. It is required that active project with nonzero remaining resources is specified to enter the qlong. Only 60 nodes without acceleration may be accessed via the qlong queue. Full nodes, 16 cores per node are allocated. The queue runs with medium priority and no special authorization is required to use it. The maximum runtime in qlong is 144 hours (three times of the standard qprod time - 3 x 48 h).
Lukáš Krupčík's avatar
Lukáš Krupčík committed
* **qnvidia**, **qmic**, **qfat**, the Dedicated queues: The queue qnvidia is dedicated to access the Nvidia accelerated nodes, the qmic to access MIC nodes and qfat the Fat nodes. It is required that active project with nonzero remaining resources is specified to enter these queues. 23 nvidia, 4 mic and 2 fat nodes are included. Full nodes, 16 cores per node are allocated. The queues run with very high priority, the jobs will be scheduled before the jobs coming from the qexp queue. An PI needs explicitly ask [support](https://support.it4i.cz/rt/) for authorization to enter the dedicated queues for all users associated to her/his Project.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
* **qfree**, The Free resource queue: The queue qfree is intended for utilization of free resources, after a Project exhausted all its allocated computational resources (Does not apply to DD projects by default. DD projects have to request for persmission on qfree after exhaustion of computational resources.). It is required that active project is specified to enter the queue, however no remaining resources are required. Consumed resources will be accounted to the Project. Only 178 nodes without accelerator may be accessed from this queue. Full nodes, 16 cores per node are allocated. The queue runs with very low priority and no special authorization is required to use it. The maximum runtime in qfree is 12 hours.
Lukáš Krupčík's avatar
Lukáš Krupčík committed

David Hrbáč's avatar
David Hrbáč committed
## Queue Notes
Lukáš Krupčík's avatar
Lukáš Krupčík committed

David Hrbáč's avatar
David Hrbáč committed
The job wall clock time defaults to **half the maximum time**, see table above. Longer wall time limits can be  [set manually, see examples](job-submission-and-execution/).
Lukáš Krupčík's avatar
Lukáš Krupčík committed

Lukáš Krupčík's avatar
Lukáš Krupčík committed
Jobs that exceed the reserved wall clock time (Req'd Time) get killed automatically. Wall clock time limit can be changed for queuing jobs (state Q) using the qalter command, however can not be changed for a running job (state R).
Lukáš Krupčík's avatar
Lukáš Krupčík committed

Anselm users may check current queue configuration at <https://extranet.it4i.cz/anselm/queues>.
Lukáš Krupčík's avatar
Lukáš Krupčík committed

David Hrbáč's avatar
David Hrbáč committed
## Queue Status
Lukáš Krupčík's avatar
Lukáš Krupčík committed

Lukáš Krupčík's avatar
Lukáš Krupčík committed
!!! tip
    Check the status of jobs, queues and compute nodes at <https://extranet.it4i.cz/anselm/>
Lukáš Krupčík's avatar
Lukáš Krupčík committed

Pavel Jirásek's avatar
Pavel Jirásek committed
![rspbs web interface](../img/rsweb.png)
Lukáš Krupčík's avatar
Lukáš Krupčík committed

Display the queue status on Anselm:

Lukáš Krupčík's avatar
Lukáš Krupčík committed
```console
Lukáš Krupčík's avatar
Lukáš Krupčík committed
$ qstat -q
Lukáš Krupčík's avatar
Lukáš Krupčík committed
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed

Lukáš Krupčík's avatar
Lukáš Krupčík committed
The PBS allocation overview may be obtained also using the rspbs command.
Lukáš Krupčík's avatar
Lukáš Krupčík committed

Lukáš Krupčík's avatar
Lukáš Krupčík committed
```console
Lukáš Krupčík's avatar
Lukáš Krupčík committed
$ rspbs
Usage: rspbs [options]

Options:
David Hrbáč's avatar
David Hrbáč committed
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  --get-node-ncpu-chart
                        Print chart of allocated ncpus per node
  --summary             Print summary
Lukáš Krupčík's avatar
Lukáš Krupčík committed
  --get-server-details Print server
David Hrbáč's avatar
David Hrbáč committed
  --get-queues          Print queues
Lukáš Krupčík's avatar
Lukáš Krupčík committed
  --get-queues-details Print queues details
David Hrbáč's avatar
David Hrbáč committed
  --get-reservations    Print reservations
  --get-reservations-details
                        Print reservations details
  --get-nodes           Print nodes of PBS complex
  --get-nodeset         Print nodeset of PBS complex
  --get-nodes-details   Print nodes details
  --get-jobs            Print jobs
  --get-jobs-details    Print jobs details
  --get-jobs-check-params
                        Print jobid, job state, session_id, user, nodes
  --get-users           Print users of jobs
  --get-allocated-nodes
                        Print allocated nodes of jobs
  --get-allocated-nodeset
                        Print allocated nodeset of jobs
  --get-node-users      Print node users
  --get-node-jobs       Print node jobs
  --get-node-ncpus      Print number of ncpus per node
  --get-node-allocated-ncpus
                        Print number of allocated ncpus per node
  --get-node-qlist      Print node qlist
  --get-node-ibswitch   Print node ibswitch
  --get-user-nodes      Print user nodes
  --get-user-nodeset    Print user nodeset
  --get-user-jobs       Print user jobs
  --get-user-jobc       Print number of jobs per user
  --get-user-nodec      Print number of allocated nodes per user
  --get-user-ncpus      Print number of allocated ncpus per user
  --get-qlist-nodes     Print qlist nodes
  --get-qlist-nodeset   Print qlist nodeset
Lukáš Krupčík's avatar
Lukáš Krupčík committed
  --get-ibswitch-nodes Print ibswitch nodes
David Hrbáč's avatar
David Hrbáč committed
  --get-ibswitch-nodeset
                        Print ibswitch nodeset
  --state=STATE         Only for given job state
  --jobid=JOBID         Only for given job ID
  --user=USER           Only for given user
  --node=NODE           Only for given node
  --nodestate=NODESTATE
                        Only for given node state (affects only --get-node*
                        --get-qlist-* --get-ibswitch-* actions)
  --incl-finished       Include finished jobs
Lukáš Krupčík's avatar
Lukáš Krupčík committed
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed

---8<--- "resource_accounting.md"
Lukáš Krupčík's avatar
Lukáš Krupčík committed

---8<--- "mathjax.md"