diff --git a/docs.it4i/general/resource_allocation_and_job_execution.md b/docs.it4i/general/resource_allocation_and_job_execution.md index f32a8506e190cfb1d9c59f19fca1ddd58f4a38f9..c3c8d4b1ff95465be77a9e1b25310acc6f19dcc1 100644 --- a/docs.it4i/general/resource_allocation_and_job_execution.md +++ b/docs.it4i/general/resource_allocation_and_job_execution.md @@ -10,11 +10,13 @@ Resources are allocated to the job in a fair-share fashion, subject to constrain * **qprod** - Production queue * **qlong** - Long queue * **qmpp** - Massively parallel queue -* **qnvidia**, **qmic**, **qfat** - Dedicated queues +* **qnvidia**, **qfat** - Dedicated queues +* **qcpu_biz**, **qgpu_biz** - Queues for commercial users +* **qcpu_eurohpc**, **qgpu_eurohpc** - Queues for EuroHPC users * **qfree** - Free resource utilization queue !!! note - Check the queue status [here][a]. + See the queue status for [Karolina][a] or [Barbora][c]. Read more on the [Resources Allocation Policy][4] page. @@ -45,5 +47,6 @@ Read more on [Capacity Computing][6] page. [5]: job-submission-and-execution.md [6]: capacity-computing.md -[a]: https://extranet.it4i.cz/rsweb/ +[a]: https://extranet.it4i.cz/rsweb/karolina/queues [b]: https://www.altair.com/pbs-works/ +[c]: https://extranet.it4i.cz/rsweb/barbora/queues diff --git a/docs.it4i/general/resources-allocation-policy.md b/docs.it4i/general/resources-allocation-policy.md index fd042855b0903ac454b9139f164b1e04b3e752f3..30c9c5e7f1895f8f8c450c293d2b123f158f7fac 100644 --- a/docs.it4i/general/resources-allocation-policy.md +++ b/docs.it4i/general/resources-allocation-policy.md @@ -15,39 +15,48 @@ Resources are allocated to jobs in a fair-share fashion, subject to constraints ### Karolina -| queue | active project | project resources | nodes | min ncpus | priority | authorization | walltime | -| --------- | -------------- | -------------------- | ------------------------------------------------------------- | --------- | -------- | ------------- | --------- | -| **qexp** | no | none required | 756 nodes, max 2 nodes per user | 128 | 150 | no | 1 / 1h | -| **qprod** | yes | > 0 | 756 nodes | 128 | 0 | no | 24 / 48h | -| **qlong** | yes | > 0 | 200 nodes, max 20 nodes per job, only non-accelerated nodes allowed | 128 | 0 | no | 72 / 144h | -| **qnvidia** | yes | > 0 | 72 nodes | 128 | 0 | yes | 24 / 48h | -| **qfat** | yes | > 0 | 1 (sdf1) | 24 | 200 | yes | 24 / 48h | -| **qfree** | yes | < 120% of allocation | 756 nodes, max 8 nodes per job | 128 | -1024 | no | 12 / 12h | -| **qviz** | yes | none required | 2 nodes (with NVIDIA® Quadro RTX™ 6000) | 8 | 150 | no | 1 / 8h | - -* **qexp**, Express queue: This queue is dedicated for testing and running very small jobs. It is not required to specify a project to enter the qexp. There are 2 nodes always reserved for this queue (w/o accelerators), a maximum 8 nodes are available via the qexp for a particular user. The nodes may be allocated on a per core basis. No special authorization is required to use the queue. Maximum runtime is 1 hour. -* **qprod**, Production queue: This queue is intended for normal production runs. It is required that active project with nonzero remaining resources is specified to enter the qprod. All nodes may be accessed via the qprod queue. Full nodes, 128 cores per node are allocated. The queue runs with medium priority and no special authorization is required to use it. Maximum runtime is 48 hours. -* **qlong**, Long queue: This queue is intended for long production runs. It is required that active project with nonzero remaining resources is specified to enter the qlong. Only 200 nodes without acceleration may be accessed via the qlong queue. Full nodes, 128 cores per node are allocated. The queue runs with medium priority and no special authorization is required to use it. Maximum runtime is 144 hours (3 \* qprod time) -* **qnvidia** Dedicated queue: This queue is dedicated to accessing the NVIDIA accelerated nodes. It is required that an active project with nonzero remaining resources is specified to enter this queue. It utilizes 8x NVIDIA A100 with 320GB HBM2 memory per node. Full nodes, 128 cores per node, are allocated. The queue runs with a very high priority. The PI needs to explicitly ask [support][a] for authorization to enter the queue for all users associated with their project. -* **qfat**, HPE Superdome Flex queue. This queue is dedicated to access the fat HPE Superdome Flex machine. The machine (sdf1) has 768 Intel® Xeon® Platinum cores at 2.9GHz and 24TB RAM. The PI needs to explicitly ask support for authorization to enter the queue for all users associated to their Project. -* **qfree**, Free resource queue: The queue qfree is intended for utilization of free resources, after a Project exhausted all its allocated computational resources. (Does not apply to DD projects by default. DD projects have to request for permission on qfree after exhaustion of computational resources.) It is required that an active project is specified to enter the queue. Consumed resources will be accounted to the Project. Access to the qfree queue is automatically removed if consumed resources exceed 120% of the resources allocated to the Project. Only 756 nodes without accelerator may be accessed from this queue. Full nodes, 128 cores per node are allocated. The queue runs with a very low priority and no special authorization is required to use it. The maximum runtime in qfree is 12 hours. -* **qviz**, Visualization queue: Intended for pre-/post-processing using OpenGL accelerated graphics. Currently when accessing the node, each user gets 8 cores of a CPU allocated, thus approximately 64 GB of RAM and 1/8 of the GPU capacity (default "chunk"). If more GPU power or RAM is required, it is recommended to allocate more chunks (with 8 cores each) up to one whole node per user, so that all 64 cores, 256 GB RAM and a whole GPU is exclusive. This is currently also the maximum allowed allocation per one user. One hour of work is allocated by default, the user may ask for 2 hours maximum. +| queue | active project | project resources | nodes | min ncpus | priority | authorization | walltime (default/max) | +| ---------------- | -------------- | -------------------- | ------------------------------------------------------------- | --------- | -------- | ------------- | ----------------------- | +| **qexp** | no | none required | 756 nodes, max 2 nodes per user | 128 | 150 | no | 1 / 1h | +| **qprod** | yes | > 0 | 756 nodes | 128 | 0 | no | 24 / 48h | +| **qlong** | yes | > 0 | 200 nodes, max 20 nodes per job, only non-accelerated nodes allowed | 128 | 0 | no | 72 / 144h | +| **qnvidia** | yes | > 0 | 72 nodes | 128 | 0 | yes | 24 / 48h | +| **qfat** | yes | > 0 | 1 (sdf1) | 24 | 200 | yes | 24 / 48h | +| **qcpu_biz** | yes | > 0 | 756 nodes | 128 | 50 | no | 24 / 48h | +| **qgpu_biz** | yes | > 0 | 72 nodes | 128 | 50 | yes | 24 / 48h | +| **qcpu_eurohpc** | yes | > 0 | 756 nodes | 128 | 50 | no | 24 / 48h | +| **qgpu_eurohpc** | yes | > 0 | 72 nodes | 128 | 50 | yes | 24 / 48h | +| **qfree** | yes | < 120% of allocation | 756 nodes, max 8 nodes per job | 128 | -1024 | no | 12 / 12h | +| **qviz** | yes | none required | 2 nodes (with NVIDIA® Quadro RTX™ 6000) | 8 | 150 | no | 1 / 8h | + +* **qexp** Express queue: This queue is dedicated for testing and running very small jobs. It is not required to specify a project to enter the qexp. There are 2 nodes always reserved for this queue (w/o accelerators), a maximum 8 nodes are available via the qexp for a particular user. The nodes may be allocated on a per core basis. No special authorization is required to use the queue. Maximum runtime is 1 hour. +* **qprod** Production queue: This queue is intended for normal production runs. It is required that active project with nonzero remaining resources is specified to enter the qprod. All nodes may be accessed via the qprod queue. Full nodes, 128 cores per node are allocated. The queue runs with medium priority and no special authorization is required to use it. Maximum runtime is 48 hours. +* **qlong** Long queue: This queue is intended for long production runs. It is required that active project with nonzero remaining resources is specified to enter the qlong. Only 200 nodes without acceleration may be accessed via the qlong queue. Full nodes, 128 cores per node are allocated. The queue runs with medium priority and no special authorization is required to use it. Maximum runtime is 144 hours (3 \* qprod time) +* **qnvidia** Dedicated queue: This queue is dedicated to accessing the NVIDIA accelerated nodes. It is required that an active project with nonzero remaining resources is specified to enter this queue. It utilizes 8x NVIDIA A100 with 320GB HBM2 memory per node. Full nodes, 128 cores per node, are allocated. The PI needs to explicitly ask [support][a] for authorization to enter the queue for all users associated with their project. +* **qfat** HPE Superdome Flex queue. This queue is dedicated to access the fat HPE Superdome Flex machine. The machine (sdf1) has 768 Intel® Xeon® Platinum cores at 2.9GHz and 24TB RAM. The PI needs to explicitly ask support for authorization to enter the queue for all users associated to their Project. +* **qcpu_biz**, **qgpu_biz** Commercial queues similar to qprod and qnvidia. These queues are reserved for commercial customers and have slightly higher priorities. +* **qcpu_eurohpc**, **qgpu_eurohpc** EuroHPC queues similar to qprod and qnvidia. These queues are reserved for EuroHPC users. +* **qfree** Free resource queue: The queue qfree is intended for utilization of free resources, after a Project exhausted all its allocated computational resources. (Does not apply to DD projects by default. DD projects have to request for permission on qfree after exhaustion of computational resources.) It is required that an active project is specified to enter the queue. Consumed resources will be accounted to the Project. Access to the qfree queue is automatically removed if consumed resources exceed 120% of the resources allocated to the Project. Only 756 nodes without accelerator may be accessed from this queue. Full nodes, 128 cores per node are allocated. The queue runs with a very low priority and no special authorization is required to use it. The maximum runtime in qfree is 12 hours. +* **qviz** Visualization queue: Intended for pre-/post-processing using OpenGL accelerated graphics. Currently when accessing the node, each user gets 8 cores of a CPU allocated, thus approximately 64 GB of RAM and 1/8 of the GPU capacity (default "chunk"). If more GPU power or RAM is required, it is recommended to allocate more chunks (with 8 cores each) up to one whole node per user, so that all 64 cores, 256 GB RAM and a whole GPU is exclusive. This is currently also the maximum allowed allocation per one user. One hour of work is allocated by default, the user may ask for 2 hours maximum. ### Barbora -| queue | active project | project resources | nodes | min ncpus | priority | authorization | walltime | -| ------- | -------------- | -------------------- | ------------------------- | --------- | -------- | ------------- | -------- | -| qexp | no | none required | 189 nodes | 36 | 150 | no | 1 h | -| qprod | yes | > 0 | 187 nodes w/o accelerator | 36 | 0 | no | 24/48 h | -| qlong | yes | > 0 | 20 nodes w/o accelerator | 36 | 0 | no | 72/144 h | -| qnvidia | yes | > 0 | 8 NVIDIA nodes | 24 | 200 | yes | 24/48 h | -| qfat | yes | > 0 | 1 fat node | 8 | 200 | yes | 24/144 h | -| qfree | yes | < 120% of allocation | 189 w/o accelerator | 36 | -1024 | no | 12 h | +| queue | active project | project resources | nodes | min ncpus | priority | authorization | walltime (default/max) | +| -------- | -------------- | -------------------- | ------------------------- | --------- | -------- | ------------- | ---------------------- | +| qexp | no | none required | 189 nodes | 36 | 150 | no | 1 / 1h | +| qprod | yes | > 0 | 187 nodes w/o accelerator | 36 | 0 | no | 24 / 48h | +| qlong | yes | > 0 | 20 nodes w/o accelerator | 36 | 0 | no | 72 / 144h | +| qnvidia | yes | > 0 | 8 NVIDIA nodes | 24 | 0 | yes | 24 / 48h | +| qfat | yes | > 0 | 1 fat node | 8 | 200 | yes | 24 / 144h | +| qcpu_biz | yes | > 0 | 187 nodes w/o accelerator | 36 | 50 | no | 24 / 48h | +| qgpu_biz | yes | > 0 | 8 NVIDIA nodes | 24 | 50 | yes | 24 / 48h | +| qfree | yes | < 120% of allocation | 189 w/o accelerator | 36 | -1024 | no | 12 / 12h | * **qexp**, Express queue: This queue is dedicated for testing and running very small jobs. It is not required to specify a project to enter the qexp. There are 2 nodes always reserved for this queue (w/o accelerators), a maximum 8 nodes are available via the qexp for a particular user. The nodes may be allocated on a per core basis. No special authorization is required to use the queue. The maximum runtime in qexp is 1 hour. * **qprod**, Production queue: This queue is intended for normal production runs. It is required that an active project with nonzero remaining resources is specified to enter the qprod. All nodes may be accessed via the qprod queue, except the reserved ones. 187 nodes without accelerators are included. Full nodes, 36 cores per node, are allocated. The queue runs with medium priority and no special authorization is required to use it. The maximum runtime in qprod is 48 hours. * **qlong**, Long queue: This queue is intended for long production runs. It is required that an active project with nonzero remaining resources is specified to enter the qlong. Only 20 nodes without acceleration may be accessed via the qlong queue. Full nodes, 36 cores per node, are allocated. The queue runs with medium priority and no special authorization is required to use it. The maximum runtime in qlong is 144 hours (three times that of the standard qprod time - 3 x 48 h). -* **qnvidia**, **qfat**, Dedicated queues: The queue qnvidia is dedicated to accessing the NVIDIA accelerated nodes and qfat the Fat nodes. It is required that an active project with nonzero remaining resources is specified to enter these queues. Included are 8 NVIDIA (4 NVIDIA cards per node) and 1 fat nodes. Full nodes, 24 cores per node, are allocated. The queues run with very high priority. The PI needs to explicitly ask [support][a] for authorization to enter the dedicated queues for all users associated with their project. +* **qnvidia**, **qfat**, Dedicated queues: The queue qnvidia is dedicated to accessing the NVIDIA accelerated nodes and qfat the Fat nodes. It is required that an active project with nonzero remaining resources is specified to enter these queues. Included are 8 NVIDIA (4 NVIDIA cards per node) and 1 fat nodes. Full nodes, 24 cores per node, are allocated. qfat runs with very high priority. The PI needs to explicitly ask [support][a] for authorization to enter the dedicated queues for all users associated with their project. +* **qcpu_biz**, **qgpu_biz** Commercial queues similar to qprod and qnvidia: These queues are reserved for commercial customers and have slightly higher priorities. * **qfree**, Free resource queue: The queue qfree is intended for utilization of free resources, after a project has exhausted all of its allocated computational resources (Does not apply to DD projects by default; DD projects have to request permission to use qfree after exhaustion of computational resources). It is required that an active project is specified to enter the queue. Consumed resources will be accounted to the Project. Access to the qfree queue is automatically removed if consumed resources exceed 120% of the resources allocated to the Project. Only 189 nodes without accelerators may be accessed from this queue. Full nodes, 16 cores per node, are allocated. The queue runs with a very low priority and no special authorization is required to use it. The maximum runtime in qfree is 12 hours. ## Queue Notes