diff --git a/docs.it4i/general/vnode-allocation.md b/docs.it4i/general/vnode-allocation.md deleted file mode 100644 index 413358f7d5a2789a9c14c5355e32711dedf3f378..0000000000000000000000000000000000000000 --- a/docs.it4i/general/vnode-allocation.md +++ /dev/null @@ -1,147 +0,0 @@ -# Allocation of vnodes on qgpu - -## Introduction - -The `qgpu` queue on Karolina takes advantage of the division of nodes into vnodes. -Accelerated node equipped with two 64-core processors and eight GPU cards is treated as eight vnodes, -each containing 16 CPU cores and 1 GPU card. -Vnodes can be allocated to jobs individually –â -through precise definition of resource list at job submission, -you may allocate varying number of resources/GPU cards according to your needs. - -!!! important "Vnodes and Security" - Division of nodes into vnodes was implemented to be as secure as possible, but it is still a "multi-user mode", - which means that if two users allocate a portion of the same node, they can see each other's running processes. - If this solution is inconvenient for you, consider allocating a whole node. - -## Selection Statement and Chunks - -Requested resources are specified using a selection statement: - -``` --l select=[<N>:]<chunk>[+[<N>:]<chunk> ...] -``` - -`N` specifies the number of chunks; if not specified then `N = 1`.<br> -`chunk` declares the value of each resource in a set of resources which are to be allocated as a unit to a job. - -* `chunk` is seen by the MPI as one node. -* Multiple chunks are then seen as multiple nodes. -* Maximum chunk size is equal to the size of a full physical node (8 GPU cards, 128 cores) - -Default chunk for the `qgpu` queue is configured to contain 1 GPU card and 16 CPU cores, i.e. `ncpus=16:ngpus=1`. - -* `ncpus` specifies number of CPU cores -* `ngpus` specifies number of GPU cards - -### Allocating Single GPU - -Single GPU can be allocated in an interactive session using - -```console -qsub -q qgpu -A OPEN-00-00 -l select=1 -I -``` - -or simply - -```console -qsub -q qgpu -A OPEN-00-00 -I -``` - -In this case, the `ngpus` parameter is optional, since it defaults to `1`. -You can verify your allocation either in the PBS using the `qstat` command, -or by checking the number of allocated GPU cards in the `CUDA_VISIBLE_DEVICES` variable: - -```console -$ qstat -F json -f $PBS_JOBID | grep exec_vnode - "exec_vnode":"(acn53[0]:ncpus=16:ngpus=1)" - -$ echo $CUDA_VISIBLE_DEVICES -GPU-8772c06c-0e5e-9f87-8a41-30f1a70baa00 -``` - -The output shows that you have been allocated vnode acn53[0]. - -### Allocating Single Accelerated Node - -!!! tip "Security tip" - Allocating a whole node prevents other users from seeing your running processes. - -Single accelerated node can be allocated in an interactive session using - -```console -qsub -q qgpu -A OPEN-00-00 -l select=8 -I -``` - -Setting `select=8` automatically allocates a whole accelerated node and sets `mpiproc`. -So for `N` full nodes, set `select` to `N x 8`. -However, note that it may take some time before your jobs are executed -if the required amount of full nodes isn't available. - -### Allocating Multiple GPUs - -!!! important "Security risk" - If two users allocate a portion of the same node, they can see each other's running processes. - When required for security reasons, consider allocating a whole node. - -Again, the following examples use only the selection statement, so no additional setting is required. - -```console -qsub -q qgpu -A OPEN-00-00 -l select=2 -I -``` - -In this example two chunks will be allocated on the same node, if possible. - -```console -qsub -q qgpu -A OPEN-00-00 -l select=16 -I -``` - -This example allocates two whole accelerated nodes. - -Multiple vnodes within the same chunk can be allocated using the `ngpus` parameter. -For example, to allocate 2 vnodes in an interactive mode, run - -```console -qsub -q qgpu -A OPEN-00-00 -l select=1:ngpus=2:mpiprocs=2 -I -``` - -Remember to **set the number of `mpiprocs` equal to that of `ngpus`** to spawn an according number of MPI processes. - -To verify the correctness: - -```console -$ qstat -F json -f $PBS_JOBID | grep exec_vnode - "exec_vnode":"(acn53[0]:ncpus=16:ngpus=1+acn53[1]:ncpus=16:ngpus=1)" - -$ echo $CUDA_VISIBLE_DEVICES | tr ',' '\n' -GPU-8772c06c-0e5e-9f87-8a41-30f1a70baa00 -GPU-5e88c15c-e331-a1e4-c80c-ceb3f49c300e -``` - -The number of chunks to allocate is specified in the `select` parameter. -For example, to allocate 2 chunks, each with 4 GPUs, run - -```console -qsub -q qgpu -A OPEN-00-00 -l select=2:ngpus=4:mpiprocs=4 -I -``` - -To verify the correctness: - -```console -$ cat > print-cuda-devices.sh <<EOF -#!/bin/bash -echo \$CUDA_VISIBLE_DEVICES -EOF - -$ chmod +x print-cuda-devices.sh -$ ml OpenMPI/4.1.4-GCC-11.3.0 -$ mpirun ./print-cuda-devices.sh | tr ',' '\n' | sort | uniq -GPU-0910c544-aef7-eab8-f49e-f90d4d9b7560 -GPU-1422a1c6-15b4-7b23-dd58-af3a233cda51 -GPU-3dbf6187-9833-b50b-b536-a83e18688cff -GPU-3dd0ae4b-e196-7c77-146d-ae16368152d0 -GPU-93edfee0-4cfa-3f82-18a1-1e5f93e614b9 -GPU-9c8143a6-274d-d9fc-e793-a7833adde729 -GPU-ad06ab8b-99cd-e1eb-6f40-d0f9694601c0 -GPU-dc0bc3d6-e300-a80a-79d9-3e5373cb84c9 -``` diff --git a/mkdocs.yml b/mkdocs.yml index 1f10b2e38c39a6c49b310b815335816d1cd83afb..b3c7009f70085d611a8eb256b1fd38b0f610160e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -109,7 +109,6 @@ nav: - Job Arrays: general/job-arrays.md - HyperQueue: general/hyperqueue.md - Parallel Computing and MPI: general/karolina-mpi.md - - Vnode Allocation: general/vnode-allocation.md - Other Services: - OpenCode: general/opencode.md - Technical Information: