Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
# Allocation of vnodes on qgpu
## Introduction
The `qgpu` queue on Karolina takes advantage of the division of nodes into vnodes.
Accelerated node equipped with two 64-core processors and eight GPU cards is treated as eight vnodes,
each containing 16 CPU cores and 1 GPU card.
Vnodes can be allocated to jobs individually –
through precise definition of resource list at job submission,
you may allocate varying number of resources/GPU cards according to your needs.
!!! important "Vnodes and Security"
Division of nodes into vnodes was implemented to be as secure as possible, but it is still a "multi-user mode",
which means that if two users allocate a portion of the same node, they can see each other's running processes.
If this solution is inconvenient for you, consider allocating a whole node.
## Selection Statement and Chunks
Requested resources are specified using a selection statement:
```
-l select=[<N>:]<chunk>[+[<N>:]<chunk> ...]
```
`N` specifies the number of chunks; if not specified then `N = 1`.<br>
`chunk` declares the value of each resource in a set of resources which are to be allocated as a unit to a job.
* `chunk` is seen by the MPI as one node.
* Multiple chunks are then seen as multiple nodes.
* Maximum chunk size is equal to the size of a full physical node (8 GPU cards, 128 cores)
Default chunk for the `qgpu` queue is configured to contain 1 GPU card and 16 CPU cores, i.e. `ncpus=16:ngpus=1`.
* `ncpus` specifies number of CPU cores
* `ngpus` specifies number of GPU cards
### Allocating Single GPU
Single GPU can be allocated in an interactive session using
```console
qsub -q qgpu -A OPEN-00-00 -l select=1 -I
```
or simply
```console
qsub -q qgpu -A OPEN-00-00 -I
```
In this case, the `ngpus` parameter is optional, since it defaults to `1`.
You can verify your allocation either in the PBS using the `qstat` command,
or by checking the number of allocated GPU cards in the `CUDA_VISIBLE_DEVICES` variable:
```console
$ qstat -F json -f $PBS_JOBID | grep exec_vnode
"exec_vnode":"(acn53[0]:ncpus=16:ngpus=1)"
$ echo $CUDA_VISIBLE_DEVICES
GPU-8772c06c-0e5e-9f87-8a41-30f1a70baa00
```
The output shows that you have been allocated vnode acn53[0].
### Allocating Single Accelerated Node
!!! tip "Security tip"
Allocating a whole node prevents other users from seeing your running processes.
Single accelerated node can be allocated in an interactive session using
```console
qsub -q qgpu -A OPEN-00-00 -l select=8 -I
```
Setting `select=8` automatically allocates a whole accelerated node and sets `mpiproc`.
So for `N` full nodes, set `select` to `N x 8`.
However, note that it may take some time before your jobs are executed
if the required amount of full nodes isn't available.
### Allocating Multiple GPUs
!!! important "Security risk"
If two users allocate a portion of the same node, they can see each other's running processes.
When required for security reasons, consider allocating a whole node.
Again, the following examples use only the selection statement, so no additional setting is required.
```console
qsub -q qgpu -A OPEN-00-00 -l select=2 -I
```
In this example two chunks will be allocated on the same node, if possible.
```console
qsub -q qgpu -A OPEN-00-00 -l select=16 -I
```
This example allocates two whole accelerated nodes.
Multiple vnodes within the same chunk can be allocated using the `ngpus` parameter.
For example, to allocate 2 vnodes in an interactive mode, run
```console
qsub -q qgpu -A OPEN-00-00 -l select=1:ngpus=2:mpiprocs=2 -I
```
Remember to **set the number of `mpiprocs` equal to that of `ngpus`** to spawn an according number of MPI processes.
To verify the correctness:
```console
$ qstat -F json -f $PBS_JOBID | grep exec_vnode
"exec_vnode":"(acn53[0]:ncpus=16:ngpus=1+acn53[1]:ncpus=16:ngpus=1)"
$ echo $CUDA_VISIBLE_DEVICES | tr ',' '\n'
GPU-8772c06c-0e5e-9f87-8a41-30f1a70baa00
GPU-5e88c15c-e331-a1e4-c80c-ceb3f49c300e
```
The number of chunks to allocate is specified in the `select` parameter.
For example, to allocate 2 chunks, each with 4 GPUs, run
```console
qsub -q qgpu -A OPEN-00-00 -l select=2:ngpus=4:mpiprocs=4 -I
```
To verify the correctness:
```console
$ cat > print-cuda-devices.sh <<EOF
#!/bin/bash
echo \$CUDA_VISIBLE_DEVICES
EOF
$ chmod +x print-cuda-devices.sh
$ ml OpenMPI/4.1.4-GCC-11.3.0
$ mpirun ./print-cuda-devices.sh | tr ',' '\n' | sort | uniq
GPU-0910c544-aef7-eab8-f49e-f90d4d9b7560
GPU-1422a1c6-15b4-7b23-dd58-af3a233cda51
GPU-3dbf6187-9833-b50b-b536-a83e18688cff
GPU-3dd0ae4b-e196-7c77-146d-ae16368152d0
GPU-93edfee0-4cfa-3f82-18a1-1e5f93e614b9
GPU-9c8143a6-274d-d9fc-e793-a7833adde729
GPU-ad06ab8b-99cd-e1eb-6f40-d0f9694601c0
GPU-dc0bc3d6-e300-a80a-79d9-3e5373cb84c9
```