singularity-it4i.md 9.18 KB
Newer Older
Lukáš Krupčík's avatar
Lukáš Krupčík committed
1
2
# Singularity on IT4Innovations

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
3
On clusters, we have different versions of singularity images of operating systems. Below you can see the available operating systems singularity images.
Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
4

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
5
Our bootstrap for create image, wrappers, features are [here](https://code.it4i.cz/sccs/it4i-singularity).
Lukáš Krupčík's avatar
Lukáš Krupčík committed
6
7

```console
Lukáš Krupčík's avatar
GPU    
Lukáš Krupčík committed
8
9
10
11
   Salomon                 Anselm
      ├── CentOS             ├── CentOS
      │   ├── 6.9            │   ├── 6.9
      │   ├── 6.9-MIC        │   ├── 6.9-GPU
Lukáš Krupčík's avatar
fix MIC    
Lukáš Krupčík committed
12
13
14
15
16
17
      │   ├── 7.4            │   ├── 7.4
      │   └── 7.4-MIC        │   └── 7.4-GPU
      ├── Debian             ├── Debian
      │   └── 8.0            │   ├── 8.0
      └── Ubuntu             │   └── 8.0-GPU
          └── 16.04          └── Ubuntu
Lukáš Krupčík's avatar
GPU    
Lukáš Krupčík committed
18
19
                                 ├── 16.04
                                 └── 16.04-GPU
Lukáš Krupčík's avatar
Lukáš Krupčík committed
20
21
```

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
22
For current information on available singularity images, refer to the `ml av` and see statement the `OS`
Lukáš Krupčík's avatar
Lukáš Krupčík committed
23

Lukáš Krupčík's avatar
Lukáš Krupčík committed
24
## Available Operating Systems Images
Lukáš Krupčík's avatar
Lukáš Krupčík committed
25
26
27
28
29

* CentOS
* Debian
* Ubuntu

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
30
!!! note
Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
31
    We support graphics cards for Anselm singularity image and support Intel Xeon Phi cards on Salomon singularity image. (OS/Version[none|-GPU|-MIC])
Lukáš Krupčík's avatar
Lukáš Krupčík committed
32

Lukáš Krupčík's avatar
Lukáš Krupčík committed
33
## IT4Innovations Wrappers
Lukáš Krupčík's avatar
Lukáš Krupčík committed
34

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
35
To use our singularity images we prepared special wrappers:
Lukáš Krupčík's avatar
Lukáš Krupčík committed
36
37
38
39
40
41
42

* image-exec
* image-mpi
* image-run
* image-shell
* image-update

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
43
44
The latest version of the scripts is [here](https://code.it4i.cz/sccs/it4i-singularity/tree/master/bin)

Lukáš Krupčík's avatar
Lukáš Krupčík committed
45
46
**image-exec**

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
47
Open the image and run the command inside the image.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
48
49
50

```console
$ ml CentOS/7.3
Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
51
Your image of CentOS/7.3 is at location: /home/login/.singularity/images/CentOS-7.3_20180220104046.img
Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
52
$ image-exec cat /etc/centos-release
Lukáš Krupčík's avatar
Lukáš Krupčík committed
53
54
55
56
57
CentOS Linux release 7.3.1708 (Core)
```

**image-mpi**

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
58
MPI wrapper. More in the chapter [Examples MPI](#mpi).
Lukáš Krupčík's avatar
Lukáš Krupčík committed
59
60
61

**image-run**

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
62
Run the subscript inside image.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
63
64
65

**image-shell**

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
66
Start the shell inside image.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
67
68
69
70
71
72
73
74
75
76
77

```console
$ ml CentOS/7.3
$ image-shell
Singularity: Invoking an interactive shell within container...

Singularity CentOS-7.3_20180220104046.img:~>
```

**image-update**

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
78
If a new image (version) exists, informs the user and offers an update image.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97

```console
$ ml CentOS/6.9
New version of CentOS image was found. (New: CentOS-6.9_20180220092823.img Old: CentOS-6.9_20170220092823.img)
For updating image use: image-update
Your image of CentOS/6.9 is at location: /home/login/.singularity/images/CentOS-6.9_20170220092823.img
$ image-update
New version of CentOS image was found. (New: CentOS-6.9_20180220092823.img Old: CentOS-6.9_20170220092823.img)
Do you want to update local copy? (WARNING all user modification will be deleted) [y/N]: y
Updating image  CentOS-6.9_20180220092823.img
       2.71G 100%  199.49MB/s    0:00:12 (xfer#1, to-check=0/1)

sent 2.71G bytes  received 31 bytes  163.98M bytes/sec
total size is 2.71G  speedup is 1.00
New version is ready. (/home/login/.singularity/images/CentOS-6.9_20180220092823.img)
```

## Examples

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
98
In next examples, we will be use Singularity images on IT4Innovations clusters.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
99

Lukáš Krupčík's avatar
Lukáš Krupčík committed
100
### Load Image
Lukáš Krupčík's avatar
Lukáš Krupčík committed
101
102
103
104
105
106

```console
$ ml CentOS/6.9
Your image of CentOS/6.9 is at location: /home/login/.singularity/images/CentOS-6.9_20180220133305.img
```

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
107
108
!!! tip
    First usage module with singularity image, copy singularity image from /apps/all/OS/... to your /home (.singularity/images)
Lukáš Krupčík's avatar
Lukáš Krupčík committed
109

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
### Intel Xeon Phi Cards - MIC

For example submit job `qsub -A PROJECT -q qprod -l select=1:mpiprocs=24:accelerator=true -I`

!!! info
    MIC image was prepared for only Salomon cluster

**Code for testing offload**

```c
#include <stdio.h>
#include <thread>
#include <stdlib.h>
#include <unistd.h>

int main() {

  char hostname[1024];
  gethostname(hostname, 1024);

  unsigned int nthreads = std::thread::hardware_concurrency();
  printf("Hello world, #of cores: %d\n",nthreads);
  #pragma offload target(mic)
  {
    nthreads = std::thread::hardware_concurrency();
    printf("Hello world from MIC, #of cores: %d\n",nthreads);
  }
}
```

**Compile and run**

```console
[login@r38u03n975 ~]$ ml CentOS/6.9-MIC
Your image of CentOS/6.9-MIC is at location: /home/login/.singularity/images/CentOS-6.9-MIC_20180220112004.img
[login@r38u03n975 ~]$ image-shell
Singularity: Invoking an interactive shell within container...

Singularity CentOS-6.9-MIC_20180220112004.img:~> ml intel/2017b
Singularity CentOS-6.9-MIC_20180220112004.img:~> ml

Currently Loaded Modules:
  1) GCCcore/6.3.0                 3) icc/2017.1.132-GCC-6.3.0-2.27     5) iccifort/2017.1.132-GCC-6.3.0-2.27                   7) iimpi/2017a                   9) intel/2017a
  2) binutils/2.27-GCCcore-6.3.0   4) ifort/2017.1.132-GCC-6.3.0-2.27   6) impi/2017.1.132-iccifort-2017.1.132-GCC-6.3.0-2.27   8) imkl/2017.1.132-iimpi-2017a
Singularity CentOS-6.9-MIC_20180220112004.img:~> icpc -std=gnu++11 -qoffload=optional  hello.c -o hello-host
Singularity CentOS-6.9-MIC_20180220112004.img:~> ./hello-host
Hello world, #of cores: 24
Hello world from MIC, #of cores: 244
```

Lukáš Krupčík's avatar
Lukáš Krupčík committed
160
### GPU Image
Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
161
162
163

For example submit job `qsub -A PROJECT -q qnvidia -l select=1:ncpus=16:mpiprocs=16 -l walltime=01:00:00 -I`

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
164
!!! note
Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
165
166
167
168
169
    GPU image was prepared for only Anselm cluster

**Checking nvidia driver inside image**

```console
Lukáš Krupčík's avatar
fix GPU    
Lukáš Krupčík committed
170
[login@cn199.anselm ~]$ image-shell
Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
171
172
Singularity: Invoking an interactive shell within container...

Lukáš Krupčík's avatar
fix GPU    
Lukáš Krupčík committed
173
174
175
Singularity CentOS-6.9-GPU_20180309130604.img:~> ml
No modules loaded
Singularity CentOS-6.9-GPU_20180309130604.img:~> nvidia-smi
Lukáš Krupčík's avatar
GPU    
Lukáš Krupčík committed
176
Mon Mar 12 07:07:53 2018
Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
177
+-----------------------------------------------------------------------------+
Lukáš Krupčík's avatar
Lukáš Krupčík committed
178
| NVIDIA-SMI 390.30                 Driver Version: 390.30                    |
Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
179
180
181
182
183
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20m          Off  | 00000000:02:00.0 Off |                    0 |
Lukáš Krupčík's avatar
fix GPU    
Lukáš Krupčík committed
184
| N/A   28C    P0    51W / 225W |      0MiB /  4743MiB |     89%      Default |
Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
185
+-------------------------------+----------------------+----------------------+
Lukáš Krupčík's avatar
GPU    
Lukáš Krupčík committed
186

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
187
188
189
190
191
192
193
194
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
```

Lukáš Krupčík's avatar
Lukáš Krupčík committed
195
196
### MPI

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
197
198
199
200
201
202
203
204
205
206
207
For example submit job `qsub -A PROJECT -q qprod -l select=2:mpiprocs=24 -l walltime=00:30:00 -I`

!!! note
    We have seen no major performance impact from running a job in a Singularity container.

With Singularity, the MPI usage model is to call ‘mpirun’ from outside the container, and reference the container from your ‘mpirun’ command. Usage would look like this:

```console
$ mpirun -np 20 singularity exec container.img /path/to/contained_mpi_prog
```

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
208
By calling `mpirun` outside the container, we solve several very complicated work-flow aspects. For example, if `mpirun` is called from within the container it must have a method for spawning processes on remote nodes. Historically ssh is used for this which means that there must be an sshd running within the container on the remote nodes, and this sshd process must not conflict with the sshd running on that host! It is also possible for the resource manager to launch the job and (in Open MPI’s case) the Orted processes on the remote system, but that then requires resource manager modification and container awareness.
Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
209

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
210
In the end, we do not gain anything by calling `mpirun` from within the container except for increasing the complexity levels and possibly losing out on some added performance benefits (e.g. if a container wasn’t built with the proper OFED as the host).
Lukáš Krupčík's avatar
Lukáš Krupčík committed
211

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
212
#### MPI Inside Image
Lukáš Krupčík's avatar
Lukáš Krupčík committed
213
214

```console
Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
215
216
$ ml CentOS/6.9
$ image-shell
Lukáš Krupčík's avatar
Lukáš Krupčík committed
217
218
219
220
221
222
223
Singularity: Invoking an interactive shell within container...

Singularity CentOS-6.9_20180220092823.img:~> mpirun hostname | wc -l
24
```

!!! warning
Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
224
    You allocate two nodes, but MPI inside image use only one node.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
225

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
226
#### MPI Outside Image
Lukáš Krupčík's avatar
Lukáš Krupčík committed
227
228
229

```console
$ ml CentOS/6.9
Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
230
Your image of CentOS/6.9 is at location: /home/login/.singularity/images/CentOS-6.9_20180220092823.img
Lukáš Krupčík's avatar
Lukáš Krupčík committed
231
232
233
234
$ image-mpi hostname | wc -l
48
```

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
235
## How to Use Own Image on Cluster?
Lukáš Krupčík's avatar
Lukáš Krupčík committed
236

Lukáš Krupčík's avatar
Lukáš Krupčík committed
237
238
* Prepare the image on your  computer
* Transfer the images to your `/home` directory on the cluster (for example `.singularity/image`)
Lukáš Krupčík's avatar
add scp    
Lukáš Krupčík committed
239
240
241
242
243

```console
local $ scp image  login@login4.salomon.it4i.cz:/home/login/.singularity/image/image
```

Lukáš Krupčík's avatar
Lukáš Krupčík committed
244
245
* Load module Singularity (`ml Singularity`)
* Use your image
Lukáš Krupčík's avatar
Lukáš Krupčík committed
246

Lukáš Krupčík's avatar
fix    
Lukáš Krupčík committed
247
## How to Edit IT4Innovations Image?
Lukáš Krupčík's avatar
Lukáš Krupčík committed
248

Lukáš Krupčík's avatar
Lukáš Krupčík committed
249
* Transfer the image to your computer
Lukáš Krupčík's avatar
add scp    
Lukáš Krupčík committed
250
251
252
253
254

```console
local $ scp login@login4.salomon.it4i.cz:/home/login/.singularity/image/image image
```

Lukáš Krupčík's avatar
Lukáš Krupčík committed
255
256
* Modify the image
* Transfer the image from your computer to your `/home` directory on the cluster
Lukáš Krupčík's avatar
add scp    
Lukáš Krupčík committed
257
258
259
260
261

```console
local $ scp image  login@login4.salomon.it4i.cz:/home/login/.singularity/image/image
```

Lukáš Krupčík's avatar
Lukáš Krupčík committed
262
263
* Load module Singularity (`ml Singularity`)
* Use your image