diff --git a/docs.it4i/software/tools/singularity-it4i.md b/docs.it4i/software/tools/singularity-it4i.md index 5c416052812c979a10c485aa5e92c9daac117f56..c37a16d0e9f47a8c3375fb3d19d892b28adbfe78 100644 --- a/docs.it4i/software/tools/singularity-it4i.md +++ b/docs.it4i/software/tools/singularity-it4i.md @@ -3,18 +3,16 @@ On our clusters, the Singularity images of main linux distributions are prepared. ```console - Salomon Anselm - ├── CentOS ├── CentOS - │ ├── 6.9 │ ├── 6.9 - │ ├── 6.9-MIC │ ├── 6.9-GPU - │ ├── 7.5 │ ├── 7.4 - │ └── 7.5-intel │ └── 7.4-GPU - ├── Debian ├── Debian - │ └── 8.0 │ ├── 8.0 - ├── Fedora │ └── 8.0-GPU - │ └── 29 └── Ubuntu - └── Ubuntu ├── 16.04 - └── 16.04 └── 16.04-GPU + Salomon Anselm Barbora + ├── CentOS ├── CentOS ├── CentOS + │ ├── 6 │ ├── 6 │ ├── 6 + │ └── 7 │ └── 7 │ └── 7 + ├── Debian ├── Debian ├── Debian + │ └── latest │ └── latest │ └── latest + ├── Fedora ├── Fedora ├── Fedora + │ └── latest │ └── latest │ └── latest + └── Ubuntu └── Ubuntu └── Ubuntu + └── latest └── latest └── latest ``` !!! info @@ -22,9 +20,6 @@ On our clusters, the Singularity images of main linux distributions are prepared The bootstrap scripts, wrappers, features, etc. are located [here][a]. -!!! note - The images with graphic card support are marked as **-GPU** and images with Intel Xeon Phi support are marked as **-MIC** - ## IT4Innovations Singularity Wrappers For better user experience with Singularity containers we prepared several wrappers: @@ -56,8 +51,8 @@ In the examples, we will use the prepared Singularity images. ### Load Image ```console -$ ml CentOS/6.9 -Your image of CentOS/6.9 is at location: /home/login/.singularity/images/CentOS-6.9_20180220133305.img +$ ml CentOS/6 +Your image of CentOS/6 is at location: /home/login/.singularity/images/CentOS-6_20180220133305.img ``` !!! tip @@ -70,8 +65,8 @@ Your image of CentOS/6.9 is at location: /home/login/.singularity/images/CentOS- Executes the given command inside the Singularity image. The container is in this case started, then the command is executed and the container is stopped. ```console -$ ml CentOS/7.3 -Your image of CentOS/7.3 is at location: /home/login/.singularity/images/CentOS-7.3_20180220104046.img +$ ml CentOS/7 +Your image of CentOS/7 is at location: /home/login/.singularity/images/CentOS-7_20180220104046.img $ image-exec cat /etc/centos-release CentOS Linux release 7.3.1708 (Core) ``` @@ -89,11 +84,11 @@ This command runs the runscript inside the Singularity image. Note, that the pre Invokes an interactive shell inside the Singularity image. ```console -$ ml CentOS/7.3 +$ ml CentOS/7 $ image-shell Singularity: Invoking an interactive shell within container... -Singularity CentOS-7.3_20180220104046.img:~> +Singularity CentOS-7_20180220104046.img:~> ``` ### Update Image @@ -101,104 +96,19 @@ Singularity CentOS-7.3_20180220104046.img:~> This command is for updating your local copy of the Singularity image. The local copy is overridden in this case. ```console -$ ml CentOS/6.9 -New version of CentOS image was found. (New: CentOS-6.9_20180220092823.img Old: CentOS-6.9_20170220092823.img) +$ ml CentOS/6 +New version of CentOS image was found. (New: CentOS-6_20180220092823.img Old: CentOS-6_20170220092823.img) For updating image use: image-update -Your image of CentOS/6.9 is at location: /home/login/.singularity/images/CentOS-6.9_20170220092823.img +Your image of CentOS/6 is at location: /home/login/.singularity/images/CentOS-6_20170220092823.img $ image-update -New version of CentOS image was found. (New: CentOS-6.9_20180220092823.img Old: CentOS-6.9_20170220092823.img) +New version of CentOS image was found. (New: CentOS-6_20180220092823.img Old: CentOS-6_20170220092823.img) Do you want to update local copy? (WARNING all user modification will be deleted) [y/N]: y -Updating image CentOS-6.9_20180220092823.img +Updating image CentOS-6_20180220092823.img 2.71G 100% 199.49MB/s 0:00:12 (xfer#1, to-check=0/1) sent 2.71G bytes received 31 bytes 163.98M bytes/sec total size is 2.71G speedup is 1.00 -New version is ready. (/home/login/.singularity/images/CentOS-6.9_20180220092823.img) -``` - -### Intel Xeon Phi Cards - MIC - -In the following example, we are using a job submitted by the command: `qsub -A PROJECT -q qprod -l select=1:mpiprocs=24:accelerator=true -I` - -!!! info - The MIC image was prepared only for the Salomon cluster. - -**Code for the Offload Test** - -```c -#include <stdio.h> -#include <thread> -#include <stdlib.h> -#include <unistd.h> - -int main() { - - char hostname[1024]; - gethostname(hostname, 1024); - - unsigned int nthreads = std::thread::hardware_concurrency(); - printf("Hello world, #of cores: %d\n",nthreads); - #pragma offload target(mic) - { - nthreads = std::thread::hardware_concurrency(); - printf("Hello world from MIC, #of cores: %d\n",nthreads); - } -} -``` - -**Compile and Run** - -```console -[login@r38u03n975 ~]$ ml CentOS/6.9-MIC -Your image of CentOS/6.9-MIC is at location: /home/login/.singularity/images/CentOS-6.9-MIC_20180220112004.img -[login@r38u03n975 ~]$ image-shell -Singularity: Invoking an interactive shell within container... - -Singularity CentOS-6.9-MIC_20180220112004.img:~> ml intel/2017b -Singularity CentOS-6.9-MIC_20180220112004.img:~> ml - -Currently Loaded Modules: - 1) GCCcore/6.3.0 3) icc/2017.1.132-GCC-6.3.0-2.27 5) iccifort/2017.1.132-GCC-6.3.0-2.27 7) iimpi/2017a 9) intel/2017a - 2) binutils/2.27-GCCcore-6.3.0 4) ifort/2017.1.132-GCC-6.3.0-2.27 6) impi/2017.1.132-iccifort-2017.1.132-GCC-6.3.0-2.27 8) imkl/2017.1.132-iimpi-2017a -Singularity CentOS-6.9-MIC_20180220112004.img:~> icpc -std=gnu++11 -qoffload=optional hello.c -o hello-host -Singularity CentOS-6.9-MIC_20180220112004.img:~> ./hello-host -Hello world, #of cores: 24 -Hello world from MIC, #of cores: 244 -``` - -### GPU Image - -In the following example, we are using a job submitted by the command: `qsub -A PROJECT -q qnvidia -l select=1:ncpus=16:mpiprocs=16 -l walltime=01:00:00 -I` - -!!! note - The GPU image was prepared only for the Anselm cluster. - -**Checking NVIDIA Driver Inside Image** - -```console -[login@cn199.anselm ~]$ image-shell -Singularity: Invoking an interactive shell within container... - -Singularity CentOS-6.9-GPU_20180309130604.img:~> ml -No modules loaded -Singularity CentOS-6.9-GPU_20180309130604.img:~> nvidia-smi -Mon Mar 12 07:07:53 2018 -+-----------------------------------------------------------------------------+ -| NVIDIA-SMI 390.30 Driver Version: 390.30 | -|-------------------------------+----------------------+----------------------+ -| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | -| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | -|===============================+======================+======================| -| 0 Tesla K20m Off | 00000000:02:00.0 Off | 0 | -| N/A 28C P0 51W / 225W | 0MiB / 4743MiB | 89% Default | -+-------------------------------+----------------------+----------------------+ - -+-----------------------------------------------------------------------------+ -| Processes: GPU Memory | -| GPU PID Type Process name Usage | -|=============================================================================| -| No running processes found | -+-----------------------------------------------------------------------------+ +New version is ready. (/home/login/.singularity/images/CentOS-6_20180220092823.img) ``` ### MPI @@ -222,11 +132,11 @@ performance benefits (e.g. if a container wasn’t built with the proper OFED as #### MPI Inside Singularity Image ```console -$ ml CentOS/6.9 +$ ml CentOS/7 $ image-shell Singularity: Invoking an interactive shell within container... -Singularity CentOS-6.9_20180220092823.img:~> mpirun hostname | wc -l +Singularity CentOS-7_20180220092823.img:~> mpirun hostname | wc -l 24 ``` @@ -235,8 +145,8 @@ As you can see in this example, we allocated two nodes, but MPI can use only one #### MPI Outside Singularity Image ```console -$ ml CentOS/6.9 -Your image of CentOS/6.9 is at location: /home/login/.singularity/images/CentOS-6.9_20180220092823.img +$ ml CentOS/7 +Your image of CentOS/7 is at location: /home/login/.singularity/images/CentOS-7_20180220092823.img $ image-mpi hostname | wc -l 48 ```