singularity-it4i.md

# Singularity on IT4Innovations

On clusters we have different versions of operating systems singularity images. Below you see the available operating systems singularity images.

```console
   Salomon                      Anselm
      ├── CentOS                  ├── CentOS
      │   ├── 6.9                 │   ├── 6.9
      │   ├── 6.9-MIC             │   ├── 6.9-GPU
      │   └── 7.3                 │   └── 7.3
      ├── Debian                  ├── Debian
      │   └── 8.0                 │   └── 8.0
      ├── Fedora                  ├── Fedora
      │   └── 26                  │   └── 26
      └── Ubuntu                  └── Ubuntu
          └── 16.04                   └── 16.04
```

For current information on available singularity images, refer to the `ml av` and see statement the `OS`

## Available Operating Systems Images

* CentOS
* Debian
* Ubuntu
* Fedora

!!! note
    We support graphical cards on Anselm singularity image and support Intel Xeon Phi cards on Salomon images. (OS/Version-[none|GPU|MIC])

## IT4Innovations Wrappers

For using modules we prepared special wrappers

* image-exec
* image-mpi
* image-run
* image-shell
* image-update

**image-exec**

Open image and run command into image.

```console
$ ml CentOS/7.3
Your image of CentOS/7.3 is at location: /home/kru0052/.singularity/images/CentOS-7.3_20180220104046.img
image-exec cat /etc/centos-release
CentOS Linux release 7.3.1708 (Core)
```

**image-mpi**

MPI wrapper. More in the chapter [Examples MPI](#mpi)

**image-run**

Run subscript into image.

**image-shell**

Start shell into image.

```console
$ ml CentOS/7.3
$ image-shell
Singularity: Invoking an interactive shell within container...

Singularity CentOS-7.3_20180220104046.img:~>
```

**image-update**

If a new image exists, it informs the user and offers an update.

```console
$ ml CentOS/6.9
New version of CentOS image was found. (New: CentOS-6.9_20180220092823.img Old: CentOS-6.9_20170220092823.img)
For updating image use: image-update
Your image of CentOS/6.9 is at location: /home/login/.singularity/images/CentOS-6.9_20170220092823.img
$ image-update
New version of CentOS image was found. (New: CentOS-6.9_20180220092823.img Old: CentOS-6.9_20170220092823.img)
Do you want to update local copy? (WARNING all user modification will be deleted) [y/N]: y
Updating image  CentOS-6.9_20180220092823.img
       2.71G 100%  199.49MB/s    0:00:12 (xfer#1, to-check=0/1)

sent 2.71G bytes  received 31 bytes  163.98M bytes/sec
total size is 2.71G  speedup is 1.00
New version is ready. (/home/login/.singularity/images/CentOS-6.9_20180220092823.img)
```

## Examples

In next examples, we will be using Singularity images on IT4Innovations clusters.

### Load Image

```console
$ ml CentOS/6.9
Your image of CentOS/6.9 is at location: /home/login/.singularity/images/CentOS-6.9_20180220133305.img
```

!!! note
    First usage image copy image from /apps/all/OS/... to your /home (.singularity/images)

For GPU and MIC image

```console
$ ml CentOS/6.9-GPU
```

```console
$ ml CentOS/6.9-MIC
```

!!! note
    For the GPU image, you must allocate node with GPU card and for the MIC image, you must allocate node with the Intel Xeon Phi cards.

### MPI

For example submit job `qsub -A PROJECT -q qprod -l select=2:mpiprocs=24 -l walltime=00:30:00 -I`

!!! note
    We have seen no major performance impact from running a job in a Singularity container.

With Singularity, the MPI usage model is to call ‘mpirun’ from outside the container, and reference the container from your ‘mpirun’ command. Usage would look like this:

```console
$ mpirun -np 20 singularity exec container.img /path/to/contained_mpi_prog
```

By calling ‘mpirun’ outside the container, we solve several very complicated work-flow aspects. For example, if ‘mpirun’ is called from within the container it must have a method for spawning processes on remote nodes. Historically ssh is used for this which means that there must be an sshd running within the container on the remote nodes, and this sshd process must not conflict with the sshd running on that host! It is also possible for the resource manager to launch the job and (in Open MPI’s case) the Orted processes on the remote system, but that then requires resource manager modification and container awareness.

In the end, we do not gain anything by calling ‘mpirun’ from within the container except for increasing the complexity levels and possibly losing out on some added performance benefits (e.g. if a container wasn’t built with the proper OFED as the host).

### MPI Inside Image

```console
$ ml CentOS/6.9
$ image-shell
Singularity: Invoking an interactive shell within container...

Singularity CentOS-6.9_20180220092823.img:~> mpirun hostname | wc -l
24
```

!!! warning
    You allocate two nodes, but MPI into image use only one node.

### MPI Outside Image

```console
$ ml CentOS/6.9
Your image of CentOS/6.9 is at location: /home/kru0052/.singularity/images/CentOS-6.9_20180220092823.img
$ image-mpi hostname | wc -l
48
```

## Use Own Image on Cluster

1. Prepare image into your local computer
1. Transfer image to your `/home` (for example `.singularity/image`)
1. Load module Singularity (`ml Singularity`)
1. Use your image

## Edit IT4Innovations Image

1. Transfer image to your local computer
1. Edit image
1. Transfer image from your local computer to cluster /home
1. Load module Singularity (`ml Singularity`)
1. Use your image