Skip to content
Snippets Groups Projects
Commit 7e73b131 authored by Jan Siwiec's avatar Jan Siwiec
Browse files

Merge branch 'apptainer' into 'master' 

Update docs.it4i/software/tools/singularity-it4i.md

See merge request !419
parents 91ed8ba1 41c88c30
No related branches found
No related tags found
1 merge request!419Update docs.it4i/software/tools/singularity-it4i.md
Pipeline #29527 passed with warnings
Stage: test
Stage: build
Stage: deploy
Stage: after_test
Hide whitespace changes
Inline Side-by-side
docs.it4i/software/tools/singularity-it4i.md
+ 62
− 63
View file @ 7e73b131
# Apptainer/Singularity on IT4Innovations
# Apptainer on IT4Innovations
!!!note "Singularity name change"
On November 30th, 2021, the Singularity project has officially moved into the Linux Foundation. As part of this move, and to differentiate from the other like-named projects and commercial products, the project is renamed to "Apptainer".
On our clusters, the Apptainer/Singularity images of main Linux distributions are prepared.
On our clusters, the Apptainer images of main Linux distributions are prepared.
```console
Barbora
├── CentOS
│ ├── 6
│ └── 7
├── Debian
│ └── latest
├── Fedora
│ └── latest
└── Ubuntu
└── latest
Barbora Karolina
├── CentOS ├── CentOS
| └── 7 | └── 7
├── Rocky ├── Rocky
| ├── 8 | ├── 8
│ └── 9 │ └── 9
├── Fedora ├── Fedora
│ └── latest │ └── latest
└── Ubuntu └── Ubuntu
└── latest └── latest
```
!!! info
Current information about available Apptainer/Singularity images can be obtained by the `ml av` command. The images are listed in the `OS` section.
Current information about available Apptainer images can be obtained by the `ml av` command. The images are listed in the `OS` section.
The bootstrap scripts, wrappers, features, etc. are located on [it4i-singularity GitLab page][a].
## IT4Innovations Apptainer/Singularity Wrappers
## IT4Innovations Apptainer Wrappers
For better user experience with Apptainer/Singularity containers, we prepared several wrappers:
For better user experience with Apptainer containers, we prepared several wrappers:
* image-exec
* image-mpi
......@@ -33,8 +33,8 @@ For better user experience with Apptainer/Singularity containers, we prepared se
* image-shell
* image-update
Listed wrappers help you to use prepared Apptainer/Singularity images loaded as modules.
You can easily load a Apptainer/Singularity image like any other module on the cluster by the `ml OS/version` command.
Listed wrappers help you to use prepared Apptainer images loaded as modules.
You can easily load a Apptainer image like any other module on the cluster by the `ml OS/version` command.
After the module is loaded for the first time, the prepared image is copied into your home folder and is ready for use.
When you load the module next time, the version of the image is checked and an image update (if exists) is offered.
Then you can update your copy of the image by the `image-update` command.
......@@ -42,51 +42,54 @@ Then you can update your copy of the image by the `image-update` command.
!!! warning
With an image update, all user changes to the image will be overridden.
The runscript inside the Apptainer/Singularity image can be run by the `image-run` command.
This command automatically mounts the `/scratch` and `/apps` storage and invokes the image as writable, so user changes can be made.
The runscript inside the Apptainer image can be run by the `image-run` command.
!!! note " CentOS/7 module only"
This command automatically mounts the `/scratch` and `/apps` storage and invokes the image as writable, so user changes can be made.
Very similar to `image-run` is the `image-exec` command.
The only difference is that `image-exec` runs a user-defined command instead of a runscript.
In this case, the command to be run is specified as a parameter.
Using the interactive shell inside the Apptainer/Singularity container is very useful for development.
Using the interactive shell inside the Apptainer container is very useful for development.
In this interactive shell, you can make any changes to the image you want,
but be aware that you can not use the `sudo` privileged commands directly on the cluster.
To simply invoke interactive shell, use the `image-shell` command.
Another useful feature of the Apptainer/Singularity is the direct support of OpenMPI.
Another useful feature of the Apptainer is the direct support of OpenMPI.
For proper MPI function, you have to install the same version of OpenMPI inside the image as you use on the cluster.
OpenMPI/3.1.4 is installed in prepared images.
OpenMPI/4.1.2 is installed in prepared images (CentOS 7, Rocky 8).
The MPI must be started outside the container.
The easiest way to start the MPI is to use the `image-mpi` command.
This command has the same parameters as `mpirun`, so there is no difference between running normal MPI application
and MPI application in a Apptainer/Singularity container.
and MPI application in a Apptainer container.
## Examples
In the examples, we will use prepared Apptainer/Singularity images.
In the examples, we will use prepared Apptainer images.
### Load Image
```console
$ ml CentOS/6
Your image of CentOS/6 is at location: /home/login/.singularity/images/CentOS-6_20180220133305.img
$ ml CentOS/7
Preparing image CentOS-7_20230116143612.sif
261.20M 100% 412.36MB/s 0:00:00 (xfr#1, to-chk=0/1)
Your image of CentOS/7 is at location: /home/username/.apptainer/images/CentOS-7_20230116143612.sif
```
!!! tip
After the module is loaded for the first time, the prepared image is copied into your home folder to the *.singularity/images* subfolder.
After the module is loaded for the first time, the prepared image is copied into your home folder to the *.apptainer/images* subfolder.
### Wrappers
**image-exec**
Executes the given command inside the Apptainer/Singularity image. The container is in this case started, then the command is executed and the container is stopped.
Executes the given command inside the Apptainer image. The container is in this case started, then the command is executed and the container is stopped.
```console
$ ml CentOS/7
Your image of CentOS/7 is at location: /home/login/.singularity/images/CentOS-7_20180220104046.img
$ image-exec cat /etc/centos-release
CentOS Linux release 7.3.1708 (Core)
$ image-exec cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)
```
**image-mpi**
......@@ -95,55 +98,53 @@ MPI wrapper - see more in the [Examples MPI][1] section.
**image-run**
This command runs the runscript inside the Apptainer/Singularity image. Note, that the prepared images do not contain a runscript.
This command runs the runscript inside the Apptainer image. Note, that the prepared images do not contain a runscript.
**image-shell**
Invokes an interactive shell inside the Apptainer/Singularity image.
Invokes an interactive shell inside the Apptainer image.
```console
$ ml CentOS/7
$ image-shell
Singularity: Invoking an interactive shell within container...
Singularity CentOS-7_20180220104046.img:~>
Apptainer>
```
### Update Image
This command is for updating your local Apptainer/Singularity image copy.
This command is for updating your local Apptainer image copy.
The local copy is overridden in this case.
```console
$ ml CentOS/6
New version of CentOS image was found. (New: CentOS-6_20180220092823.img Old: CentOS-6_20170220092823.img)
$ ml CentOS/7
New version of CentOS image was found. (New: CentOS-7_20230116143612.sif Old: CentOS-7_20230115143612.sif)
For updating image use: image-update
Your image of CentOS/6 is at location: /home/login/.singularity/images/CentOS-6_20170220092823.img
Your image of CentOS/7 is at location: /home/username/.apptainer/images/CentOS-7_20230115143612.sif
$ image-update
New version of CentOS image was found. (New: CentOS-6_20180220092823.img Old: CentOS-6_20170220092823.img)
New version of CentOS image was found. (New: CentOS-7_20230116143612.sif Old: CentOS-7_20230115143612.sif)
Do you want to update local copy? (WARNING all user modification will be deleted) [y/N]: y
Updating image CentOS-6_20180220092823.img
Updating image CentOS-7_20230116143612.sif
2.71G 100% 199.49MB/s 0:00:12 (xfer#1, to-check=0/1)
sent 2.71G bytes received 31 bytes 163.98M bytes/sec
total size is 2.71G speedup is 1.00
New version is ready. (/home/login/.singularity/images/CentOS-6_20180220092823.img)
New version is ready. (/home/username/.apptainer/images/CentOS-7_20230116143612.sif)
```
### MPI
In the following example, we are using a job submitted by the command:
`qsub -A PROJECT -q qprod -l select=2:mpiprocs=24 -l walltime=00:30:00 -I`
`qsub -A PROJECT -q qprod -l select=2:mpiprocs=128 -l walltime=00:30:00 -I`
!!! note
We have seen no major performance impact for a job running in a Apptainer/Singularity container.
We have seen no major performance impact for a job running in a Apptainer container.
With Apptainer/Singularity, the MPI usage model is to call `mpirun` from outside the container
With Apptainer, the MPI usage model is to call `mpirun` from outside the container
and reference the container from your `mpirun` command.
Usage would look like this:
```console
$ mpirun -np 24 singularity exec container.img /path/to/contained_mpi_prog
$ mpirun -np 128 apptainer exec container.img /path/to/contained_mpi_prog
```
By calling `mpirun` outside of the container, we solve several very complicated work-flow aspects.
......@@ -158,26 +159,24 @@ In the end, we do not gain anything by calling `mpirun` from within the containe
except for increasing the complexity levels and possibly losing out on some added
performance benefits (e.g. if a container was not built with the proper OFED as the host).
#### MPI Inside Apptainer/Singularity Image
#### MPI Inside Apptainer Image
```console
$ ml CentOS/7
$ image-shell
Singularity: Invoking an interactive shell within container...
Singularity CentOS-7_20180220092823.img:~> mpirun hostname | wc -l
24
Apptainer> mpirun hostname | wc -l
128
```
As you can see in this example, we allocated two nodes, but MPI can use only one node (24 processes) when used inside the Apptainer/Singularity image.
As you can see in this example, we allocated two nodes, but MPI can use only one node (128 processes) when used inside the Apptainer image.
#### MPI Outside Apptainer/Singularity Image
#### MPI Outside Apptainer Image
```console
$ ml CentOS/7
Your image of CentOS/7 is at location: /home/login/.singularity/images/CentOS-7_20180220092823.img
Your image of CentOS/7 is at location: /home/username/.apptainer/images/CentOS-7_20230116143612.sif
$ image-mpi hostname | wc -l
48
256
```
In this case, the MPI wrapper behaves like the `mpirun` command.
......@@ -187,34 +186,34 @@ and the communication between nodes are propagated into the container automatica
## How to Use Own Image on Cluster?
* Prepare the image on your computer
* Transfer the images to your `/home` directory on the cluster (for example `.singularity/image`)
* Transfer the images to your `/home` directory on the cluster (for example `.apptainer/image`)
```console
local:$ scp container.img login@login4.clustername.it4i.cz:~/.singularity/image/container.img
local:$ scp container.img login@login2.clustername.it4i.cz:~/.apptainer/image/container.img
```
* Load module Apptainer/Singularity (`ml Singularity`)
* Load module Apptainer (`ml apptainer`)
* Use your image
!!! note
If you want to use the Apptainer/Singularity wrappers with your own images, load the `Singularity-wrappers/master` module and set the environment variable `IMAGE_PATH_LOCAL=/path/to/container.img`.
If you want to use the Apptainer wrappers with your own images, load the `apptainer-wrappers/1.0` module and set the environment variable `IMAGE_PATH_LOCAL=/path/to/container.img`.
## How to Edit IT4Innovations Image?
* Transfer the image to your computer
```console
local:$ scp login@login4.clustername.it4i.cz:/home/login/.singularity/image/container.img container.img
local:$ scp login@login2.clustername.it4i.cz:/home/username/.apptainer/image/container.img container.img
```
* Modify the image
* Transfer the image from your computer to your `/home` directory on the cluster
```console
local:$ scp container.img login@login4.clustername.it4i.cz:/home/login/.singularity/image/container.img
local:$ scp container.img login@login2.clustername.it4i.cz:/home/username/.apptainer/image/container.img
```
* Load module Apptainer/Singularity (`ml Singularity`)
* Load module Apptainer (`ml apptainer`)
* Use your image
[1]: #mpi
......
docs.it4i/software/tools/singularity.md
+ 75
− 79
View file @ 7e73b131
# Singularity Container
# Apptainer Container
[Singularity][a] enables users to have full control of their environment. A non-privileged user can "swap out" the operating system on the host for one they control. So if the host system is running RHEL6 but your application runs in Ubuntu/RHEL7, you can create an Ubuntu/RHEL7 image, install your applications into that image, copy the image to another host, and run your application on that host in its native Ubuntu/RHEL7 environment.
Singularity also allows you to leverage the resources of whatever host you are on. This includes HPC interconnects, resource managers, file systems, GPUs and/or accelerators, etc. Singularity does this by enabling several key facets:
* Encapsulation of the environment
* Containers are image based
* No user contextual changes or root escalation allowed
* No root owned daemon processes
This documentation is for Singularity version 2.4 and newer.
<div align="center">
<iframe width="250" height="50%" src="https://www.youtube.com/embed/m8llDjFuXlc" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<iframe width="250" height="50%" src="https://www.youtube.com/embed/SJHizTjwyFk" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<iframe width="250" height="50%" src="https://www.youtube.com/embed/97VuBVnfcwg" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<iframe width="250" height="50%" src="https://www.youtube.com/embed/wGJnkRmW5iU" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
</div>
[Apptainer][a] is a container platform. It allows you to create and run containers that package up pieces of software in a way that is portable and reproducible. You can build a container using Apptainer on your laptop, and then run it on many of the largest HPC clusters in the world, local university or company clusters, a single server, in the cloud, or on a workstation down the hall. Your container is a single file, and you don’t have to worry about how to install all the software you need on each different operating system.
## Using Docker Images
Singularity can import, bootstrap, and even run Docker images directly from [Docker Hub][b]. You can easily run an RHEL7 container like this:
Apptainer can import, bootstrap, and even run Docker images directly from [Docker Hub][b]. You can easily run an CentOS container like this:
```console
hra0031@login4:~$ cat /etc/redhat-release
CentOS release 6.9 (Final)
hra0031@login4:~$ ml Singularity
hra0031@login4:~$ singularity shell docker://centos:latest
Docker image path: index.docker.io/library/centos:latest
Cache folder set to /home/hra0031/.singularity/docker
[1/1] |===================================| 100.0%
Creating container runtime...
Singularity: Invoking an interactive shell within container...
Singularity centos:latest:~> cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
$ cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)
$ ml apptainer
$ apptainer shell docker://centos:latest
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob a1d0c7532777 done
Copying config 8c1402b22a done
Writing manifest to image destination
Storing signatures
2023/01/17 12:55:08 info unpack layer: sha256:a1d0c75327776413fa0db9ed3adcdbadedc95a662eb1d360dad82bb913f8a1d1
2023/01/17 12:55:09 warn rootless{usr/bin/newgidmap} ignoring (usually) harmless EPERM on setxattr "security.capability"
2023/01/17 12:55:09 warn rootless{usr/bin/newuidmap} ignoring (usually) harmless EPERM on setxattr "security.capability"
2023/01/17 12:55:09 warn rootless{usr/bin/ping} ignoring (usually) harmless EPERM on setxattr "security.capability"
2023/01/17 12:55:10 warn rootless{usr/sbin/arping} ignoring (usually) harmless EPERM on setxattr "security.capability"
2023/01/17 12:55:10 warn rootless{usr/sbin/clockdiff} ignoring (usually) harmless EPERM on setxattr "security.capability"
INFO: Creating SIF file...
Apptainer> cat /etc/redhat-release
CentOS Linux release 8.4.2105
```
In this case, the image is downloaded from Docker Hub, extracted to a temporary directory, and Singularity interactive shell is invoked. This procedure can take a lot of time, especially with large images.
In this case, the image is downloaded from Docker Hub, extracted to a temporary directory, and Apptainer interactive shell is invoked. This procedure can take a lot of time, especially with large images.
## Importing Docker Image
Singularity containers can be in three different formats:
Apptainer containers can be in three different formats:
* read-only **squashfs** (default) - best for production
* writable **ext3** (--writable option)
* writable **(ch)root directory** (--sandbox option) - best for development
Squashfs and (ch)root directory images can be built from Docker source directly on the cluster, no root privileges are needed. It is strongly recommended to create a native Singularity image to speed up the launch of the container.
Squashfs and (ch)root directory images can be built from Docker source directly on the cluster, no root privileges are needed. It is strongly recommended to create a native Apptainer image to speed up the launch of the container.
```console
hra0031@login4:~$ ml Singularity
hra0031@login4:~$ singularity build ubuntu.img docker://ubuntu:latest
Docker image path: index.docker.io/library/ubuntu:latest
Cache folder set to /home/hra0031/.singularity/docker
Importing: base Singularity environment
Importing: /home/hra0031/.singularity/docker/sha256:50aff78429b146489e8a6cb9334d93a6d81d5de2edc4fbf5e2d4d9253625753e.tar.gz
Importing: /home/hra0031/.singularity/docker/sha256:f6d82e297bce031a3de1fa8c1587535e34579abce09a61e37f5a225a8667422f.tar.gz
Importing: /home/hra0031/.singularity/docker/sha256:275abb2c8a6f1ce8e67a388a11f3cc014e98b36ff993a6ed1cc7cd6ecb4dd61b.tar.gz
Importing: /home/hra0031/.singularity/docker/sha256:9f15a39356d6fc1df0a77012bf1aa2150b683e46be39d1c51bc7a320f913e322.tar.gz
Importing: /home/hra0031/.singularity/docker/sha256:fc0342a94c89e477c821328ccb542e6fb86ce4ef4ebbf1098e85669e051ef0dd.tar.gz
Importing: /home/hra0031/.singularity/metadata/sha256:c6a9ef4b9995d615851d7786fbc2fe72f72321bee1a87d66919b881a0336525a.tar.gz
WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.
Building Singularity image...
Singularity container built: ubuntu.img
Cleaning up...
$ ml apptainer
$ apptainer build ubuntu.sif docker://ubuntu:latest
INFO: Starting build...
Getting image source signatures
Copying blob 6e3729cf69e0 done
Copying config 415250ec06 done
Writing manifest to image destination
Storing signatures
2023/01/17 12:58:04 info unpack layer: sha256:6e3729cf69e0ce2de9e779575a1fec8b7fb5efdfa822829290ab6d5d1bc3e797
INFO: Creating SIF file...
INFO: Build complete: ubuntu.sif
```
alternatively, you can create your own docker image and import it to singularity.
alternatively, you can create your own docker image and import it to Apptainer.
For example, we show how to create and run ubuntu docker image with gvim installed:
```console
......@@ -83,48 +70,59 @@ your_local_machine $ docker tag 4dd97cefde62 ubuntu_gvim
your_local_machine $ docker save -o ubuntu_gvim.tar ubuntu_gvim
```
copy the `ubuntu_gvim.tar` archive to IT4I supercomputers, convert to Singularity image and run:
copy the `ubuntu_gvim.tar` archive to IT4I supercomputers, convert to Apptainer image and run:
```console
$ ml Singularity
$ singularity build ubuntu_givm.img docker-archive://ubuntu_gvim.tar
$ singularity shell -B /usr/user/$ID ubuntu_gvim.img
$ ml Apptainer
$ apptainer build ubuntu_givm.sif docker-archive://ubuntu_gvim.tar
$ apptainer shell -B /usr/user/$ID ubuntu_gvim.sif
```
Note the bind to `/usr/user/$ID` directory.
## Launching the Container
The interactive shell can be invoked by the `singularity shell` command. This is useful for development purposes. Use the `-w | --writable` option to make changes inside the container permanent.
The interactive shell can be invoked by the `apptainer shell` command. This is useful for development purposes. Use the `-w | --writable` option to make changes inside the container permanent.
```console
hra0031@login4:~$ singularity shell -w ubuntu.img
Singularity: Invoking an interactive shell within container...
$ apptainer shell ubuntu.sif
Singularity ubuntu.img:~> cat /etc/lsb-release
Apptainer> cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"
```
A command can be run inside the container (without an interactive shell) by invoking the `singularity exec` command.
A command can be run inside the container (without an interactive shell) by invoking the `apptainer exec` command.
```
hra0031@login4:~$ singularity exec ubuntu.img cat /etc/lsb-release
$ apptainer exec ubuntu.sif cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS""
```
A singularity image can contain a runscript. This script is executed inside the container after the `singularity run` command is used. The runscript is mostly used to run an application for which the container is built. In the following example, it is the `fortune | cowsay` command:
An Apptainer image can contain a runscript. This script is executed inside the container after the `apptainer run` command is used. The runscript is mostly used to run an application for which the container is built. In the following example, it is the `fortune | cowsay` command:
```
hra0031@login4:~$ singularity run ubuntu.img
___________________
< Are you a turtle? >
-------------------
$ apptainer build lolcow.sif docker://ghcr.io/apptainer/lolcow
INFO: Starting build...
Getting image source signatures
Copying blob 5ca731fc36c2 skipped: already exists
Copying blob 16ec32c2132b skipped: already exists
Copying config fd0daa4d89 done
Writing manifest to image destination
Storing signatures
2023/01/17 13:06:01 info unpack layer: sha256:16ec32c2132b43494832a05f2b02f7a822479f8250c173d0ab27b3de78b2f058
2023/01/17 13:06:01 info unpack layer: sha256:5ca731fc36c28789c5ddc3216563e8bfca2ab3ea10347e07554ebba1c953242e
INFO: Creating SIF file...
INFO: Build complete: lolcow.sif
$ apptainer exec lolcow.sif cowsay moo
_____
< moo >
-----
\ ^__^
\ (oo)\_______
(__)\ )\/\
......@@ -140,15 +138,13 @@ A user home directory is mounted inside the container automatically. If you need
The mounted folder has to exist inside the container or the container image has to be writable!
```console
hra0031@login4:~$ singularity shell -B /scratch -w ubuntu.img
Singularity: Invoking an interactive shell within container...
Singularity ubuntu.img:~> ls /scratch
$ apptainer shell -B /scratch ubuntu.sif
Apptainer> ls /scratch
ddn sys temp work
```
A comprehensive documentation can be found at the [Singularity][c] website.
A comprehensive documentation can be found at the [Apptainer Quick Start][c] website.
[a]: http://singularity.lbl.gov/
[a]: https://apptainer.org/docs/user/latest/introduction.html
[b]: https://hub.docker.com/
[c]: http://singularity.lbl.gov/quickstart
[c]: https://apptainer.org/docs/user/latest/quick_start.html
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment