Skip to content
Snippets Groups Projects
Commit 97ff6034 authored by Lukáš Krupčík's avatar Lukáš Krupčík
Browse files

Update docs.it4i/software/tools/singularity.md

parent 61426102
No related branches found
Tags
1 merge request!419Update docs.it4i/software/tools/singularity-it4i.md
Pipeline #29524 passed with warnings
# Singularity Container
# Apptainer Container
[Singularity][a] enables users to have full control of their environment. A non-privileged user can "swap out" the operating system on the host for one they control. So if the host system is running RHEL6 but your application runs in Ubuntu/RHEL7, you can create an Ubuntu/RHEL7 image, install your applications into that image, copy the image to another host, and run your application on that host in its native Ubuntu/RHEL7 environment.
Singularity also allows you to leverage the resources of whatever host you are on. This includes HPC interconnects, resource managers, file systems, GPUs and/or accelerators, etc. Singularity does this by enabling several key facets:
* Encapsulation of the environment
* Containers are image based
* No user contextual changes or root escalation allowed
* No root owned daemon processes
This documentation is for Singularity version 2.4 and newer.
<div align="center">
<iframe width="250" height="50%" src="https://www.youtube.com/embed/m8llDjFuXlc" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<iframe width="250" height="50%" src="https://www.youtube.com/embed/SJHizTjwyFk" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<iframe width="250" height="50%" src="https://www.youtube.com/embed/97VuBVnfcwg" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<iframe width="250" height="50%" src="https://www.youtube.com/embed/wGJnkRmW5iU" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
</div>
[Apptainer][a] is a container platform. It allows you to create and run containers that package up pieces of software in a way that is portable and reproducible. You can build a container using Apptainer on your laptop, and then run it on many of the largest HPC clusters in the world, local university or company clusters, a single server, in the cloud, or on a workstation down the hall. Your container is a single file, and you don’t have to worry about how to install all the software you need on each different operating system.
## Using Docker Images
Singularity can import, bootstrap, and even run Docker images directly from [Docker Hub][b]. You can easily run an RHEL7 container like this:
Apptainer can import, bootstrap, and even run Docker images directly from [Docker Hub][b]. You can easily run an CentOS container like this:
```console
hra0031@login4:~$ cat /etc/redhat-release
CentOS release 6.9 (Final)
hra0031@login4:~$ ml Singularity
hra0031@login4:~$ singularity shell docker://centos:latest
Docker image path: index.docker.io/library/centos:latest
Cache folder set to /home/hra0031/.singularity/docker
[1/1] |===================================| 100.0%
Creating container runtime...
Singularity: Invoking an interactive shell within container...
Singularity centos:latest:~> cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
$ cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)
$ ml apptainer
$ apptainer shell docker://centos:latest
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob a1d0c7532777 done
Copying config 8c1402b22a done
Writing manifest to image destination
Storing signatures
2023/01/17 12:55:08 info unpack layer: sha256:a1d0c75327776413fa0db9ed3adcdbadedc95a662eb1d360dad82bb913f8a1d1
2023/01/17 12:55:09 warn rootless{usr/bin/newgidmap} ignoring (usually) harmless EPERM on setxattr "security.capability"
2023/01/17 12:55:09 warn rootless{usr/bin/newuidmap} ignoring (usually) harmless EPERM on setxattr "security.capability"
2023/01/17 12:55:09 warn rootless{usr/bin/ping} ignoring (usually) harmless EPERM on setxattr "security.capability"
2023/01/17 12:55:10 warn rootless{usr/sbin/arping} ignoring (usually) harmless EPERM on setxattr "security.capability"
2023/01/17 12:55:10 warn rootless{usr/sbin/clockdiff} ignoring (usually) harmless EPERM on setxattr "security.capability"
INFO: Creating SIF file...
Apptainer> cat /etc/redhat-release
CentOS Linux release 8.4.2105
```
In this case, the image is downloaded from Docker Hub, extracted to a temporary directory, and Singularity interactive shell is invoked. This procedure can take a lot of time, especially with large images.
In this case, the image is downloaded from Docker Hub, extracted to a temporary directory, and Apptainer interactive shell is invoked. This procedure can take a lot of time, especially with large images.
## Importing Docker Image
Singularity containers can be in three different formats:
Apptainer containers can be in three different formats:
* read-only **squashfs** (default) - best for production
* writable **ext3** (--writable option)
* writable **(ch)root directory** (--sandbox option) - best for development
Squashfs and (ch)root directory images can be built from Docker source directly on the cluster, no root privileges are needed. It is strongly recommended to create a native Singularity image to speed up the launch of the container.
Squashfs and (ch)root directory images can be built from Docker source directly on the cluster, no root privileges are needed. It is strongly recommended to create a native Apptainer image to speed up the launch of the container.
```console
hra0031@login4:~$ ml Singularity
hra0031@login4:~$ singularity build ubuntu.img docker://ubuntu:latest
Docker image path: index.docker.io/library/ubuntu:latest
Cache folder set to /home/hra0031/.singularity/docker
Importing: base Singularity environment
Importing: /home/hra0031/.singularity/docker/sha256:50aff78429b146489e8a6cb9334d93a6d81d5de2edc4fbf5e2d4d9253625753e.tar.gz
Importing: /home/hra0031/.singularity/docker/sha256:f6d82e297bce031a3de1fa8c1587535e34579abce09a61e37f5a225a8667422f.tar.gz
Importing: /home/hra0031/.singularity/docker/sha256:275abb2c8a6f1ce8e67a388a11f3cc014e98b36ff993a6ed1cc7cd6ecb4dd61b.tar.gz
Importing: /home/hra0031/.singularity/docker/sha256:9f15a39356d6fc1df0a77012bf1aa2150b683e46be39d1c51bc7a320f913e322.tar.gz
Importing: /home/hra0031/.singularity/docker/sha256:fc0342a94c89e477c821328ccb542e6fb86ce4ef4ebbf1098e85669e051ef0dd.tar.gz
Importing: /home/hra0031/.singularity/metadata/sha256:c6a9ef4b9995d615851d7786fbc2fe72f72321bee1a87d66919b881a0336525a.tar.gz
WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.
Building Singularity image...
Singularity container built: ubuntu.img
Cleaning up...
$ ml apptainer
$ apptainer build ubuntu.sif docker://ubuntu:latest
INFO: Starting build...
Getting image source signatures
Copying blob 6e3729cf69e0 done
Copying config 415250ec06 done
Writing manifest to image destination
Storing signatures
2023/01/17 12:58:04 info unpack layer: sha256:6e3729cf69e0ce2de9e779575a1fec8b7fb5efdfa822829290ab6d5d1bc3e797
INFO: Creating SIF file...
INFO: Build complete: ubuntu.sif
```
alternatively, you can create your own docker image and import it to singularity.
alternatively, you can create your own docker image and import it to Apptainer.
For example, we show how to create and run ubuntu docker image with gvim installed:
```console
......@@ -83,48 +70,59 @@ your_local_machine $ docker tag 4dd97cefde62 ubuntu_gvim
your_local_machine $ docker save -o ubuntu_gvim.tar ubuntu_gvim
```
copy the `ubuntu_gvim.tar` archive to IT4I supercomputers, convert to Singularity image and run:
copy the `ubuntu_gvim.tar` archive to IT4I supercomputers, convert to Apptainer image and run:
```console
$ ml Singularity
$ singularity build ubuntu_givm.img docker-archive://ubuntu_gvim.tar
$ singularity shell -B /usr/user/$ID ubuntu_gvim.img
$ ml Apptainer
$ apptainer build ubuntu_givm.sif docker-archive://ubuntu_gvim.tar
$ apptainer shell -B /usr/user/$ID ubuntu_gvim.sif
```
Note the bind to `/usr/user/$ID` directory.
## Launching the Container
The interactive shell can be invoked by the `singularity shell` command. This is useful for development purposes. Use the `-w | --writable` option to make changes inside the container permanent.
The interactive shell can be invoked by the `apptainer shell` command. This is useful for development purposes. Use the `-w | --writable` option to make changes inside the container permanent.
```console
hra0031@login4:~$ singularity shell -w ubuntu.img
Singularity: Invoking an interactive shell within container...
$ apptainer shell ubuntu.sif
Singularity ubuntu.img:~> cat /etc/lsb-release
Apptainer> cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"
```
A command can be run inside the container (without an interactive shell) by invoking the `singularity exec` command.
A command can be run inside the container (without an interactive shell) by invoking the `apptainer exec` command.
```
hra0031@login4:~$ singularity exec ubuntu.img cat /etc/lsb-release
$ apptainer exec ubuntu.sif cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS""
```
A singularity image can contain a runscript. This script is executed inside the container after the `singularity run` command is used. The runscript is mostly used to run an application for which the container is built. In the following example, it is the `fortune | cowsay` command:
A Apptainer image can contain a runscript. This script is executed inside the container after the `apptainer run` command is used. The runscript is mostly used to run an application for which the container is built. In the following example, it is the `fortune | cowsay` command:
```
hra0031@login4:~$ singularity run ubuntu.img
___________________
< Are you a turtle? >
-------------------
$ apptainer build lolcow.sif docker://ghcr.io/apptainer/lolcow
INFO: Starting build...
Getting image source signatures
Copying blob 5ca731fc36c2 skipped: already exists
Copying blob 16ec32c2132b skipped: already exists
Copying config fd0daa4d89 done
Writing manifest to image destination
Storing signatures
2023/01/17 13:06:01 info unpack layer: sha256:16ec32c2132b43494832a05f2b02f7a822479f8250c173d0ab27b3de78b2f058
2023/01/17 13:06:01 info unpack layer: sha256:5ca731fc36c28789c5ddc3216563e8bfca2ab3ea10347e07554ebba1c953242e
INFO: Creating SIF file...
INFO: Build complete: lolcow.sif
$ apptainer exec lolcow.sif cowsay moo
_____
< moo >
-----
\ ^__^
\ (oo)\_______
(__)\ )\/\
......@@ -140,15 +138,13 @@ A user home directory is mounted inside the container automatically. If you need
The mounted folder has to exist inside the container or the container image has to be writable!
```console
hra0031@login4:~$ singularity shell -B /scratch -w ubuntu.img
Singularity: Invoking an interactive shell within container...
Singularity ubuntu.img:~> ls /scratch
$ apptainer shell -B /scratch ubuntu.sif
Apptainer> ls /scratch
ddn sys temp work
```
A comprehensive documentation can be found at the [Singularity][c] website.
A comprehensive documentation can be found at the [Apptainer Quick Start][c] website.
[a]: http://singularity.lbl.gov/
[a]: https://apptainer.org/docs/user/latest/introduction.html
[b]: https://hub.docker.com/
[c]: http://singularity.lbl.gov/quickstart
[c]: https://apptainer.org/docs/user/latest/quick_start.html
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment