From bc50e2543db642a0eceaf6d1348dc0aa145b60e8 Mon Sep 17 00:00:00 2001 From: Josef Hrabal <josef.hrabal@vsb.cz> Date: Tue, 19 Dec 2017 15:39:14 +0100 Subject: [PATCH] Singularity documentation reworked for version 2.4 --- docs.it4i/software/tools/singularity.md | 173 +++++++++++------------- 1 file changed, 80 insertions(+), 93 deletions(-) diff --git a/docs.it4i/software/tools/singularity.md b/docs.it4i/software/tools/singularity.md index 1e3ff9a18..85361121b 100644 --- a/docs.it4i/software/tools/singularity.md +++ b/docs.it4i/software/tools/singularity.md @@ -9,122 +9,109 @@ Singularity also allows you to leverage the resources of whatever host you are o * No user contextual changes or root escalation allowed * No root owned daemon processes +These documentation are for Singularity Version 2.4 and newer. + ## Using Docker Images Singularity can import, bootstrap, and even run Docker images directly from [Docker Hub](https://hub.docker.com/). You can easily run RHEL7 like this: ```console -[hrb33@r33u01n865 ~]$ cat /etc/redhat-release -CentOS release 6.7 (Final) -[hrb33@r33u01n865 ~]$ ml Singularity -[hrb33@r33u01n865 ~]$ singularity shell docker://centos:latest -library/centos:latest -Downloading layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 -Downloading layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 -Downloading layer: sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729 -Downloading layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 +hra0031@login4:~$ cat /etc/redhat-release +CentOS release 6.9 (Final) +hra0031@login4:~$ ml Singularity +hra0031@login4:~$ singularity shell docker://centos:latest +Docker image path: index.docker.io/library/centos:latest +Cache folder set to /home/hra0031/.singularity/docker +[1/1] |===================================| 100.0% +Creating container runtime... Singularity: Invoking an interactive shell within container... -Singularity.centos:latest> cat /etc/redhat-release -CentOS Linux release 7.3.1611 (Core) +Singularity centos:latest:~> cat /etc/redhat-release +CentOS Linux release 7.4.1708 (Core) ``` +In this case, Image is downloaded from Docker Hub, extracted to a temporary directory and singularity interactive shell is invoked. This procedure can take a lot of time, especially for larger images. + ## Creating Own Image From Docker Image +Singularity containers can be in three different formats: +* read-only **squashfs** (default) - best for production +* writable **ext3** (--writable option) +* writable **(ch)root directory** (--sandbox option) - best for development + +Squashfs and (ch)root directory image can be built from docker source directly on the cluster, no root privileges are needed. Creating native singularity image is strongly recommended due to the speed of launching the container. + +```console +hra0031@login4:~$ ml Singularity +hra0031@login4:~$ singularity build ubuntu.img docker://ubuntu:latest +Docker image path: index.docker.io/library/ubuntu:latest +Cache folder set to /home/hra0031/.singularity/docker +Importing: base Singularity environment +Importing: /home/hra0031/.singularity/docker/sha256:50aff78429b146489e8a6cb9334d93a6d81d5de2edc4fbf5e2d4d9253625753e.tar.gz +Importing: /home/hra0031/.singularity/docker/sha256:f6d82e297bce031a3de1fa8c1587535e34579abce09a61e37f5a225a8667422f.tar.gz +Importing: /home/hra0031/.singularity/docker/sha256:275abb2c8a6f1ce8e67a388a11f3cc014e98b36ff993a6ed1cc7cd6ecb4dd61b.tar.gz +Importing: /home/hra0031/.singularity/docker/sha256:9f15a39356d6fc1df0a77012bf1aa2150b683e46be39d1c51bc7a320f913e322.tar.gz +Importing: /home/hra0031/.singularity/docker/sha256:fc0342a94c89e477c821328ccb542e6fb86ce4ef4ebbf1098e85669e051ef0dd.tar.gz +Importing: /home/hra0031/.singularity/metadata/sha256:c6a9ef4b9995d615851d7786fbc2fe72f72321bee1a87d66919b881a0336525a.tar.gz +WARNING: Building container as an unprivileged user. If you run this container as root +WARNING: it may be missing some functionality. +Building Singularity image... +Singularity container built: ubuntu.img +Cleaning up... +``` + +## Launching the Container + +The interactive shell can be invoked by the **shell** command. This is useful for development purposes. To make changes inside the container permanent use **-w | --writable** option. + ```console -hrb33@hrb33-toshiba:/$ cd /tmp/ -hrb33@hrb33-toshiba:/tmp$ sudo singularity create /tmp/c7.img -[sudo] password for hrb33: -Creating a new image with a maximum size of 768MiB... -Executing image create helper -Formatting image with ext3 file system -Done. -hrb33@hrb33-toshiba:/tmp$ sudo singularity import c7.img docker://centos:latest -library/centos:latest -Downloading layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 -Downloading layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 -Downloading layer: sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729 -Downloading layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 -Adding Docker CMD as Singularity runscript... -Bootstrap initialization -No bootstrap definition passed, updating container -Executing Prebootstrap module -Executing Postbootstrap module -Done. -hrb33@hrb33-toshiba:/tmp$ sudo singularity shell --writable c7.img +hra0031@login4:~$ singularity shell -w ubuntu.img Singularity: Invoking an interactive shell within container... -Singularity.c7.img> mkdir /apps /scratch -Singularity.c7.img> exit -hrb33@hrb33-toshiba:/tmp$ rsync -av c7.img hrb33@login4.salomon:/home/hrb33/c7.img -sending incremental file list -c7.img +Singularity ubuntu.img:~> cat /etc/lsb-release +DISTRIB_ID=Ubuntu +DISTRIB_RELEASE=16.04 +DISTRIB_CODENAME=xenial +DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS" +``` -sent 805,503,090 bytes received 34 bytes 9,205,749.99 bytes/sec -total size is 805,306,399 speedup is 1.00 +To execute command inside container (without interactive shell) use **exec** command. +``` +hra0031@login4:~$ singularity exec ubuntu.img cat /etc/lsb-release +DISTRIB_ID=Ubuntu +DISTRIB_RELEASE=16.04 +DISTRIB_CODENAME=xenial +DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS" ``` -Accessing /HOME and /SCRATCH Within Container +Singularity image can contain a runscript. This script is executed inside container after the **run** command is used. The runscript is mostly used to run an application for which is the container build, in the following example it is **fortune | cowsay** command. -```console -hrb33@hrb33-toshiba:/tmp$ ssh hrb33@login4.salomon +``` +hra0031@login4:~$ singularity run ubuntu.img + ___________________ +< Are you a turtle? > + ------------------- + \ ^__^ + \ (oo)\_______ + (__)\ )\/\ + ||----w | + || || +``` - _____ _ - / ____| | | - | (___ __ _| | ___ _ __ ___ ___ _ __ - \___ \ / _` | |/ _ \| '_ ` _ \ / _ \| '_ \ - ____) | (_| | | (_) | | | | | | (_) | | | | - |_____/ \__,_|_|\___/|_| |_| |_|\___/|_| |_| +## Accessing /HOME and /SCRATCH Within Container - http://www.it4i.cz/?lang=en +By default, user home directory is mounted inside the container, therefore you can access your files directly. If you need access to **/SCRATCH** storage for your computation, this must by mounted be **-B | --bind** option. +!!!Warning + Be aware, that the mounted folder has to exist inside the container or the container image has to be writable! -Last login: Fri Feb 10 14:38:36 2017 from 10.0.131.12 -[hrb33@login4.salomon ~]$ ml Singularity -[hrb33@login4.salomon ~]$ singularity shell --bind /scratch --bind /apps --writable c7.img +```console +hra0031@login4:~$ singularity shell -B /scratch -w ubuntu.img Singularity: Invoking an interactive shell within container... -Singularity.c7.img> ls /apps/ -l -total 68 -drwx------ 4 root root 29 Sep 29 10:28 SCS -drwxrwxr-x 301 2757 2796 8192 Feb 16 10:58 all -drwxrwxr-x 3 2757 2796 19 Jul 9 2015 base -drwxrwxr-x 16 2757 2796 4096 Nov 24 21:47 bio -drwxrwxr-x 10 2757 2796 116 Apr 8 2016 cae -drwxrwxr-x 18 2757 2796 4096 Jan 17 09:49 chem -drwxrwxr-x 11 2757 2796 122 Dec 7 09:25 compiler -drwxrwxr-x 7 2757 2796 73 Jun 29 2016 data -drwxr-xr-x 7 2757 2796 88 Jan 8 2016 debugger -drwxrwxr-x 38 2757 2796 4096 Feb 16 13:37 devel -drwxrwxr-x 9 2757 2796 130 Jan 9 08:40 easybuild -drwxr-xr-x 11 3900 4011 4096 Feb 15 09:50 gentoo -drwxr-xr-x 10 3900 4011 4096 Feb 10 17:01 gentoo_uv -drwxrwxr-x 5 2757 2796 39 Jan 18 2016 geo -drwxr-xr-x 18 2757 2796 4096 Sep 6 16:03 intel2017 -drwxrwxr-x 20 2757 2796 4096 Nov 28 08:50 lang -drwxrwxr-x 31 2757 2796 4096 Dec 7 07:48 lib -drwxrwxr-x 4 2757 2796 32 Nov 9 09:19 licenses -drwxrwxr-x 17 2757 2796 4096 Nov 15 09:24 math -drwxr-xr-x 22 2757 2796 4096 Jan 19 13:15 modules -drwxrwxr-x 8 2757 2796 82 Apr 18 2016 mpi -drwxrwxr-x 13 2757 2796 4096 Oct 24 09:08 numlib -drwxrwxr-x 10 2757 2796 108 Feb 3 11:01 perf -drwxrwxr-x 5 2757 2796 41 Jan 17 09:49 phys -drwxrwxr-x 2 2757 2796 6 Feb 3 11:01 prace -drwxr-xr-x 4 root root 36 Jun 18 2015 sw -drwxrwxr-x 5 2757 2796 49 Feb 15 2016 system -drwxr-xr-x 3 root root 19 Dec 4 2015 test -drwxrwxr-x 13 2757 2796 138 May 31 2016 toolchain -drwxrwxr-x 39 2757 2796 4096 Feb 3 11:27 tools -drwxr-xr-x 4 root root 31 Aug 11 2015 user -drwxrwxr-x 21 2757 2796 4096 Jan 5 18:56 uv -drwxrwxr-x 40 2757 2796 4096 Feb 3 11:01 vis -Singularity.c7.img> ls /scratch/ -l -total 32 -drwx------ 3 root root 4096 Aug 15 2016 backup -drwxr-x--- 2 root root 4096 Dec 5 10:34 sys -drwxrwxrwt 154 root root 20480 Feb 14 14:03 temp -drwxr-xr-x 4 root root 4096 Jan 25 10:48 work -Singularity.c7.img> +Singularity ubuntu.img:~> ls /scratch +ddn sys temp work ``` + +Comprehensive documentation can be found at the [Singularity](http://singularity.lbl.gov/quickstart) website. \ No newline at end of file -- GitLab