Skip to content
Snippets Groups Projects
singularity-it4i.md 8.18 KiB
Newer Older
  • Learn to ignore specific revisions
  • # Apptainer on IT4Innovations
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    !!!note "Singularity name change"
        On November 30th, 2021, the Singularity project has officially moved into the Linux Foundation. As part of this move, and to differentiate from the other like-named projects and commercial products, the project is renamed to "Apptainer".
    
    
    On our clusters, the Apptainer images of main Linux distributions are prepared.
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    ```console
    
    Barbora             Karolina
     ├── CentOS          ├── CentOS
     |    └── 7          |    └── 7
     ├── Rocky           ├── Rocky
     |    ├── 8          |    ├── 8
     │    └── 9          │    └── 9
     ├── Fedora          ├── Fedora
     │    └── latest     │    └── latest
     └── Ubuntu          └── Ubuntu
          └── latest          └── latest
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    ```
    
    !!! info
    
        Current information about available Apptainer images can be obtained by the `ml av` command. The images are listed in the `OS` section.
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    The bootstrap scripts, wrappers, features, etc. are located on [it4i-singularity GitLab page][a].
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    
    ## IT4Innovations Apptainer Wrappers
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    
    For better user experience with Apptainer containers, we prepared several wrappers:
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    * image-exec
    * image-mpi
    * image-run
    * image-shell
    * image-update
    
    
    Listed wrappers help you to use prepared Apptainer images loaded as modules.
    You can easily load a Apptainer image like any other module on the cluster by the `ml OS/version` command.
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    After the module is loaded for the first time, the prepared image is copied into your home folder and is ready for use.
    When you load the module next time, the version of the image is checked and an image update (if exists) is offered.
    Then you can update your copy of the image by the `image-update` command.
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    !!! warning
        With an image update, all user changes to the image will be overridden.
    
    
    The runscript inside the Apptainer image can be run by the `image-run` command.
    
    !!! note " CentOS/7 module only"
        This command automatically mounts the `/scratch` and `/apps` storage and invokes the image as writable, so user changes can be made.
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    Very similar to `image-run` is the `image-exec` command.
    The only difference is that `image-exec` runs a user-defined command instead of a runscript.
    In this case, the command to be run is specified as a parameter.
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    
    Using the interactive shell inside the Apptainer container is very useful for development.
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    In this interactive shell, you can make any changes to the image you want,
    but be aware that you can not use the `sudo` privileged commands directly on the cluster.
    To simply invoke interactive shell, use the `image-shell` command.
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    
    Another useful feature of the Apptainer is the direct support of OpenMPI.
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    For proper MPI function, you have to install the same version of OpenMPI inside the image as you use on the cluster.
    
    OpenMPI/4.1.2 is installed in prepared images  (CentOS 7, Rocky 8).
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    The MPI must be started outside the container.
    The easiest way to start the MPI is to use the `image-mpi` command.
    This command has the same parameters as `mpirun`, so there is no difference between running normal MPI application
    
    and MPI application in a Apptainer container.
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    ## Examples
    
    
    In the examples, we will use prepared Apptainer images.
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    ### Load Image
    
    ```console
    
    $ ml CentOS/7
    Preparing image CentOS-7_20230116143612.sif
            261.20M 100%  412.36MB/s    0:00:00 (xfr#1, to-chk=0/1)
    Your image of CentOS/7 is at location: /home/username/.apptainer/images/CentOS-7_20230116143612.sif
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    ```
    
    !!! tip
    
        After the module is loaded for the first time, the prepared image is copied into your home folder to the *.apptainer/images* subfolder.
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    ### Wrappers
    
    **image-exec**
    
    
    Executes the given command inside the Apptainer image. The container is in this case started, then the command is executed and the container is stopped.
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    ```console
    $ ml CentOS/7
    
    $ image-exec cat /etc/redhat-release
    CentOS Linux release 7.9.2009 (Core)
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    ```
    
    **image-mpi**
    
    MPI wrapper - see more in the [Examples MPI][1] section.
    
    **image-run**
    
    
    This command runs the runscript inside the Apptainer image. Note, that the prepared images do not contain a runscript.
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    **image-shell**
    
    
    Invokes an interactive shell inside the Apptainer image.
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    ```console
    $ ml CentOS/7
    $ image-shell
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    ```
    
    ### Update Image
    
    
    This command is for updating your local Apptainer image copy.
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    The local copy is overridden in this case.
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    ```console
    
    $ ml CentOS/7
    New version of CentOS image was found. (New: CentOS-7_20230116143612.sif Old: CentOS-7_20230115143612.sif)
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    For updating image use: image-update
    
    Your image of CentOS/7 is at location: /home/username/.apptainer/images/CentOS-7_20230115143612.sif
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    $ image-update
    
    New version of CentOS image was found. (New: CentOS-7_20230116143612.sif Old: CentOS-7_20230115143612.sif)
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    Do you want to update local copy? (WARNING all user modification will be deleted) [y/N]: y
    
    Updating image  CentOS-7_20230116143612.sif
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
           2.71G 100%  199.49MB/s    0:00:12 (xfer#1, to-check=0/1)
    
    sent 2.71G bytes  received 31 bytes  163.98M bytes/sec
    total size is 2.71G  speedup is 1.00
    
    New version is ready. (/home/username/.apptainer/images/CentOS-7_20230116143612.sif)
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    In the following example, we are using a job submitted by the command:
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    ```bash
    $ salloc -A PROJECT_ID -p qcpu --nodes=2 --ntasks-per-node=128 --time=00:30:00
    ```
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    !!! note
    
        We have seen no major performance impact for a job running in a Apptainer container.
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    
    With Apptainer, the MPI usage model is to call `mpirun` from outside the container
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    and reference the container from your `mpirun` command.
    Usage would look like this:
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    ```console
    
    $ mpirun -np 128 apptainer exec container.img /path/to/contained_mpi_prog
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    By calling `mpirun` outside of the container, we solve several very complicated work-flow aspects.
    For example, if `mpirun` is called from within the container, it must have a method for spawning processes on remote nodes.
    Historically the SSH is used for this, which means that there must be an `sshd` running within the container on the remote nodes
    and this `sshd` process must not conflict with the `sshd` running on that host.
    It is also possible for the resource manager to launch the job
    and (in OpenMPI’s case) the Orted (Open RTE User-Level Daemon) processes on the remote system,
    but that then requires resource manager modification and container awareness.
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    In the end, we do not gain anything by calling `mpirun` from within the container
    except for increasing the complexity levels and possibly losing out on some added
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    performance benefits (e.g. if a container was not built with the proper OFED as the host).
    
    
    #### MPI Inside Apptainer Image
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    ```console
    $ ml CentOS/7
    $ image-shell
    
    Apptainer> mpirun hostname | wc -l
    128
    
    As you can see in this example, we allocated two nodes, but MPI can use only one node (128 processes) when used inside the Apptainer image.
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    
    #### MPI Outside Apptainer Image
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    ```console
    $ ml CentOS/7
    
    Your image of CentOS/7 is at location: /home/username/.apptainer/images/CentOS-7_20230116143612.sif
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    $ image-mpi hostname | wc -l
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    In this case, the MPI wrapper behaves like the `mpirun` command.
    The `mpirun` command is called outside the container
    and the communication between nodes are propagated into the container automatically.
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    ## How to Use Own Image on Cluster?
    
    * Prepare the image on your computer
    
    * Transfer the images to your `/home` directory on the cluster (for example `.apptainer/image`)
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    ```console
    
    local:$ scp container.img login@login2.clustername.it4i.cz:~/.apptainer/image/container.img
    
    * Load module Apptainer (`ml apptainer`)
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    * Use your image
    
    !!! note
    
        If you want to use the Apptainer wrappers with your own images, load the `apptainer-wrappers/1.0` module and set the environment variable `IMAGE_PATH_LOCAL=/path/to/container.img`.
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    
    ## How to Edit IT4Innovations Image?
    
    * Transfer the image to your computer
    
    ```console
    
    local:$ scp login@login2.clustername.it4i.cz:/home/username/.apptainer/image/container.img container.img
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    ```
    
    * Modify the image
    * Transfer the image from your computer to your `/home` directory on the cluster
    
    ```console
    
    local:$ scp container.img login@login2.clustername.it4i.cz:/home/username/.apptainer/image/container.img
    
    * Load module Apptainer (`ml apptainer`)
    
    Lukáš Krupčík's avatar
    Lukáš Krupčík committed
    * Use your image
    
    [1]: #mpi
    
    [a]: https://code.it4i.cz/sccs/it4i-singularity