"...git@code.it4i.cz:davidciz/docs.it4i.cz.git" did not exist on "8f40edf3588341959c7061e77cdafdcacb652541"
CI/CD documentation - UPDATE
Compare changes
Files
2+ 58
− 14
@@ -4,15 +4,17 @@
@@ -4,15 +4,17 @@
Continuous Integration (CI) is the practice of automatically executing a compilation script and set of test cases to ensure that the integrated codebase is in a workable state. The integration is often followed by Continuous Benchmarking (CB) to evaluate the impact of the code change on the application performance and Continuous Deployment (CD) to distribute a new version of the developed code.
Continuous Integration (CI) is the practice of automatically executing a compilation script and set of test cases to ensure that the integrated codebase is in a workable state. The integration is often followed by Continuous Benchmarking (CB) to evaluate the impact of the code change on the application performance and Continuous Deployment (CD) to distribute a new version of the developed code.
IT4I offers its users the possibility to set up CI for their projects to execute their dedicated CI jobs in a customizable virtual environment (docker containers), or test the code directly in computational nodes of the production HPC clusters (Karolina, Barbora) and Complementary systems. The Complementary systems gives a possibility to run the tests on emerging, non-traditional, and highly specialized hardware architectures. It consists of computational nodes built on Intel Sapphire Rapids + HBM, NVIDIA Grace CPU, IBM Power10, A64FX, and many more.
IT4I offers its users a possibility to set up CI for their projects and to execute their dedicated CI jobs directly in computational nodes of the production HPC clusters (Karolina, Barbora) and Complementary systems. The Complementary systems gives a possibility to run the tests on emerging, non-traditional, and highly specialized hardware architectures. It consists of computational nodes built on Intel Sapphire Rapids + HBM, NVIDIA Grace CPU, IBM Power10, A64FX, and many more.
Besides that, there is also a possibility to execute CI jobs in a customizable virtual environment (Docker containers). This allows to test the code in a clean build environment. It also makes dependency management more straight-forward since all dependencies for building the project can be put in the Docker image, from which the corresponding containers are created.
IT4Innovations maintains a GitLab server (code.it4i.cz), which has built-in support for CI/CD. It provides a set of GitLab runners, which is an application that executes jobs specified in the project CI/CD pipelines, consisting of jobs and stages. Grouping jobs together in collections is called stages. Stages run in sequence, while all jobs in a stage can run in parallel.
IT4Innovations maintains a GitLab server (code.it4i.cz), which has built-in support for CI/CD. It provides a set of GitLab runners, which is an application that executes jobs specified in the project CI pipelines, consisting of jobs and stages. Grouping jobs together in collections is called stages. Stages run in sequence, while all jobs in a stage can run in parallel.
For all the users, a unified solution is provided to let them execute their CI jobs at Karolina, Barbora, and Complementary systems without the need to create their own project runners. For each of the HPC clusters, a GitLab instance runner has been deployed. The runners are running in the login nodes and are visible to all the projects of the IT4I GitLab server. These runners are shared by all users.
For all the users, a unified solution is provided to let them execute their CI jobs at Karolina, Barbora, and Complementary systems without the need to create their own project runners. For each of the HPC clusters, a GitLab instance runner has been deployed. The runners are running in the login nodes and are visible to all the projects of the IT4I GitLab server. These runners are shared by all users.
@@ -20,20 +22,31 @@ These runners are using **Jacamar CI driver** – an HPC-focused open-source CI/
@@ -20,20 +22,31 @@ These runners are using **Jacamar CI driver** – an HPC-focused open-source CI/
The execution of CI pipelines works as follows. First, a user in the IT4I GitLab server triggers a CI pipeline (for example, by making push to a repository, etc.). Then, the jobs, which the pipeline consists of, are sent to the corresponding runner, running in the login node. Lastly, for every CI job, the runner clones the repository (or just fetches changes to an already cloned one, if there are any) and submits the job as a Slurm job to the corresponding HPC cluster using the `sbatch` command. After each execution of a job, the runner reports the results back to the server and uploads the artifacts (if specified).
The execution of CI pipelines works as follows. First, a user in the IT4I GitLab server triggers a CI pipeline (for example, by making push to a repository, etc.). Then, the jobs, which the pipeline consists of, are sent to the corresponding runner, running in the login node. Lastly, for every CI job, the runner clones the repository (or just fetches changes to an already cloned one, if there are any), restores [cache][3], downloads [artifacts][4] (if specified), and submits the job as a Slurm job to the corresponding HPC cluster using the `sbatch` command. After each execution of a job, the runner reports the results back to the server, creates cache, and uploads artifacts (if specified).
The GitLab runners at Karolina and Barbora are able to submit (as a Slurm job) and execute 32 CI jobs concurrently, while the runner at Complementary systems can submit 16 jobs concurrently at most. Jobs above this limit are postponed in submission to respective slurm queue until a previous job has finished.
The GitLab runners at Karolina and Barbora are able to submit (as a Slurm job) and execute 32 CI jobs concurrently, while the runner at Complementary systems can submit 16 jobs concurrently at most. Jobs above this limit are postponed in submission to respective slurm queue until a previous job has finished.
There are also 5 GitLab instance runners with Docker executor configured, which have been deployed in the local virtual infrastructure (each runs in a dedicated virtual machine). The runners use Docker Engine to execute each job in a separate and isolated container created from the image specified beforehand. These runners are also visible to all the projects of the IT4I GitLab server.
In addition, these runners have distributed caching enabled. This feature uses pre-configured object storage server and allows to share the [cache][3] between subsequent CI jobs (of the same project) executed on multiple runners (2 or more of the 5 deployed). Refer to [Caching in GitLab CI/CD][6] for information about cache and how cache is different from artifacts.
To begin with, a CI pipeline of a project must be defined in a YAML file. The most common name of this file is `.gitlab-ci.yml` and it should be located in the repository top level. For detailed information, see [tutorial][3] on how to create your first pipeline. Additionally, [CI/CD YAML syntax reference][4] lists all possible keywords, that can be specified in the definition of CI/CD pipelines and jobs.
To begin with, a CI pipeline of a project must be defined in a YAML file. The most common name of this file is `.gitlab-ci.yml` and it should be located in the repository top level. For detailed information, see [tutorial][7] on how to create your first pipeline. Additionally, [CI/CD YAML syntax reference][8] lists all possible keywords, that can be specified in the definition of CI/CD pipelines and jobs.
Every CI job in the project CI pipeline, intended to be submitted as a Slurm job to one of the HPC clusters, must have 3 following keywords specified in its definition.
* `id_tokens`, in which `SITE_ID_TOKEN` must be defined with `aud` set to the URL of IT4I GitLab server.
@@ -56,7 +69,7 @@ tags:
@@ -56,7 +69,7 @@ tags:
Optionally, a custom build directory can also be specified. The deployed GitLab runners are configured to store all files and directories for the CI job in the home directory of the user, who triggers the associated CI pipeline (the repository is also cloned there in a unique subpath). This behavior can be changed by specifying the `CUSTOM_CI_BUILDS_DIR` variable in the `variables` clause of the CI job.
Optionally, a custom build directory can also be specified. The deployed GitLab runners are configured to store all files and directories for the CI job in the home directory of the user, who triggers the associated CI pipeline (the repository is also cloned there in a unique subpath). This behavior can be changed by specifying the `CUSTOM_CI_BUILDS_DIR` variable in the `variables` clause of the CI job.
@@ -67,10 +80,41 @@ variables:
@@ -67,10 +80,41 @@ variables: