From b157d5dd5ae0346718585a86e0ac3fffa514599f Mon Sep 17 00:00:00 2001
From: Jan Siwiec <jan.siwiec@vsb.cz>
Date: Fri, 5 Nov 2021 10:54:45 +0100
Subject: [PATCH 1/7] Update mkdocs.yml
---
mkdocs.yml | 1 +
1 file changed, 1 insertion(+)
diff --git a/mkdocs.yml b/mkdocs.yml
index 27cf4c858..417d191bd 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -59,6 +59,7 @@ nav:
- Job Priority: general/job-priority.md
- Job Submission and Execution: general/job-submission-and-execution.md
- Capacity Computing: general/capacity-computing.md
+ - Migrating from SLURM: general/slurmtopbs.md
- Connect to the Clusters:
- OpenSSH Keys (UNIX): general/accessing-the-clusters/shell-access-and-data-transfer/ssh-keys.md
- PuTTY (Windows): general/accessing-the-clusters/shell-access-and-data-transfer/putty.md
--
GitLab
From 37301493a3c487c481154e13c825498c655f2755 Mon Sep 17 00:00:00 2001
From: Jan Siwiec <jan.siwiec@vsb.cz>
Date: Fri, 5 Nov 2021 11:10:52 +0100
Subject: [PATCH 2/7] Add new file
---
docs.it4i/general/slurmtopbs.md | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
create mode 100644 docs.it4i/general/slurmtopbs.md
diff --git a/docs.it4i/general/slurmtopbs.md b/docs.it4i/general/slurmtopbs.md
new file mode 100644
index 000000000..ad0d06d60
--- /dev/null
+++ b/docs.it4i/general/slurmtopbs.md
@@ -0,0 +1,24 @@
+# Migrating from SLURM
+
+SLURM-optimized distributed jobs do not run on Karolina out of the box.
+
+In typical multi-gpu multi-node setting one needs to know:
+
+- World-size - i.e. the total number of GPUs in the system
+- Rank of a job in the world - i.e. the number of the current GPU in the system
+- Local GPU ID for assignment purposes inside PyTorch/TF
+
+The combination of PBS and MPI does not require large changes in the codebase so you can test and debug large-scale jobs on the CIIRC cluster and deploy them on Karolina without much of a change.
+
+The following example assumes that you use PyTorch and your `DistributedDataParallel` process is being initialized via `init_method` pointing to some file.
+
+The required changes are:
+
+- To get the world size, access the `PMI_SIZE` variable of the MPI
+- To get the process rank in the world, access `PMI_RANK` variable of the MPI
+- To get local GPU ID on the node (can be used to manually set `CUDA_VISIBLE_DEVICES`), access the `MPI_LOCALRANKID` variable.
+
+!!! hint
+ Some jobs may greatly benefit from [mixed precision training][1] since the new NVIDIA A100 GPUs have excellent support for it.
+
+[1]: https://pytorch.org/docs/stable/amp.html
--
GitLab
From eaaa03ffbba858a86de51b1c4d179ee162474d7d Mon Sep 17 00:00:00 2001
From: Jan Siwiec <jan.siwiec@vsb.cz>
Date: Fri, 5 Nov 2021 11:12:12 +0100
Subject: [PATCH 3/7] Update slurmtopbs.md
---
docs.it4i/general/slurmtopbs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs.it4i/general/slurmtopbs.md b/docs.it4i/general/slurmtopbs.md
index ad0d06d60..bd2b5acdc 100644
--- a/docs.it4i/general/slurmtopbs.md
+++ b/docs.it4i/general/slurmtopbs.md
@@ -1,4 +1,4 @@
-# Migrating from SLURM
+# Migrating from Slurm
SLURM-optimized distributed jobs do not run on Karolina out of the box.
--
GitLab
From 53a0e5d8d43d7e97c4ad956c3497293b3896909f Mon Sep 17 00:00:00 2001
From: Jan Siwiec <jan.siwiec@vsb.cz>
Date: Fri, 5 Nov 2021 11:13:11 +0100
Subject: [PATCH 4/7] Update slurmtopbs.md
---
docs.it4i/general/slurmtopbs.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs.it4i/general/slurmtopbs.md b/docs.it4i/general/slurmtopbs.md
index bd2b5acdc..e99fbaf43 100644
--- a/docs.it4i/general/slurmtopbs.md
+++ b/docs.it4i/general/slurmtopbs.md
@@ -1,4 +1,4 @@
-# Migrating from Slurm
+# Migrating From Slurm
SLURM-optimized distributed jobs do not run on Karolina out of the box.
--
GitLab
From 314af8dcb0e278c09df9f3a23127b67a02c7b085 Mon Sep 17 00:00:00 2001
From: Branislav Jansik <branislav.jansik@vsb.cz>
Date: Mon, 8 Nov 2021 13:12:59 +0100
Subject: [PATCH 5/7] Update slurmtopbs.md
---
docs.it4i/general/slurmtopbs.md | 29 ++++++++++++++++++++++++-----
1 file changed, 24 insertions(+), 5 deletions(-)
diff --git a/docs.it4i/general/slurmtopbs.md b/docs.it4i/general/slurmtopbs.md
index e99fbaf43..7a32cf94b 100644
--- a/docs.it4i/general/slurmtopbs.md
+++ b/docs.it4i/general/slurmtopbs.md
@@ -1,15 +1,34 @@
# Migrating From Slurm
-SLURM-optimized distributed jobs do not run on Karolina out of the box.
-
-In typical multi-gpu multi-node setting one needs to know:
+SLURM-optimized parallel jobs will not under PBS out of the box.
+Conversion to PBS standards is necessary. Here we provide hints on how to proceed.
+
+It is important to notice that `mpirun` is used here as an alternative to the `srun` in SLURM. The `-n` flag is used to regulate the number of tasks spawned by the MPI. The path to the script being run by MPI must be absolute. The script rights should be set to allow execution and reading.
+
+The PBS provides some useful variables that may be used in the jobscripts
+`PBS_O_WORKDIR` and `PBS_JOBID`. For example:
+
+The `PBS_O_WORKDIR` returns the directory, where the `qsub` command was submitted.
+The `PBS_JOBID` returns the numercal identifyer of the job.
+The `qsub` always starts execution in the `$HOME` directory.
+
+## Migrating PyTorch from SLURM
+
+The Intel MPI provides some useful variables that may be used in the scripts executed via the MPI.
+these include `PMI_RANK`,`PMI_SIZE` and `MPI_LOCALRANKID`. For example:
+```
+$ mpirun -n 4 /bin/bash -c 'echo $PMI_SIZE'
+4
+4
+4
+4
+```
+In typical multi-gpu multi-node setting using PyTorch one needs to know:
- World-size - i.e. the total number of GPUs in the system
- Rank of a job in the world - i.e. the number of the current GPU in the system
- Local GPU ID for assignment purposes inside PyTorch/TF
-The combination of PBS and MPI does not require large changes in the codebase so you can test and debug large-scale jobs on the CIIRC cluster and deploy them on Karolina without much of a change.
-
The following example assumes that you use PyTorch and your `DistributedDataParallel` process is being initialized via `init_method` pointing to some file.
The required changes are:
--
GitLab
From 8c1993b3a279335815dfd7d52d16f83036a3368e Mon Sep 17 00:00:00 2001
From: Branislav Jansik <branislav.jansik@vsb.cz>
Date: Mon, 8 Nov 2021 13:16:50 +0100
Subject: [PATCH 6/7] Update slurmtopbs.md
---
docs.it4i/general/slurmtopbs.md | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/docs.it4i/general/slurmtopbs.md b/docs.it4i/general/slurmtopbs.md
index 7a32cf94b..810bd5af1 100644
--- a/docs.it4i/general/slurmtopbs.md
+++ b/docs.it4i/general/slurmtopbs.md
@@ -15,7 +15,12 @@ The `qsub` always starts execution in the `$HOME` directory.
## Migrating PyTorch from SLURM
The Intel MPI provides some useful variables that may be used in the scripts executed via the MPI.
-these include `PMI_RANK`,`PMI_SIZE` and `MPI_LOCALRANKID`. For example:
+these include `PMI_RANK`,`PMI_SIZE` and `MPI_LOCALRANKID`.
+
+- The `PMI_RANK` and `MPI_LOCALRANKID` returns the process rank within the MPI_COMM_WORLD communicator - the process number
+- The `PMI_SIZE` returns the process rank within the MPI_COMM_WORLD communicator - the number of processes
+
+For example:
```
$ mpirun -n 4 /bin/bash -c 'echo $PMI_SIZE'
4
--
GitLab
From 4525ac1f4c3fe0e0b6ff96be67f58cad5efd4ecc Mon Sep 17 00:00:00 2001
From: Jan Siwiec <jan.siwiec@vsb.cz>
Date: Mon, 8 Nov 2021 13:22:51 +0100
Subject: [PATCH 7/7] Update slurmtopbs.md
---
docs.it4i/general/slurmtopbs.md | 20 +++++++++++---------
1 file changed, 11 insertions(+), 9 deletions(-)
diff --git a/docs.it4i/general/slurmtopbs.md b/docs.it4i/general/slurmtopbs.md
index 810bd5af1..fc2588da5 100644
--- a/docs.it4i/general/slurmtopbs.md
+++ b/docs.it4i/general/slurmtopbs.md
@@ -1,4 +1,4 @@
-# Migrating From Slurm
+# Migrating From SLURM
SLURM-optimized parallel jobs will not under PBS out of the box.
Conversion to PBS standards is necessary. Here we provide hints on how to proceed.
@@ -6,21 +6,22 @@ Conversion to PBS standards is necessary. Here we provide hints on how to procee
It is important to notice that `mpirun` is used here as an alternative to the `srun` in SLURM. The `-n` flag is used to regulate the number of tasks spawned by the MPI. The path to the script being run by MPI must be absolute. The script rights should be set to allow execution and reading.
The PBS provides some useful variables that may be used in the jobscripts
-`PBS_O_WORKDIR` and `PBS_JOBID`. For example:
-
-The `PBS_O_WORKDIR` returns the directory, where the `qsub` command was submitted.
+`PBS_O_WORKDIR` and `PBS_JOBID`. For example:
+
+The `PBS_O_WORKDIR` returns the directory, where the `qsub` command was submitted.
The `PBS_JOBID` returns the numercal identifyer of the job.
The `qsub` always starts execution in the `$HOME` directory.
-## Migrating PyTorch from SLURM
+## Migrating PyTorch From SLURM
The Intel MPI provides some useful variables that may be used in the scripts executed via the MPI.
-these include `PMI_RANK`,`PMI_SIZE` and `MPI_LOCALRANKID`.
+these include `PMI_RANK`,`PMI_SIZE` and `MPI_LOCALRANKID`.
-- The `PMI_RANK` and `MPI_LOCALRANKID` returns the process rank within the MPI_COMM_WORLD communicator - the process number
-- The `PMI_SIZE` returns the process rank within the MPI_COMM_WORLD communicator - the number of processes
+- The `PMI_RANK` and `MPI_LOCALRANKID` returns the process rank within the MPI_COMM_WORLD communicator - the process number
+- The `PMI_SIZE` returns the process rank within the MPI_COMM_WORLD communicator - the number of processes
For example:
+
```
$ mpirun -n 4 /bin/bash -c 'echo $PMI_SIZE'
4
@@ -28,6 +29,7 @@ $ mpirun -n 4 /bin/bash -c 'echo $PMI_SIZE'
4
4
```
+
In typical multi-gpu multi-node setting using PyTorch one needs to know:
- World-size - i.e. the total number of GPUs in the system
@@ -43,6 +45,6 @@ The required changes are:
- To get local GPU ID on the node (can be used to manually set `CUDA_VISIBLE_DEVICES`), access the `MPI_LOCALRANKID` variable.
!!! hint
- Some jobs may greatly benefit from [mixed precision training][1] since the new NVIDIA A100 GPUs have excellent support for it.
+ Some jobs may greatly benefit from [mixed precision training][1] since the new NVIDIA A100 GPUs have excellent support for it.
[1]: https://pytorch.org/docs/stable/amp.html
--
GitLab