mpi.md

MPI

Setting Up MPI Environment

The Karolina cluster provides several implementations of the MPI library:

• OpenMPI
• Intel MPI (impi)
• MPICH

MPI libraries are activated via the environment modules.

Look up the modulefiles/mpi section in ml av:

$ ml av
------------------------------------------------------- /apps/modules/mpi -------------------------------------------------------
   OpenMPI/3.1.4-GCC-6.3.0-2.27               OpenMPI/4.1.1-GCC-10.2.0
   OpenMPI/4.0.3-GCC-9.3.0                    OpenMPI/4.1.1-GCC-10.3.0                             (D)
   OpenMPI/4.0.5-GCC-10.2.0                   impi/2017.4.239-iccifort-2017.8.262-GCC-6.3.0-2.27
   OpenMPI/4.0.5-gcccuda-2020b                impi/2018.4.274-iccifort-2018.5.274-GCC-8.3.0-2.32
   OpenMPI/4.0.5-iccifort-2020.4.304          impi/2018.4.274-iccifort-2019.1.144-GCC-8.2.0-2.31.1
   OpenMPI/4.0.5-NVHPC-21.2-CUDA-11.2.2       impi/2019.9.304-iccifort-2020.1.217
   OpenMPI/4.0.5-NVHPC-21.2-CUDA-11.3.0       impi/2019.9.304-iccifort-2020.4.304
   OpenMPI/4.1.1-GCC-10.2.0-Java-1.8.0_221    impi/2021.2.0-intel-compilers-2021.2.0               (D)
   MPICH/3.3.2-GCC-10.2.0

There are default compilers associated with any particular MPI implementation. The defaults may be changed; the MPI libraries may be used in conjunction with any compiler.

Examples:

$ ml gompi/2020b

In this example, we activate the OpenMPI with the GNU compilers (OpenMPI 4.0.5 and GCC 10.2.0). For more information about toolchains, see the Environment and Modules section.

To use OpenMPI with the Intel compiler suite, use:

$ ml iompi/2020b

In this example, the OpenMPI 4.0.5 using the Intel compilers 2020.4.304 is activated. It uses the iompi toolchain.

Compiling MPI Programs

After setting up your MPI environment, compile your program using one of the MPI wrappers:

For module gompi/2020b

$ mpicc -v
Using built-in specs.
COLLECT_GCC=/apps/all/GCCcore/10.2.0/bin/gcc
COLLECT_LTO_WRAPPER=/apps/all/GCCcore/10.2.0/libexec/gcc/x86_64-pc-linux-gnu/10.2.0/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
Target: x86_64-pc-linux-gnu
Configured with: ../configure --enable-languages=c,c++,fortran --without-cuda-driver --enable-offload-targets=nvptx-none --enable-lto --enable-checking=release --disable-multilib --enable-shared=yes --enable-static=yes --enable-threads=posix --enable-plugins --enable-gold=default --enable-ld --with-plugin-ld=ld.gold --prefix=/apps/all/GCCcore/10.2.0 --with-local-prefix=/apps/all/GCCcore/10.2.0 --enable-bootstrap --with-isl=/dev/shm/easybuild/build/GCCcore/10.2.0/system-system/gcc-10.2.0/stage2_stuff
Thread model: posix
Supported LTO compression algorithms: zlib
gcc version 10.2.0 (GCC)
$ mpif77 -v
Using built-in specs.
COLLECT_GCC=/apps/all/GCCcore/10.2.0/bin/gfortran
COLLECT_LTO_WRAPPER=/apps/all/GCCcore/10.2.0/libexec/gcc/x86_64-pc-linux-gnu/10.2.0/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
Target: x86_64-pc-linux-gnu
Configured with: ../configure --enable-languages=c,c++,fortran --without-cuda-driver --enable-offload-targets=nvptx-none --enable-lto --enable-checking=release --disable-multilib --enable-shared=yes --enable-static=yes --enable-threads=posix --enable-plugins --enable-gold=default --enable-ld --with-plugin-ld=ld.gold --prefix=/apps/all/GCCcore/10.2.0 --with-local-prefix=/apps/all/GCCcore/10.2.0 --enable-bootstrap --with-isl=/dev/shm/easybuild/build/GCCcore/10.2.0/system-system/gcc-10.2.0/stage2_stuff
Thread model: posix
Supported LTO compression algorithms: zlib
gcc version 10.2.0 (GCC)
~$ mpif90 -v
Using built-in specs.
COLLECT_GCC=/apps/all/GCCcore/10.2.0/bin/gfortran
COLLECT_LTO_WRAPPER=/apps/all/GCCcore/10.2.0/libexec/gcc/x86_64-pc-linux-gnu/10.2.0/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
Target: x86_64-pc-linux-gnu
Configured with: ../configure --enable-languages=c,c++,fortran --without-cuda-driver --enable-offload-targets=nvptx-none --enable-lto --enable-checking=release --disable-multilib --enable-shared=yes --enable-static=yes --enable-threads=posix --enable-plugins --enable-gold=default --enable-ld --with-plugin-ld=ld.gold --prefix=/apps/all/GCCcore/10.2.0 --with-local-prefix=/apps/all/GCCcore/10.2.0 --enable-bootstrap --with-isl=/dev/shm/easybuild/build/GCCcore/10.2.0/system-system/gcc-10.2.0/stage2_stuff
Thread model: posix
Supported LTO compression algorithms: zlib
gcc version 10.2.0 (GCC)

When using Intel MPI, use the following MPI wrappers:

For module intel/2020b