There are default compilers associated with any particular MPI implementation. The defaults may be changed; the MPI libraries may be used in conjunction with any compiler. The defaults are selected via the modules in the following way:
There are default compilers associated with any particular MPI implementation. The defaults may be changed; the MPI libraries may be used in conjunction with any compiler.
In this example, we activate the latest OpenMPI with the latest GNU compilers (OpenMPI 1.8.6 and GCC 5.1). For more information about toolchains, see the [Environment and Modules][1] section.
In this example, we activate the OpenMPI with the GNU compilers (OpenMPI 4.0.5 and GCC 10.2.0). For more information about toolchains, see the [Environment and Modules][1] section.
To use OpenMPI with the Intel compiler suite, use:
To use OpenMPI with the Intel compiler suite, use:
```console
```console
$ml iompi/2015.03
$ml iompi/2020b
```
```
In this example, the OpenMPI 1.8.6 using the Intel compilers is activated. It uses the `iompi` toolchain.
In this example, the OpenMPI 4.0.5 using the Intel compilers 2020.4.304 is activated. It uses the `iompi` toolchain.
## Compiling MPI Programs
## Compiling MPI Programs
After setting up your MPI environment, compile your program using one of the MPI wrappers:
After setting up your MPI environment, compile your program using one of the MPI wrappers:
Wrappers `mpif90` and `mpif77` provided by Intel MPI are designed for GCC and GFortran. You might be able to compile MPI code by them even with Intel compilers, but you might run into problems (for example, native MIC compilation with `-mmic` does not work with `mpif90`).
Wrappers `mpif90` and `mpif77` provided by Intel MPI are designed for GCC and GFortran. You might be able to compile MPI code by them even with Intel compilers, but you might run into problems.
Example program:
Example program:
...
@@ -119,29 +154,26 @@ The optimal way to run an MPI program depends on its memory requirements, memory
...
@@ -119,29 +154,26 @@ The optimal way to run an MPI program depends on its memory requirements, memory
!!! note
!!! note
Consider these ways to run an MPI program:
Consider these ways to run an MPI program:
1. One MPI process per node, 24 threads per process
1. One MPI process per node, 128 threads per process
2. Two MPI processes per node, 12 threads per process
2. Two MPI processes per node, 64 threads per process
3.24 MPI processes per node, 1 thread per process.
3.128 MPI processes per node, 1 thread per process.
**One MPI** process per node, using 24 threads, is most useful for memory demanding applications that make good use of processor cache memory and are not memory-bound. This is also a preferred way for communication intensive applications as one process per node enjoys full bandwidth access to the network interface.
**One MPI** process per node, using 128 threads, is most useful for memory demanding applications that make good use of processor cache memory and are not memory-bound. This is also a preferred way for communication intensive applications as one process per node enjoys full bandwidth access to the network interface.
**Two MPI** processes per node, using 12 threads each, bound to processor socket is most useful for memory bandwidth-bound applications such as BLAS1 or FFT with scalable memory demand. However, note that the two processes will share access to the network interface. The 12 threads and socket binding should ensure maximum memory access bandwidth and minimize communication, migration, and NUMA effect overheads.
**Two MPI** processes per node, using 64 threads each, bound to processor socket is most useful for memory bandwidth-bound applications such as BLAS1 or FFT with scalable memory demand. However, note that the two processes will share access to the network interface. The 64 threads and socket binding should ensure maximum memory access bandwidth and minimize communication, migration, and NUMA effect overheads.
!!! note
!!! note
Important! Bind every OpenMP thread to a core!
Important! Bind every OpenMP thread to a core!
In the previous two cases with one or two MPI processes per node, the operating system might still migrate OpenMP threads between cores. You want to avoid this by setting the `KMP_AFFINITY` or `GOMP_CPU_AFFINITY` environment variables.
In the previous two cases with one or two MPI processes per node, the operating system might still migrate OpenMP threads between cores. You want to avoid this by setting the `KMP_AFFINITY` or `GOMP_CPU_AFFINITY` environment variables.
**24 MPI** processes per node, using 1 thread each bound to a processor core is most suitable for highly scalable applications with low communication demand.
**128 MPI** processes per node, using 1 thread each bound to a processor core is most suitable for highly scalable applications with low communication demand.
### Running OpenMPI
### Running OpenMPI
The [OpenMPI 1.8.6][a] is based on OpenMPI. Read more on [how to run OpenMPI][2].
The [OpenMPI 4.1.1][a] is based on OpenMPI. Read more on [how to run OpenMPI][2].
The Intel MPI may run on the [Intel Xeon Ph][3] accelerators as well. Read more on [how to run Intel MPI on accelerators][3].
