diff --git a/docs.it4i/software/intel/intel-xeon-phi-anselm.md b/docs.it4i/software/intel/intel-xeon-phi-anselm.md index bf7f7e4882d06313727de3885e17ca405b07b393..7b6ed0c3b62a1322711711c2b0b37264f4bbe156 100644 --- a/docs.it4i/software/intel/intel-xeon-phi-anselm.md +++ b/docs.it4i/software/intel/intel-xeon-phi-anselm.md @@ -6,25 +6,25 @@ Intel Xeon Phi can be programmed in several modes. The default mode on Anselm is ## Intel Utilities for Xeon Phi -To get access to a compute node with Intel Xeon Phi accelerator, use the PBS interactive session +To get access to a compute node with the Intel Xeon Phi accelerator, use the PBS interactive session: ```console $ qsub -I -q qmic -A NONE-0-0 ``` -To set up the environment module "Intel" has to be loaded +To set up the environment, the intel module has to be loaded: ```console $ ml intel ``` -Information about the hardware can be obtained by running the micinfo program on the host. +Information about the hardware can be obtained by running the micinfo program on the host: ```console $ /usr/bin/micinfo ``` -The output of the "micinfo" utility executed on one of the Anselm node is as follows. (note: to get PCIe related details the command has to be run with root privileges) +The output of the "micinfo" utility executed on one of the Anselm node is as follows (note: to get PCIe related details, the command has to be run with root privileges): ```console MicInfo Utility Log @@ -90,23 +90,23 @@ Device No: 0, Device Name: mic0 ## Offload Mode -To compile a code for Intel Xeon Phi a MPSS stack has to be installed on the machine where compilation is executed. Currently the MPSS stack is only installed on compute nodes equipped with accelerators. +To compile code for Intel Xeon Phi, an MPSS stack has to be installed on the machine where compilation is executed. Currently, the MPSS stack is only installed on compute nodes equipped with the accelerators. ```console $ qsub -I -q qmic -A NONE-0-0 $ ml intel ``` -For debugging purposes it is also recommended to set environment variable "OFFLOAD_REPORT". Value can be set from 0 to 3, where higher number means more debugging information. +For debugging purposes, it is also recommended to set the "OFFLOAD_REPORT" environment variable. The value can be set from 0 to 3, where a higher number means more debugging information: ```console export OFFLOAD_REPORT=3 ``` -A very basic example of code that employs offload programming technique is shown in the next listing. +A very basic example of code that employs the offload programming technique is shown below. !!! note - This code is sequential and utilizes only single core of the accelerator. + This code is sequential and utilizes only a single core of the accelerator. ```cpp $ vim source-offload.cpp @@ -128,13 +128,13 @@ int main(int argc, char* argv[]) } ``` -To compile a code using Intel compiler run +To compile the code using the Intel compiler, run: ```console $ icc source-offload.cpp -o bin-offload ``` -To execute the code, run the following command on the host +To execute the code, run the following command on the host: ```console $ ./bin-offload @@ -142,7 +142,7 @@ $ ./bin-offload ### Parallelization in Offload Mode Using OpenMP -One way of paralelization a code for Xeon Phi is using OpenMP directives. The following example shows code for parallel vector addition. +One way of parallelization the code for Xeon Phi is using OpenMP directives. The following example shows the code for parallel vector addition: ```cpp $ vim ./vect-add @@ -222,7 +222,7 @@ int main() } ``` -During the compilation Intel compiler shows which loops have been vectorized in both host and accelerator. This can be enabled with compiler option "-vec-report2". To compile and execute the code run +During the compilation, the Intel compiler shows which loops have been vectorized in both the host and the accelerator. This can be enabled with the "-vec-report2" compiler option. To compile and execute the code, run: ```console $ icc vect-add.c -openmp_report2 -vec-report2 -o vect-add @@ -237,15 +237,15 @@ Some interesting compiler flags useful not only for code debugging are: openmp_report[0|1|2] - controls the compiler based vectorization diagnostic level vec-report[0|1|2] - controls the OpenMP parallelizer diagnostic level - Performance ooptimization + Performance optimization xhost - FOR HOST ONLY - to generate AVX (Advanced Vector Extensions) instructions. ## Automatic Offload Using Intel MKL Library -Intel MKL includes an Automatic Offload (AO) feature that enables computationally intensive MKL functions called in user code to benefit from attached Intel Xeon Phi coprocessors automatically and transparently. +Intel MKL includes the Automatic Offload (AO) feature that enables computationally intensive MKL functions called in user code to benefit from attached Intel Xeon Phi coprocessors automatically and transparently. !!! note - Behavioral of automatic offload mode is controlled by functions called within the program or by environmental variables. Complete list of controls is listed [here][a]. + Behavior of the automatic offload mode is controlled by functions called within the program or by environment variables. The complete list of controls is listed [here][a]. The Automatic Offload may be enabled by either an MKL function call within the code: @@ -253,24 +253,24 @@ The Automatic Offload may be enabled by either an MKL function call within the c mkl_mic_enable(); ``` -or by setting environment variable +or by setting the environment variable: ```console $ export MKL_MIC_ENABLE=1 ``` -To get more information about automatic offload refer to "[Using Intel® MKL Automatic Offload on Intel ® Xeon Phi™ Coprocessors][b]" white paper or [Intel MKL documentation][c]. +To get more information about the automatic offload, refer to the "[Using Intel® MKL Automatic Offload on Intel ® Xeon Phi™ Coprocessors][b]" white paper or the [Intel MKL documentation][c]. ### Automatic Offload Example -At first get an interactive PBS session on a node with MIC accelerator and load "intel" module that automatically loads "mkl" module as well. +At first, get an interactive PBS session on a node with the MIC accelerator and load the "intel" module that automatically loads the "mkl" module as well. ```console $ qsub -I -q qmic -A OPEN-0-0 -l select=1:ncpus=16 $ ml intel ``` -Following example show how to automatically offload an SGEMM (single precision - general matrix multiply) function to MIC coprocessor. The code can be copied to a file and compiled without any necessary modification. +The following example shows how to automatically offload an SGEMM (single precision - general matrix multiply) function to MIC coprocessor. The code can be copied to a file and compiled without any necessary modification. ```cpp $ vim sgemm-ao-short.c @@ -330,21 +330,21 @@ int main(int argc, char **argv) ``` !!! note - This example is simplified version of an example from MKL. The expanded version can be found here: `$MKL_EXAMPLES/mic_ao/blasc/source/sgemm.c`. + This example is a simplified version of an example from MKL. The expanded version can be found here: `$MKL_EXAMPLES/mic_ao/blasc/source/sgemm.c`. -To compile a code using Intel compiler use: +To compile the code using the Intel compiler, use: ```console $ icc -mkl sgemm-ao-short.c -o sgemm ``` -For debugging purposes enable the offload report to see more information about automatic offloading. +For debugging purposes, enable the offload report to see more information about the automatic offloading: ```console $ export OFFLOAD_REPORT=2 ``` -The output of a code should look similar to following listing, where lines starting with [MKL] are generated by offload reporting: +The output of the code should look similar to the following listing, where lines starting with [MKL] are generated by the offload reporting: ```console Computing SGEMM on the host @@ -364,7 +364,7 @@ The output of a code should look similar to following listing, where lines start In the native mode a program is executed directly on Intel Xeon Phi without involvement of the host machine. Similarly to offload mode, the code is compiled on the host computer with Intel compilers. -To compile a code user has to be connected to a compute with MIC and load Intel compilers module. To get an interactive session on a compute node with an Intel Xeon Phi and load the module use following commands: +To compile a code, the user has to be connected to a compute node with MIC and load the Intel compilers module. To get an interactive session on a compute node with an Intel Xeon Phi and load the module, use the following commands: ```console $ qsub -I -q qmic -A NONE-0-0 @@ -372,9 +372,9 @@ $ ml intel ``` !!! note - Particular version of the Intel module is specified. This information is used later to specify the correct library paths. + A particular version of the Intel module is specified. This information is used later to specify the correct library paths. -To produce a binary compatible with Intel Xeon Phi architecture user has to specify "-mmic" compiler flag. Two compilation examples are shown below. The first example shows how to compile OpenMP parallel code "vect-add.c" for host only: +To produce a binary compatible with the Intel Xeon Phi architecture, the user has to specify the "-mmic" compiler flag. Two compilation examples are shown below. The first example shows how to compile an OpenMP parallel code "vect-add.c" for the host only: ```console $ icc -xhost -no-offload -fopenmp vect-add.c -o vect-add-host @@ -394,9 +394,9 @@ $ icc -mmic -fopenmp vect-add.c -o vect-add-mic ### Execution of the Program in Native Mode on Intel Xeon Phi -The user access to the Intel Xeon Phi is through the SSH. Since user home directories are mounted using NFS on the accelerator, users do not have to copy binary files or libraries between the host and accelerator. +User access to Intel Xeon Phi is via SSH. Since user home directories are mounted using NFS on the accelerator, users do not have to copy binary files or libraries between the host and the accelerator. -To connect to the accelerator run: +To connect to the accelerator, run: ```console $ ssh mic0 @@ -408,16 +408,16 @@ If the code is sequential, it can be executed directly: mic0 $ ~/path_to_binary/vect-add-seq-mic ``` -If the code is parallelized using OpenMP a set of additional libraries is required for execution. To locate these libraries new path has to be added to the LD_LIBRARY_PATH environment variable prior to the execution: +If the code is parallelized using OpenMP, a set of additional libraries is required for execution. To locate these libraries a new path has to be added to the LD_LIBRARY_PATH environment variable prior to the execution: ```console mic0 $ export LD_LIBRARY_PATH=/apps/intel/composer_xe_2013.5.192/compiler/lib/mic:$LD_LIBRARY_PATH ``` !!! note - The path exported in the previous example contains path to a specific compiler (here the version is 5.192). This version number has to match with the version number of the Intel compiler module that was used to compile the code on the host computer. + The path exported in the previous example contains a path to a specific compiler (here the version is 5.192). This version number has to match the version number of the Intel compiler module that was used to compile the code on the host computer. -For your information the list of libraries and their location required for execution of an OpenMP parallel code on Intel Xeon Phi is: +The list of libraries and their location required for execution of an OpenMP parallel code on Intel Xeon Phi is: !!! note /apps/intel/composer_xe_2013.5.192/compiler/lib/mic @@ -428,7 +428,7 @@ For your information the list of libraries and their location required for execu - libirng.so - libintlc.so.5 -Finally, to run the compiled code use: +Finally, to run the compiled code, use: ```console $ ~/path_to_binary/vect-add-mic @@ -438,13 +438,13 @@ $ ~/path_to_binary/vect-add-mic OpenCL (Open Computing Language) is an open standard for general-purpose parallel programming for diverse mix of multi-core CPUs, GPU coprocessors, and other parallel processors. OpenCL provides a flexible execution model and uniform programming environment for software developers to write portable code for systems running on both the CPU and graphics processors or accelerators like the Intel® Xeon Phi. -On Anselm OpenCL is installed only on compute nodes with MIC accelerator, therefore OpenCL code can be compiled only on these nodes. +On Anselm, OpenCL is installed only on compute nodes with the MIC accelerator, so OpenCL code can be compiled only on these nodes. ```console ml opencl-sdk opencl-rt ``` -Always load "opencl-sdk" (providing devel files like headers) and "opencl-rt" (providing dynamic library libOpenCL.so) modules to compile and link OpenCL code. Load "opencl-rt" for running your compiled code. +Always load the "opencl-sdk" (providing devel files like headers) and "opencl-rt" (providing dynamic library libOpenCL.so) modules to compile and link OpenCL code. Load "opencl-rt" for running your compiled code. There are two basic examples of OpenCL code in the following directory: @@ -452,13 +452,13 @@ There are two basic examples of OpenCL code in the following directory: /apps/intel/opencl-examples/ ``` -First example "CapsBasic" detects OpenCL compatible hardware, here CPU and MIC, and prints basic information about the capabilities of it. +The first example "CapsBasic" detects OpenCL compatible hardware, here CPU and MIC, and prints basic information about its capabilities. ```console /apps/intel/opencl-examples/CapsBasic/capsbasic ``` -To compile and run the example copy it to your home directory, get a PBS interactive session on of the nodes with MIC and run make for compilation. Make files are very basic and shows how the OpenCL code can be compiled on Anselm. +To compile and run the example, copy it to your home directory, get a PBS interactive session on one of the nodes with MIC, and run make for compilation. Make files are very basic and show how the OpenCL code can be compiled on Anselm. ```console $ cp /apps/intel/opencl-examples/CapsBasic/* . @@ -472,7 +472,7 @@ The compilation command for this example is: $ g++ capsbasic.cpp -lOpenCL -o capsbasic -I/apps/intel/opencl/include/ ``` -After executing the complied binary file, following output should be displayed. +After executing the complied binary file, the following output should be displayed: ```console $ ./capsbasic @@ -501,9 +501,9 @@ CL_DEVICE_TYPE_ACCELERATOR[0] ``` !!! note - More information about this example can be found on Intel website: <http://software.intel.com/en-us/vcsource/samples/caps-basic/> + More information about this example can be found on the Intel website: <http://software.intel.com/en-us/vcsource/samples/caps-basic/> -The second example that can be found in "/apps/intel/opencl-examples" directory is General Matrix Multiply. You can follow the the same procedure to download the example to your directory and compile it. +The second example that can be found in the "/apps/intel/opencl-examples" directory is General Matrix Multiply. You can follow the same procedure to download the example to your directory and compile it: ```console $ cp -r /apps/intel/opencl-examples/* . @@ -518,7 +518,7 @@ The compilation command for this example is: $ g++ cmdoptions.cpp gemm.cpp ../common/basic.cpp ../common/cmdparser.cpp ../common/oclobject.cpp -I../common -lOpenCL -o gemm -I/apps/intel/opencl/include/ ``` -To see the performance of Intel Xeon Phi performing the DGEMM run the example as follows: +To see the performance of Intel Xeon Phi performing the DGEMM, run the example as follows: ```console ./gemm -d 1 @@ -541,37 +541,37 @@ To see the performance of Intel Xeon Phi performing the DGEMM run the example as ``` !!! warning - GNU compiler is used to compile the OpenCL codes for Intel MIC. You do not need to load Intel compiler module. + The GNU compiler is used to compile the OpenCL codes for Intel MIC. You do not need to load the Intel compiler module. ## MPI ### Environment Setup and Compilation -Again an MPI code for Intel Xeon Phi has to be compiled on a compute node with accelerator and MPSS software stack installed. To get to a compute node with accelerator use: +Again, an MPI code for Intel Xeon Phi has to be compiled on a compute node with the accelerator and the MPSS software stack installed. To get to a compute node with the accelerator, use: ```console $ qsub -I -q qmic -A NONE-0-0 ``` -The only supported implementation of MPI standard for Intel Xeon Phi is Intel MPI. To setup a fully functional development environment a combination of Intel compiler and Intel MPI has to be used. On a host load following modules before compilation: +The only supported implementation of MPI standard for Intel Xeon Phi is Intel MPI. To set up a fully functional development environment, a combination of the Intel compiler and Intel MPI has to be used. On a host, load the following modules before compilation: ```console $ ml intel ``` -To compile an MPI code for host use: +To compile an MPI code for the host, use: ````console $ mpiicc -xhost -o mpi-test mpi-test.c ``` -To compile the same code for Intel Xeon Phi architecture use: +To compile the same code for the Intel Xeon Phi architecture, use: ```console $ mpiicc -mmic -o mpi-test-mic mpi-test.c ```` -An example of basic MPI version of "hello-world" example in C language, that can be executed on both host and Xeon Phi is (can be directly copy and pasted to a .c file) +A basic MPI version of the "hello-world" example in C language that can be executed on both the host and Xeon Phi is (can be directly copied and pasted to a .c file): ```cpp #include <stdio.h> @@ -628,12 +628,12 @@ Hello world from process 0 of 4 on host cn207 ### Coprocessor-Only Model -There are two ways how to execute an MPI code on a single coprocessor: 1.) lunch the program using "**mpirun**" from the -coprocessor; or 2.) lunch the task using "**mpiexec.hydra**" from a host. +There are two ways to execute MPI code on a single coprocessor: 1) launch the program using "**mpirun**" from the +coprocessor; or 2) launch the task using "**mpiexec.hydra**" from a host. #### Execution on Coprocessor -Similarly to execution of OpenMP programs in native mode, since the environmental module are not supported on MIC, user has to setup paths to Intel MPI libraries and binaries manually. One time setup can be done by creating a "**.profile**" file in user's home directory. This file sets up the environment on the MIC automatically once user access to the accelerator through the SSH. +Similarly to execution of OpenMP programs in native mode, since the environmental module is not supported on MIC, the user has to setup paths to Intel MPI libraries and binaries manually. One time setup can be done by creating a "**.profile**" file in user's home directory. This file sets up the environment on the MIC automatically once the user accesses the accelerator via SSH. ```console $ vim ~/.profile @@ -650,22 +650,22 @@ export PATH=/apps/intel/impi/4.1.1.036/mic/bin/:$PATH ``` !!! note - \* this file sets up both environmental variable for both MPI and OpenMP libraries. - \* this file sets up the paths to a particular version of Intel MPI library and particular version of an Intel compiler. These versions have to match with loaded modules. + \* this file sets up the environment variable for both MPI and OpenMP libraries. + \* this file sets up the paths to a particular version of the Intel MPI library and a particular version of the Intel compiler. These versions have to match with loaded modules. -To access a MIC accelerator located on a node that user is currently connected to, use: +To access the MIC accelerator located on a node that the user is currently connected to, use: ```console $ ssh mic0 ``` -or in case you need specify a MIC accelerator on a particular node, use: +or in case you need to specify the MIC accelerator on a particular node, use: ```console $ ssh cn207-mic0 ``` -To run the MPI code in parallel on multiple core of the accelerator, use: +To run the MPI code in parallel on multiple cores of the accelerator, use: ```console $ mpirun -np 4 ./mpi-test-mic @@ -682,9 +682,9 @@ Hello world from process 0 of 4 on host cn207-mic0 #### Execution on Host -If the MPI program is launched from host instead of the coprocessor, the environmental variables are not set using the ".profile" file. Therefore user has to specify library paths from the command line when calling "mpiexec". +If the MPI program is launched from host instead of the coprocessor, the environmental variables are not set using the ".profile" file. Therefore, the user has to specify library paths from the command line when calling "mpiexec". -First step is to tell mpiexec that the MPI should be executed on a local accelerator by setting up the environmental variable "I_MPI_MIC" +First step is to tell mpiexec that the MPI should be executed on a local accelerator by setting up the "I_MPI_MIC" environment variable: ```console $ export I_MPI_MIC=1 @@ -696,15 +696,15 @@ Now the MPI program can be executed as: $ mpiexec.hydra -genv LD_LIBRARY_PATH /apps/intel/impi/4.1.1.036/mic/lib/ -host mic0 -n 4 ~/mpi-test-mic ``` -or using mpirun +or using mpirun: ```console $ mpirun -genv LD_LIBRARY_PATH /apps/intel/impi/4.1.1.036/mic/lib/ -host mic0 -n 4 ~/mpi-test-mic ``` !!! note - \* the full path to the binary has to specified (here: `>~/mpi-test-mic`) - \* the `LD_LIBRARY_PATH` has to match with Intel MPI module used to compile the MPI code + \* the full path to the binary has to be specified (here: `>~/mpi-test-mic`) + \* the `LD_LIBRARY_PATH` has to match with the Intel MPI module used to compile the MPI code The output should be again similar to: @@ -716,7 +716,7 @@ Hello world from process 0 of 4 on host cn207-mic0 ``` !!! note - `mpiexec.hydra` requires a file the MIC filesystem. If the file is missing contact the system administrators. + `mpiexec.hydra` requires a file in the MIC filesystem. If the file is missing, contact the system administrators. A simple test to see if the file is present is to execute: @@ -727,14 +727,14 @@ $ ssh mic0 ls /bin/pmi_proxy #### Execution on Host - MPI Processes Distributed Over Multiple Accelerators on Multiple Nodes** -To get access to multiple nodes with MIC accelerator, user has to use PBS to allocate the resources. To start interactive session, that allocates 2 compute nodes = 2 MIC accelerators run qsub command with following parameters: +To get access to multiple nodes with the MIC accelerator, the user has to use PBS to allocate the resources. To start an interactive session, which allocates 2 compute nodes = 2 MIC accelerators, run the qsub command with the following parameters: ```console $ qsub -I -q qmic -A NONE-0-0 -l select=2:ncpus=16 $ ml intel/13.5.192 impi/4.1.1.036 ``` -This command connects user through ssh to one of the nodes immediately. To see the other nodes that have been allocated use: +This command connects the user via SSH to one of the nodes immediately. To see the other nodes that have been allocated, use: ```console $ cat $PBS_NODEFILE @@ -747,21 +747,21 @@ For example: cn205.bullx ``` -This output means that the PBS allocated nodes cn204 and cn205, which means that user has direct access to "**cn204-mic0**" and "**cn-205-mic0**" accelerators. +This output means that the PBS allocated nodes cn204 and cn205, which means that the user has direct access to the "**cn204-mic0**" and "**cn-205-mic0**" accelerators. !!! note - At this point user can connect to any of the allocated nodes or any of the allocated MIC accelerators using ssh: + At this point, the user can connect to any of the allocated nodes or any of the allocated MIC accelerators using SSH: - to connect to the second node : `$ ssh cn205` - to connect to the accelerator on the first node from the first node: `$ ssh cn204-mic0` or `$ ssh mic0` - to connect to the accelerator on the second node from the first node: `$ ssh cn205-mic0` -At this point we expect that correct modules are loaded and binary is compiled. For parallel execution the mpiexec.hydra is used. Again the first step is to tell mpiexec that the MPI can be executed on MIC accelerators by setting up the environmental variable "I_MPI_MIC" +At this point, we expect that the correct modules are loaded and the binary is compiled. For parallel execution, the mpiexec.hydra is used. Again, the first step is to tell mpiexec that the MPI can be executed on the MIC accelerators by setting up the "I_MPI_MIC" environment variable: ```console $ export I_MPI_MIC=1 ``` -The launch the MPI program use: +The launch the MPI program, use: ```console $ mpiexec.hydra -genv LD_LIBRARY_PATH /apps/intel/impi/4.1.1.036/mic/lib/ @@ -783,7 +783,7 @@ $ mpirun -genv LD_LIBRARY_PATH /apps/intel/impi/4.1.1.036/mic/lib/ : -host cn205-mic0 -n 6 ~/mpi-test-mic ``` -In this case four MPI processes are executed on accelerator cn204-mic and six processes are executed on accelerator cn205-mic0. The sample output (sorted after execution) is: +In this case, four MPI processes are executed on the cn204-mic accelerator and six processes are executed on the cn205-mic0 accelerator. The sample output (sorted after execution) is: ```console Hello world from process 0 of 10 on host cn204-mic0 @@ -798,7 +798,7 @@ In this case four MPI processes are executed on accelerator cn204-mic and six pr Hello world from process 9 of 10 on host cn205-mic0 ``` -The same way MPI program can be executed on multiple hosts: +The same way, the MPI program can be executed on multiple hosts: ```console $ mpiexec.hydra -genv LD_LIBRARY_PATH /apps/intel/impi/4.1.1.036/mic/lib/ @@ -811,10 +811,10 @@ $ mpiexec.hydra -genv LD_LIBRARY_PATH /apps/intel/impi/4.1.1.036/mic/lib/ ### Symmetric Model -In a symmetric mode MPI programs are executed on both host computer(s) and MIC accelerator(s). Since MIC has a different -architecture and requires different binary file produced by the Intel compiler two different files has to be compiled before MPI program is executed. +In a symmetric mode MPI programs are executed on both the host computer(s) and the MIC accelerator(s). Since MIC has a different +architecture and requires a different binary file produced by the Intel compiler, two different files have to be compiled before the MPI program is executed. -In the previous section we have compiled two binary files, one for hosts "**mpi-test**" and one for MIC accelerators "**mpi-test-mic**". These two binaries can be executed at once using mpiexec.hydra: +In the previous section, we have compiled two binary files, one for hosts "**mpi-test**" and one for MIC accelerators "**mpi-test-mic**". These two binaries can be executed at once using mpiexec.hydra: ```console $ mpiexec.hydra @@ -826,7 +826,7 @@ $ mpiexec.hydra : -host cn205-mic0 -n 2 ~/mpi-test-mic ``` -In this example the first two parameters (line 2 and 3) sets up required environment variables for execution. The third line specifies binary that is executed on host (here cn205) and the last line specifies the binary that is execute on the accelerator (here cn205-mic0). +In this example, the first two parameters (line 2 and 3) set up required environment variables for execution. The third line specifies the binary that is executed on the host (here cn205) and the last line specifies the binary that is executed on the accelerator (here cn205-mic0). The output of the program is: @@ -837,9 +837,9 @@ The output of the program is: Hello world from process 3 of 4 on host cn205-mic0 ``` -The execution procedure can be simplified by using the mpirun command with the machine file a a parameter. Machine file contains list of all nodes and accelerators that should used to execute MPI processes. +The execution procedure can be simplified by using the mpirun command with the machine file as a parameter. The machine file contains a list of all nodes and accelerators that should be used to execute MPI processes. -An example of a machine file that uses 2 >hosts (**cn205** and **cn206**) and 2 accelerators **(cn205-mic0** and **cn206-mic0**) to run 2 MPI processes on each of them: +An example of a machine file that uses 2 hosts (**cn205** and **cn206**) and 2 accelerators **(cn205-mic0** and **cn206-mic0**) to run 2 MPI processes on each of them: ```console $ cat hosts_file_mix @@ -849,13 +849,13 @@ $ cat hosts_file_mix cn206-mic0:2 ``` -In addition if a naming convention is set in a way that the name of the binary for host is **"bin_name"** and the name of the binary for the accelerator is **"bin_name-mic"** then by setting up the environment variable **I_MPI_MIC_POSTFIX** to **"-mic"** user do not have to specify the names of booth binaries. In this case mpirun needs just the name of the host binary file (i.e. "mpi-test") and uses the suffix to get a name of the binary for accelerator (i..e. "mpi-test-mic"). +In addition if a naming convention is set in a way that the name of the binary for host is **"bin_name"** and the name of the binary for the accelerator is **"bin_name-mic"** then by setting up the environment variable **I_MPI_MIC_POSTFIX** to **"-mic"**, the user does not have to specify the names of both binaries. In this case, mpirun needs just the name of the host binary file (i.e. "mpi-test") and uses the suffix to get a name of the binary for accelerator (i..e. "mpi-test-mic"). ```console $ export I_MPI_MIC_POSTFIX=-mic ``` -To run the MPI code using mpirun and the machine file "hosts_file_mix" use: +To run the MPI code using mpirun and the machine file "hosts_file_mix", use: ```console $ mpirun @@ -882,11 +882,11 @@ Hello world from process 7 of 8 on host cn205-mic0 ``` !!! note - At this point the MPI communication between MIC accelerators on different nodes uses 1Gb Ethernet only. + At this point, the MPI communication between the MIC accelerators on different nodes uses 1Gb Ethernet only. ### Using Automatically Generated Node-Files -Set of node-files, that can be used instead of manually creating a new one every time, is generated for user convenience. Six node-files are generated: +A set of node-files, which can be used instead of manually creating a new one every time, is generated for the user's convenience. Six node-files are generated: !!! note **Node-files:** @@ -898,11 +898,11 @@ Set of node-files, that can be used instead of manually creating a new one every - /lscratch/${PBS_JOBID}/nodefile-mic-sn MICs only node-file, using short names - /lscratch/${PBS_JOBID}/nodefile-mix-sn Hosts and MICs node-file, using short names -Each host or accelerator is listed only once per file. User has to specify how many jobs should be executed per node using `-n` parameter of the mpirun command. +Each host or accelerator is listed only once per file. User has to specify how many jobs should be executed per node using the `-n` parameter of the mpirun command. ## Optimization -For more details about optimization techniques read Intel document [Optimization and Performance Tuning for Intel® Xeon Phi™ Coprocessors][d]. +For more details about optimization techniques, read the Intel document [Optimization and Performance Tuning for Intel® Xeon Phi™ Coprocessors][d]. [a]: http://software.intel.com/sites/products/documentation/doclib/mkl_sa/11/mkl_userguide_lnx/GUID-3DC4FC7D-A1E4-423D-9C0C-06AB265FFA86.htm [b]: http://software.intel.com/sites/default/files/11MIC42_How_to_Use_MKL_Automatic_Offload_0.pdf