intel-xeon-phi.md

Intel Xeon Phi

##A guide to Intel Xeon Phi usage

Intel Xeon Phi can be programmed in several modes. The default mode on Anselm is offload mode, but all modes described in this document are supported.

Intel Utilities for Xeon Phi

To get access to a compute node with Intel Xeon Phi accelerator, use the PBS interactive session

    $ qsub -I -q qmic -A NONE-0-0

To set up the environment module "Intel" has to be loaded

    $ module load intel/13.5.192

Information about the hardware can be obtained by running the micinfo program on the host.

    $ /usr/bin/micinfo

The output of the "micinfo" utility executed on one of the Anselm node is as follows. (note: to get PCIe related details the command has to be run with root privileges)

    MicInfo Utility Log

    Created Mon Jul 22 00:23:50 2013

            System Info
                    HOST OS                 : Linux
                    OS Version              : 2.6.32-279.5.2.bl6.Bull.33.x86_64
                    Driver Version          : 6720-15
                    MPSS Version            : 2.1.6720-15
                    Host Physical Memory    : 98843 MB

    Device No: 0, Device Name: mic0

            Version
                    Flash Version            : 2.1.03.0386
                    SMC Firmware Version     : 1.15.4830
                    SMC Boot Loader Version  : 1.8.4326
                    uOS Version              : 2.6.38.8-g2593b11
                    Device Serial Number     : ADKC30102482

            Board
                    Vendor ID                : 0x8086
                    Device ID                : 0x2250
                    Subsystem ID             : 0x2500
                    Coprocessor Stepping ID  : 3
                    PCIe Width               : x16
                    PCIe Speed               : 5 GT/s
                    PCIe Max payload size    : 256 bytes
                    PCIe Max read req size   : 512 bytes
                    Coprocessor Model        : 0x01
                    Coprocessor Model Ext    : 0x00
                    Coprocessor Type         : 0x00
                    Coprocessor Family       : 0x0b
                    Coprocessor Family Ext   : 0x00
                    Coprocessor Stepping     : B1
                    Board SKU                : B1PRQ-5110P/5120D
                    ECC Mode                 : Enabled
                    SMC HW Revision          : Product 225W Passive CS

            Cores
                    Total No of Active Cores : 60
                    Voltage                  : 1032000 uV
                    Frequency                : 1052631 kHz

            Thermal
                    Fan Speed Control        : N/A
                    Fan RPM                  : N/A
                    Fan PWM                  : N/A
                    Die Temp                 : 49 C

            GDDR
                    GDDR Vendor              : Elpida
                    GDDR Version             : 0x1
                    GDDR Density             : 2048 Mb
                    GDDR Size                : 7936 MB
                    GDDR Technology          : GDDR5
                    GDDR Speed               : 5.000000 GT/s
                    GDDR Frequency           : 2500000 kHz
                    GDDR Voltage             : 1501000 uV

Offload Mode

To compile a code for Intel Xeon Phi a MPSS stack has to be installed on the machine where compilation is executed. Currently the MPSS stack is only installed on compute nodes equipped with accelerators.

    $ qsub -I -q qmic -A NONE-0-0
    $ module load intel/13.5.192

For debugging purposes it is also recommended to set environment variable "OFFLOAD_REPORT". Value can be set from 0 to 3, where higher number means more debugging information.

    export OFFLOAD_REPORT=3

A very basic example of code that employs offload programming technique is shown in the next listing. Please note that this code is sequential and utilizes only single core of the accelerator.

    $ vim source-offload.cpp

    #include <iostream>

    int main(int argc, char* argv[])
    {
        const int niter = 100000;
        double result = 0;

     #pragma offload target(mic)
        for (int i = 0; i < niter; ++i) {
            const double t = (i + 0.5) / niter;
            result += 4.0 / (t * t + 1.0);
        }
        result /= niter;
        std::cout << "Pi ~ " << result << 'n';
    }

To compile a code using Intel compiler run

    $ icc source-offload.cpp -o bin-offload

To execute the code, run the following command on the host

    ./bin-offload

Parallelization in Offload Mode Using OpenMP

One way of paralelization a code for Xeon Phi is using OpenMP directives. The following example shows code for parallel vector addition.

    $ vim ./vect-add

    #include <stdio.h>

    typedef int T;

    #define SIZE 1000

    #pragma offload_attribute(push, target(mic))
    T in1[SIZE];
    T in2[SIZE];
    T res[SIZE];
    #pragma offload_attribute(pop)

    // MIC function to add two vectors
    __attribute__((target(mic))) add_mic(T *a, T *b, T *c, int size) {
      int i = 0;
      #pragma omp parallel for
        for (i = 0; i < size; i++)
          c[i] = a[i] + b[i];
    }

    // CPU function to add two vectors
    void add_cpu (T *a, T *b, T *c, int size) {
      int i;
      for (i = 0; i < size; i++)
        c[i] = a[i] + b[i];
    }

    // CPU function to generate a vector of random numbers
    void random_T (T *a, int size) {
      int i;
      for (i = 0; i < size; i++)
        a[i] = rand() % 10000; // random number between 0 and 9999
    }

    // CPU function to compare two vectors
    int compare(T *a, T *b, T size ){
      int pass = 0;
      int i;
      for (i = 0; i < size; i++){
        if (a[i] != b[i]) {
          printf("Value mismatch at location %d, values %d and %dn",i, a[i], b[i]);
          pass = 1;
        }
      }
      if (pass == 0) printf ("Test passedn"); else printf ("Test Failedn");
      return pass;
    }

    int main()
    {
      int i;
      random_T(in1, SIZE);
      random_T(in2, SIZE);

      #pragma offload target(mic) in(in1,in2)  inout(res)
      {

        // Parallel loop from main function
        #pragma omp parallel for
        for (i=0; i<SIZE; i++)
          res[i] = in1[i] + in2[i];

        // or parallel loop is called inside the function
        add_mic(in1, in2, res, SIZE);

      }

      //Check the results with CPU implementation
      T res_cpu[SIZE];
      add_cpu(in1, in2, res_cpu, SIZE);
      compare(res, res_cpu, SIZE);

    }

During the compilation Intel compiler shows which loops have been vectorized in both host and accelerator. This can be enabled with compiler option "-vec-report2". To compile and execute the code run

    $ icc vect-add.c -openmp_report2 -vec-report2 -o vect-add

    $ ./vect-add

Some interesting compiler flags useful not only for code debugging are:

!!! Note "Note" Debugging

openmp_report[0|1|2] - controls the compiler based vectorization diagnostic level
vec-report[0|1|2] - controls the OpenMP parallelizer diagnostic level

Performance ooptimization
xhost - FOR HOST ONLY - to generate AVX (Advanced Vector Extensions) instructions.

Automatic Offload using Intel MKL Library

Intel MKL includes an Automatic Offload (AO) feature that enables computationally intensive MKL functions called in user code to benefit from attached Intel Xeon Phi coprocessors automatically and transparently.

Behavioral of automatic offload mode is controlled by functions called within the program or by environmental variables. Complete list of controls is listed here.

The Automatic Offload may be enabled by either an MKL function call within the code:

    mkl_mic_enable();

or by setting environment variable

    $ export MKL_MIC_ENABLE=1

To get more information about automatic offload please refer to "Using Intel® MKL Automatic Offload on Intel ® Xeon Phi™ Coprocessors" white paper or Intel MKL documentation.

Automatic offload example

At first get an interactive PBS session on a node with MIC accelerator and load "intel" module that automatically loads "mkl" module as well.

    $ qsub -I -q qmic -A OPEN-0-0 -l select=1:ncpus=16
    $ module load intel

Following example show how to automatically offload an SGEMM (single precision - g dir="auto">eneral matrix multiply) function to MIC coprocessor. The code can be copied to a file and compiled without any necessary modification.

    $ vim sgemm-ao-short.c

    #include <stdio.h>
    #include <stdlib.h>
    #include <malloc.h>
    #include <stdint.h>

    #include "mkl.h"

    int main(int argc, char **argv)
    {
            float *A, *B, *C; /* Matrices */

            MKL_INT N = 2560; /* Matrix dimensions */
            MKL_INT LD = N; /* Leading dimension */
            int matrix_bytes; /* Matrix size in bytes */
            int matrix_elements; /* Matrix size in elements */

            float alpha = 1.0, beta = 1.0; /* Scaling factors */
            char transa = 'N', transb = 'N'; /* Transposition options */

            int i, j; /* Counters */

            matrix_elements = N * N;
            matrix_bytes = sizeof(float) * matrix_elements;

            /* Allocate the matrices */
            A = malloc(matrix_bytes); B = malloc(matrix_bytes); C = malloc(matrix_bytes);

            /* Initialize the matrices */
            for (i = 0; i < matrix_elements; i++) {
                    A[i] = 1.0; B[i] = 2.0; C[i] = 0.0;
            }

            printf("Computing SGEMM on the hostn");
            sgemm(&transa, &transb, &N, &N, &N, &alpha, A, &N, B, &N, &beta, C, &N);

            printf("Enabling Automatic Offloadn");
            /* Alternatively, set environment variable MKL_MIC_ENABLE=1 */
            mkl_mic_enable();

            int ndevices = mkl_mic_get_device_count(); /* Number of MIC devices */
            printf("Automatic Offload enabled: %d MIC devices presentn",   ndevices);

            printf("Computing SGEMM with automatic workdivisionn");
            sgemm(&transa, &transb, &N, &N, &N, &alpha, A, &N, B, &N, &beta, C, &N);

            /* Free the matrix memory */
            free(A); free(B); free(C);

            printf("Donen");

        return 0;
    }

!!! Note "Note" Please note: This example is simplified version of an example from MKL. The expanded version can be found here: $MKL_EXAMPLES/mic_ao/blasc/source/sgemm.c

To compile a code using Intel compiler use:

    $ icc -mkl sgemm-ao-short.c -o sgemm

For debugging purposes enable the offload report to see more information about automatic offloading.

    $ export OFFLOAD_REPORT=2

The output of a code should look similar to following listing, where lines starting with [MKL] are generated by offload reporting:

    Computing SGEMM on the host
    Enabling Automatic Offload
    Automatic Offload enabled: 1 MIC devices present
    Computing SGEMM with automatic workdivision
    [MKL] [MIC --] [AO Function]    SGEMM
    [MKL] [MIC --] [AO SGEMM Workdivision]  0.00 1.00
    [MKL] [MIC 00] [AO SGEMM CPU Time]      0.463351 seconds
    [MKL] [MIC 00] [AO SGEMM MIC Time]      0.179608 seconds
    [MKL] [MIC 00] [AO SGEMM CPU->MIC Data] 52428800 bytes
    [MKL] [MIC 00] [AO SGEMM MIC->CPU Data] 26214400 bytes
    Done

Native Mode

In the native mode a program is executed directly on Intel Xeon Phi without involvement of the host machine. Similarly to offload mode, the code is compiled on the host computer with Intel compilers.

To compile a code user has to be connected to a compute with MIC and load Intel compilers module. To get an interactive session on a compute node with an Intel Xeon Phi and load the module use following commands:

    $ qsub -I -q qmic -A NONE-0-0

    $ module load intel/13.5.192

!!! Note "Note" Please note that particular version of the Intel module is specified. This information is used later to specify the correct library paths.

To produce a binary compatible with Intel Xeon Phi architecture user has to specify "-mmic" compiler flag. Two compilation examples are shown below. The first example shows how to compile OpenMP parallel code "vect-add.c" for host only:

    $ icc -xhost -no-offload -fopenmp vect-add.c -o vect-add-host

To run this code on host, use:

    $ ./vect-add-host

The second example shows how to compile the same code for Intel Xeon Phi:

    $ icc -mmic -fopenmp vect-add.c -o vect-add-mic

Execution of the Program in Native Mode on Intel Xeon Phi

The user access to the Intel Xeon Phi is through the SSH. Since user home directories are mounted using NFS on the accelerator, users do not have to copy binary files or libraries between the host and accelerator.

To connect to the accelerator run:

    $ ssh mic0

If the code is sequential, it can be executed directly:

    mic0 $ ~/path_to_binary/vect-add-seq-mic

If the code is parallelized using OpenMP a set of additional libraries is required for execution. To locate these libraries new path has to be added to the LD_LIBRARY_PATH environment variable prior to the execution:

    mic0 $ export LD_LIBRARY_PATH=/apps/intel/composer_xe_2013.5.192/compiler/lib/mic:$LD_LIBRARY_PATH

!!! Note "Note" Please note that the path exported in the previous example contains path to a specific compiler (here the version is 5.192). This version number has to match with the version number of the Intel compiler module that was used to compile the code on the host computer.

For your information the list of libraries and their location required for execution of an OpenMP parallel code on Intel Xeon Phi is:

!!! Note "Note" /apps/intel/composer_xe_2013.5.192/compiler/lib/mic

- libiomp5.so
- libimf.so
- libsvml.so
- libirng.so
- libintlc.so.5

Finally, to run the compiled code use:

    $ ~/path_to_binary/vect-add-mic

OpenCL

OpenCL (Open Computing Language) is an open standard for general-purpose parallel programming for diverse mix of multi-core CPUs, GPU coprocessors, and other parallel processors. OpenCL provides a flexible execution model and uniform programming environment for software developers to write portable code for systems running on both the CPU and graphics processors or accelerators like the Intel® Xeon Phi.

On Anselm OpenCL is installed only on compute nodes with MIC accelerator, therefore OpenCL code can be compiled only on these nodes.

    module load opencl-sdk opencl-rt

Always load "opencl-sdk" (providing devel files like headers) and "opencl-rt" (providing dynamic library libOpenCL.so) modules to compile and link OpenCL code. Load "opencl-rt" for running your compiled code.

There are two basic examples of OpenCL code in the following directory:

    /apps/intel/opencl-examples/

First example "CapsBasic" detects OpenCL compatible hardware, here CPU and MIC, and prints basic information about the capabilities of it.

    /apps/intel/opencl-examples/CapsBasic/capsbasic

To compile and run the example copy it to your home directory, get a PBS interactive session on of the nodes with MIC and run make for compilation. Make files are very basic and shows how the OpenCL code can be compiled on Anselm.

    $ cp /apps/intel/opencl-examples/CapsBasic/* .
    $ qsub -I -q qmic -A NONE-0-0
    $ make

The compilation command for this example is:

    $ g++ capsbasic.cpp -lOpenCL -o capsbasic -I/apps/intel/opencl/include/

After executing the complied binary file, following output should be displayed.