docs.it4i/anselm-cluster-documentation/software/debuggers/score-p.md

+Score-P
+=======
+
+Introduction
+------------
+The [Score-P measurement infrastructure](http://www.vi-hps.org/projects/score-p/) is a highly scalable and easy-to-use tool suite for profiling, event tracing, and online analysis of HPC applications.
+
+Score-P can be used as an instrumentation tool for [Scalasca](scalasca/).
+
+Installed versions
+------------------
+There are currently two versions of Score-P version 1.2.6 [modules](../../environment-and-modules/) installed on Anselm :
+
+-   scorep/1.2.3-gcc-openmpi, for usage     with [GNU Compiler](../compilers/) and [OpenMPI](../mpi/Running_OpenMPI/)
+-   scorep/1.2.3-icc-impi, for usage with [Intel Compiler](../compilers.html)> and [Intel MPI](../mpi/running-mpich2/)>.
+
+Instrumentation
+---------------
+There are three ways to instrument your parallel applications in order to enable performance data collection:
+
+1.  Automated instrumentation using compiler
+2.  Manual instrumentation using API calls
+3.  Manual instrumentation using directives
+
+### Automated instrumentation
+
+is the easiest method. Score-P will automatically add instrumentation to every routine entry and exit using compiler hooks, and will intercept MPI calls and OpenMP regions. This method might, however, produce a large number of data. If you want to focus on profiler a specific regions of your code, consider using the manual instrumentation methods. To use automated instrumentation, simply prepend scorep to your compilation command. For example, replace:
+
+```bash
+$ mpif90 -c foo.f90
+$ mpif90 -c bar.f90
+$ mpif90 -o myapp foo.o bar.o
+```
+
+with:
+
+```bash
+$ scorep  mpif90 -c foo.f90
+$ scorep  mpif90 -c bar.f90
+$ scorep  mpif90 -o myapp foo.o bar.o
+```
+
+Usually your program is compiled using a Makefile or similar script, so it advisable to add the  scorep command to your definition of variables  CC, CXX, FCC etc.
+
+It is important that  scorep is prepended also to the linking command, in order to link with Score-P instrumentation libraries.
+
+###Manual instrumentation using API calls
+
+To use this kind of instrumentation, use scorep with switch --user. You will then mark regions to be instrumented by inserting API calls.
+
+An example in C/C++ :
+
+```cpp
+    #include <scorep/SCOREP_User.h>
+    void foo()
+    {
+        SCOREP_USER_REGION_DEFINE( my_region_handle )
+        // more declarations
+        SCOREP_USER_REGION_BEGIN( my_region_handle, "foo", SCOREP_USER_REGION_TYPE_COMMON )
+        // do something
+        SCOREP_USER_REGION_END( my_region_handle )
+    }
+```
+
+ and Fortran :
+
+```cpp
+    #include "scorep/SCOREP_User.inc"
+    subroutine foo
+        SCOREP_USER_REGION_DEFINE( my_region_handle )
+        ! more declarations
+        SCOREP_USER_REGION_BEGIN( my_region_handle, "foo", SCOREP_USER_REGION_TYPE_COMMON )
+        ! do something
+        SCOREP_USER_REGION_END( my_region_handle )
+    end subroutine foo
+```
+
+Please refer to the [documentation for description of the API](https://silc.zih.tu-dresden.de/scorep-current/pdf/scorep.pdf).
+
+###Manual instrumentation using directives
+
+This method uses POMP2 directives to mark regions to be instrumented. To use this method, use command  scorep --pomp.
+
+Example directives in C/C++ :
+
+```cpp
+    void foo(...)
+    {
+        /* declarations */
+        #pragma pomp inst begin(foo)
+        ...
+        if (<condition>)
+        {
+            #pragma pomp inst altend(foo)
+            return;
+        }
+        ...
+        #pragma pomp inst end(foo)
+    }
+```
+
+and in Fortran :
+
+```cpp
+    subroutine foo(...)
+        !declarations
+        !POMP$ INST BEGIN(foo)
+        ...
+        if (<condition>) then
+     !POMP$ INST ALTEND(foo)
+     return
+     end if
+     ...
+     !POMP$ INST END(foo)
+    end subroutine foo
+```
+
+The directives are ignored if the program is compiled without Score-P. Again, please refer to the [documentation](https://silc.zih.tu-dresden.de/scorep-current/pdf/scorep.pdf) for a more elaborate description.

docs.it4i/anselm-cluster-documentation/software/debuggers/total-view.md

+Total View
+==========
+
+##TotalView is a GUI-based source code multi-process, multi-thread debugger.
+
+License and Limitations for Anselm Users
+----------------------------------------
+On Anselm users can debug OpenMP or MPI code that runs up to 64 parallel processes. These limitation means that:
+
+```bash
+    1 user can debug up 64 processes, or
+    32 users can debug 2 processes, etc.
+```
+
+Debugging of GPU accelerated codes is also supported.
+
+You can check the status of the licenses here:
+
+```bash
+    cat /apps/user/licenses/totalview_features_state.txt
+
+    # totalview
+    # -------------------------------------------------
+    # FEATURE                       TOTAL   USED  AVAIL
+    # -------------------------------------------------
+    TotalView_Team                     64      0     64
+    Replay                             64      0     64
+    CUDA                               64      0     64
+```
+
+Compiling Code to run with TotalView
+------------------------------------
+### Modules
+
+Load all necessary modules to compile the code. For example:
+
+```bash
+    module load intel
+
+    module load impi   ... or ... module load openmpi/X.X.X-icc
+```
+
+Load the TotalView module:
+
+```bash
+    module load totalview/8.12
+```
+
+Compile the code:
+
+```bash
+    mpicc -g -O0 -o test_debug test.c
+
+    mpif90 -g -O0 -o test_debug test.f
+```
+
+### Compiler flags
+
+Before debugging, you need to compile your code with theses flags:
+
+!!! Note "Note"
+	**-g** : Generates extra debugging information usable by GDB. **-g3** includes even more debugging information. This option is available for GNU and INTEL C/C++ and Fortran compilers.
+
+	**-O0** : Suppress all optimizations.
+
+Starting a Job with TotalView
+-----------------------------
+Be sure to log in with an X window forwarding enabled. This could mean using the -X in the ssh:
+
+```bash
+    ssh -X username@anselm.it4i.cz
+```
+
+Other options is to access login node using VNC. Please see the detailed information on how to use graphic user interface on Anselm.
+
+From the login node an interactive session with X windows forwarding (-X option) can be started by following command:
+
+```bash
+    qsub -I -X -A NONE-0-0 -q qexp -lselect=1:ncpus=16:mpiprocs=16,walltime=01:00:00
+```
+
+Then launch the debugger with the totalview command followed by the name of the executable to debug.
+
+### Debugging a serial code
+
+To debug a serial code use:
+
+```bash
+    totalview test_debug
+```
+
+### Debugging a parallel code - option 1
+
+To debug a parallel code compiled with **OpenMPI** you need to setup your TotalView environment:
+
+!!! Note "Note"
+	**Please note:** To be able to run parallel debugging procedure from the command line without stopping the debugger in the mpiexec source code you have to add the following function to your **~/.tvdrc** file:
+
+```bash
+    proc mpi_auto_run_starter {loaded_id} {
+        set starter_programs {mpirun mpiexec orterun}
+        set executable_name [TV::symbol get $loaded_id full_pathname]
+        set file_component [file tail $executable_name]
+
+        if {[lsearch -exact $starter_programs $file_component] != -1} {
+            puts "*************************************"
+            puts "Automatically starting $file_component"
+            puts "*************************************"
+            dgo
+        }
+    }
+
+    # Append this function to TotalView's image load callbacks so that
+    # TotalView run this program automatically.
+
+    dlappend TV::image_load_callbacks mpi_auto_run_starter
+```
+The source code of this function can be also found in
+
+```bash
+    /apps/mpi/openmpi/intel/1.6.5/etc/openmpi-totalview.tcl
+```
+
+!!! Note "Note"
+	You can also add only following line to you ~/.tvdrc file instead of the entire function: 
+    **source /apps/mpi/openmpi/intel/1.6.5/etc/openmpi-totalview.tcl**
+
+You need to do this step only once.
+
+Now you can run the parallel debugger using:
+
+```bash
+    mpirun -tv -n 5 ./test_debug
+```
+
+When following dialog appears click on "Yes"
+
+![](../../../img/totalview1.png)
+
+At this point the main TotalView GUI window will appear and you can insert the breakpoints and start debugging:
+
+![](../../../img/totalview2.png)
+
+### Debugging a parallel code - option 2
+
+Other option to start new parallel debugging session from a command line is to let TotalView to execute mpirun by itself. In this case user has to specify a MPI implementation used to compile the source code.
+
+The following example shows how to start debugging session with Intel MPI:
+
+```bash
+    module load intel/13.5.192 impi/4.1.1.036 totalview/8/13
+
+    totalview -mpi "Intel MPI-Hydra" -np 8 ./hello_debug_impi
+```
+
+After running previous command you will see the same window as shown in the screenshot above.
+
+More information regarding the command line parameters of the TotalView can be found TotalView Reference Guide, Chapter 7: TotalView Command Syntax.
+
+Documentation
+-------------
+[1] The [TotalView documentation](http://www.roguewave.com/support/product-documentation/totalview-family.aspx#totalview) web page is a good resource for learning more about some of the advanced TotalView features.

docs.it4i/anselm-cluster-documentation/software/debuggers/valgrind.md

+Valgrind
+========
+
+Valgrind is a tool for memory debugging and profiling.
+
+About Valgrind
+--------------
+Valgrind is an open-source tool, used mainly for debuggig memory-related problems, such as memory leaks, use of uninitalized memory etc. in C/C++ applications. The toolchain was however extended over time with more functionality, such as debugging of threaded applications, cache profiling, not limited only to C/C++.
+
+Valgind is an extremely useful tool for debugging memory errors such as [off-by-one](http://en.wikipedia.org/wiki/Off-by-one_error). Valgrind uses a virtual machine and dynamic recompilation of binary code, because of that, you can expect that programs being debugged by Valgrind run 5-100 times slower.
+
+The main tools available in Valgrind are :
+
+-   **Memcheck**, the original, must used and default tool. Verifies memory access in you program and can detect use of unitialized memory, out of bounds memory access, memory leaks, double free, etc.
+-   **Massif**, a heap profiler.
+-   **Hellgrind** and **DRD** can detect race conditions in multi-threaded applications.
+-   **Cachegrind**, a cache profiler.
+-   **Callgrind**, a callgraph analyzer.
+-   For a full list and detailed documentation, please refer to the [official Valgrind documentation](http://valgrind.org/docs/).
+
+Installed versions
+------------------
+There are two versions of Valgrind available on Anselm.
+
+-   Version 3.6.0, installed by operating system vendor in /usr/bin/valgrind. This version is available by default, without the need to load any module. This version however does not provide additional MPI support.
+-   Version 3.9.0 with support for Intel MPI, available in [module](../../environment-and-modules/) valgrind/3.9.0-impi. After loading the module, this version replaces the default valgrind.
+
+Usage
+-----
+Compile the application which you want to debug as usual. It is advisable to add compilation flags  -g (to add debugging information to the binary so that you will see original source code lines in the output) and  -O0 (to disable compiler optimizations).
+
+For example, lets look at this C code, which has two problems :
+
+```cpp
+    #include <stdlib.h>
+
+    void f(void)
+    {
+       int* x = malloc(10 * sizeof(int));
+       x[10] = 0; // problem 1: heap block overrun
+    }             // problem 2: memory leak -- x not freed
+
+    int main(void)
+    {
+       f();
+       return 0;
+    }
+```
+
+Now, compile it with Intel compiler :
+
+```bash
+    $ module add intel
+    $ icc -g valgrind-example.c -o valgrind-example
+```
+
+Now, lets run it with Valgrind. The syntax is :
+
+ *valgrind [valgrind options] &lt;your program binary&gt; [your program options]*
+
+If no Valgrind options are specified, Valgrind defaults to running Memcheck tool. Please refer to the Valgrind documentation for a full description of command line options.
+
+```bash
+    $ valgrind ./valgrind-example
+    ==12652== Memcheck, a memory error detector
+    ==12652== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
+    ==12652== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
+    ==12652== Command: ./valgrind-example
+    ==12652==
+    ==12652== Invalid write of size 4
+    ==12652== at 0x40053E: f (valgrind-example.c:6)
+    ==12652== by 0x40054E: main (valgrind-example.c:11)
+    ==12652== Address 0x5861068 is 0 bytes after a block of size 40 alloc'd
+    ==12652== at 0x4C27AAA: malloc (vg_replace_malloc.c:291)
+    ==12652== by 0x400528: f (valgrind-example.c:5)
+    ==12652== by 0x40054E: main (valgrind-example.c:11)
+    ==12652==
+    ==12652==
+    ==12652== HEAP SUMMARY:
+    ==12652== in use at exit: 40 bytes in 1 blocks
+    ==12652== total heap usage: 1 allocs, 0 frees, 40 bytes allocated
+    ==12652==
+    ==12652== LEAK SUMMARY:
+    ==12652== definitely lost: 40 bytes in 1 blocks
+    ==12652== indirectly lost: 0 bytes in 0 blocks
+    ==12652== possibly lost: 0 bytes in 0 blocks
+    ==12652== still reachable: 0 bytes in 0 blocks
+    ==12652== suppressed: 0 bytes in 0 blocks
+    ==12652== Rerun with --leak-check=full to see details of leaked memory
+    ==12652==
+    ==12652== For counts of detected and suppressed errors, rerun with: -v
+    ==12652== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 6 from 6)
+```
+
+In the output we can see that Valgrind has detected both errors - the off-by-one memory access at line 5 and a memory leak of 40 bytes. If we want a detailed analysis of the memory leak, we need to run Valgrind with  --leak-check=full option :
+
+```bash
+    $ valgrind --leak-check=full ./valgrind-example
+    ==23856== Memcheck, a memory error detector
+    ==23856== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
+    ==23856== Using Valgrind-3.6.0 and LibVEX; rerun with -h for copyright info
+    ==23856== Command: ./valgrind-example
+    ==23856==
+    ==23856== Invalid write of size 4
+    ==23856== at 0x40067E: f (valgrind-example.c:6)
+    ==23856== by 0x40068E: main (valgrind-example.c:11)
+    ==23856== Address 0x66e7068 is 0 bytes after a block of size 40 alloc'd
+    ==23856== at 0x4C26FDE: malloc (vg_replace_malloc.c:236)
+    ==23856== by 0x400668: f (valgrind-example.c:5)
+    ==23856== by 0x40068E: main (valgrind-example.c:11)
+    ==23856==
+    ==23856==
+    ==23856== HEAP SUMMARY:
+    ==23856== in use at exit: 40 bytes in 1 blocks
+    ==23856== total heap usage: 1 allocs, 0 frees, 40 bytes allocated
+    ==23856==
+    ==23856== 40 bytes in 1 blocks are definitely lost in loss record 1 of 1
+    ==23856== at 0x4C26FDE: malloc (vg_replace_malloc.c:236)
+    ==23856== by 0x400668: f (valgrind-example.c:5)
+    ==23856== by 0x40068E: main (valgrind-example.c:11)
+    ==23856==
+    ==23856== LEAK SUMMARY:
+    ==23856== definitely lost: 40 bytes in 1 blocks
+    ==23856== indirectly lost: 0 bytes in 0 blocks
+    ==23856== possibly lost: 0 bytes in 0 blocks
+    ==23856== still reachable: 0 bytes in 0 blocks
+    ==23856== suppressed: 0 bytes in 0 blocks
+    ==23856==
+    ==23856== For counts of detected and suppressed errors, rerun with: -v
+    ==23856== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 6 from 6)
+```
+
+Now we can see that the memory leak is due to the malloc() at line 6.
+
+Usage with MPI
+---------------------------
+Although Valgrind is not primarily a parallel debugger, it can be used to debug parallel applications as well. When launching your parallel applications, prepend the valgrind command. For example :
+
+```bash
+    $ mpirun -np 4 valgrind myapplication
+```
+
+The default version without MPI support will however report a large number of false errors in the MPI library, such as :
+
+```bash
+    ==30166== Conditional jump or move depends on uninitialised value(s)
+    ==30166== at 0x4C287E8: strlen (mc_replace_strmem.c:282)
+    ==30166== by 0x55443BD: I_MPI_Processor_model_number (init_interface.c:427)
+    ==30166== by 0x55439E0: I_MPI_Processor_arch_code (init_interface.c:171)
+    ==30166== by 0x558D5AE: MPID_nem_impi_init_shm_configuration (mpid_nem_impi_extensions.c:1091)
+    ==30166== by 0x5598F4C: MPID_nem_init_ckpt (mpid_nem_init.c:566)
+    ==30166== by 0x5598B65: MPID_nem_init (mpid_nem_init.c:489)
+    ==30166== by 0x539BD75: MPIDI_CH3_Init (ch3_init.c:64)
+    ==30166== by 0x5578743: MPID_Init (mpid_init.c:193)
+    ==30166== by 0x554650A: MPIR_Init_thread (initthread.c:539)
+    ==30166== by 0x553369F: PMPI_Init (init.c:195)
+    ==30166== by 0x4008BD: main (valgrind-example-mpi.c:18)
+```
+
+so it is better to use the MPI-enabled valgrind from module. The MPI version requires library /apps/tools/valgrind/3.9.0/impi/lib/valgrind/libmpiwrap-amd64-linux.so, which must be included in the  LD_PRELOAD environment variable.
+
+Lets look at this MPI example :
+
+```cpp
+    #include <stdlib.h>
+    #include <mpi.h>
+
+    int main(int argc, char *argv[])
+    {
+            int *data = malloc(sizeof(int)*99);
+
+            MPI_Init(&argc, &argv);
+            MPI_Bcast(data, 100, MPI_INT, 0, MPI_COMM_WORLD);
+            MPI_Finalize();
+
+            return 0;
+    }
+```
+
+There are two errors - use of uninitialized memory and invalid length of the buffer. Lets debug it with valgrind :
+
+```bash
+    $ module add intel impi
+    $ mpicc -g valgrind-example-mpi.c -o valgrind-example-mpi
+    $ module add valgrind/3.9.0-impi
+    $ mpirun -np 2 -env LD_PRELOAD /apps/tools/valgrind/3.9.0/impi/lib/valgrind/libmpiwrap-amd64-linux.so valgrind ./valgrind-example-mpi
+```
+
+Prints this output : (note that there is output printed for every launched MPI process)
+
+```bash
+    ==31318== Memcheck, a memory error detector
+    ==31318== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
+    ==31318== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
+    ==31318== Command: ./valgrind-example-mpi
+    ==31318==
+    ==31319== Memcheck, a memory error detector
+    ==31319== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
+    ==31319== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
+    ==31319== Command: ./valgrind-example-mpi
+    ==31319==
+    valgrind MPI wrappers 31319: Active for pid 31319
+    valgrind MPI wrappers 31319: Try MPIWRAP_DEBUG=help for possible options
+    valgrind MPI wrappers 31318: Active for pid 31318
+    valgrind MPI wrappers 31318: Try MPIWRAP_DEBUG=help for possible options
+    ==31319== Unaddressable byte(s) found during client check request
+    ==31319== at 0x4E35974: check_mem_is_addressable_untyped (libmpiwrap.c:960)
+    ==31319== by 0x4E5D0FE: PMPI_Bcast (libmpiwrap.c:908)
+    ==31319== by 0x400911: main (valgrind-example-mpi.c:20)
+    ==31319== Address 0x69291cc is 0 bytes after a block of size 396 alloc'd
+    ==31319== at 0x4C27AAA: malloc (vg_replace_malloc.c:291)
+    ==31319== by 0x4007BC: main (valgrind-example-mpi.c:8)
+    ==31319==
+    ==31318== Uninitialised byte(s) found during client check request
+    ==31318== at 0x4E3591D: check_mem_is_defined_untyped (libmpiwrap.c:952)
+    ==31318== by 0x4E5D06D: PMPI_Bcast (libmpiwrap.c:908)
+    ==31318== by 0x400911: main (valgrind-example-mpi.c:20)
+    ==31318== Address 0x6929040 is 0 bytes inside a block of size 396 alloc'd
+    ==31318== at 0x4C27AAA: malloc (vg_replace_malloc.c:291)
+    ==31318== by 0x4007BC: main (valgrind-example-mpi.c:8)
+    ==31318==
+    ==31318== Unaddressable byte(s) found during client check request
+    ==31318== at 0x4E3591D: check_mem_is_defined_untyped (libmpiwrap.c:952)
+    ==31318== by 0x4E5D06D: PMPI_Bcast (libmpiwrap.c:908)
+    ==31318== by 0x400911: main (valgrind-example-mpi.c:20)
+    ==31318== Address 0x69291cc is 0 bytes after a block of size 396 alloc'd
+    ==31318== at 0x4C27AAA: malloc (vg_replace_malloc.c:291)
+    ==31318== by 0x4007BC: main (valgrind-example-mpi.c:8)
+    ==31318==
+    ==31318==
+    ==31318== HEAP SUMMARY:
+    ==31318== in use at exit: 3,172 bytes in 67 blocks
+    ==31318== total heap usage: 191 allocs, 124 frees, 81,203 bytes allocated
+    ==31318==
+    ==31319==
+    ==31319== HEAP SUMMARY:
+    ==31319== in use at exit: 3,172 bytes in 67 blocks
+    ==31319== total heap usage: 175 allocs, 108 frees, 48,435 bytes allocated
+    ==31319==
+    ==31318== LEAK SUMMARY:
+    ==31318== definitely lost: 408 bytes in 3 blocks
+    ==31318== indirectly lost: 256 bytes in 1 blocks
+    ==31318== possibly lost: 0 bytes in 0 blocks
+    ==31318== still reachable: 2,508 bytes in 63 blocks
+    ==31318== suppressed: 0 bytes in 0 blocks
+    ==31318== Rerun with --leak-check=full to see details of leaked memory
+    ==31318==
+    ==31318== For counts of detected and suppressed errors, rerun with: -v
+    ==31318== Use --track-origins=yes to see where uninitialised values come from
+    ==31318== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 4 from 4)
+    ==31319== LEAK SUMMARY:
+    ==31319== definitely lost: 408 bytes in 3 blocks
+    ==31319== indirectly lost: 256 bytes in 1 blocks
+    ==31319== possibly lost: 0 bytes in 0 blocks
+    ==31319== still reachable: 2,508 bytes in 63 blocks
+    ==31319== suppressed: 0 bytes in 0 blocks
+    ==31319== Rerun with --leak-check=full to see details of leaked memory
+    ==31319==
+    ==31319== For counts of detected and suppressed errors, rerun with: -v
+    ==31319== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 4 from 4)
+```
+
+We can see that Valgrind has reported use of unitialised memory on the master process (which reads the array to be broadcasted) and use of unaddresable memory on both processes.

docs.it4i/anselm-cluster-documentation/software/debuggers/vampir.md

+hVampir
+======
+
+Vampir is a commercial trace analysis and visualisation tool. It can work with traces in OTF and OTF2 formats. It does not have the functionality to collect traces, you need to use a trace collection tool (such as [Score-P](../../../salomon/software/debuggers/score-p/)) first to collect the traces.
+
+![](../../../img/Snmekobrazovky20160708v12.33.35.png)
+
+Installed versions
+------------------
+Version 8.5.0 is currently installed as module Vampir/8.5.0 :
+
+```bash
+    $ module load Vampir/8.5.0
+    $ vampir &
+```
+
+User manual
+-----------
+You can find the detailed user manual in PDF format in $EBROOTVAMPIR/doc/vampir-manual.pdf
+
+References
+----------
+[1].  <https://www.vampir.eu>

docs.it4i/anselm-cluster-documentation/software/gpi2.md

+GPI-2
+=====
+
+##A library that implements the GASPI specification
+
+Introduction
+------------
+Programming Next Generation Supercomputers: GPI-2 is an API library for asynchronous interprocess, cross-node communication. It provides a flexible, scalable and fault tolerant interface for parallel applications.
+
+The GPI-2 library ([www.gpi-site.com/gpi2/](http://www.gpi-site.com/gpi2/)) implements the GASPI specification (Global Address Space Programming Interface, [www.gaspi.de](http://www.gaspi.de/en/project.html)). GASPI is a Partitioned Global Address Space (PGAS) API. It aims at scalable, flexible and failure tolerant computing in massively parallel environments.
+
+Modules
+-------
+The GPI-2, version 1.0.2 is available on Anselm via module gpi2:
+
+```bash
+    $ module load gpi2
+```
+
+The module sets up environment variables, required for linking and running GPI-2 enabled applications. This particular command loads the default module, which is gpi2/1.0.2
+
+Linking
+-------
+!!! Note "Note"
+	Link with -lGPI2 -libverbs
+
+Load the gpi2 module. Link using **-lGPI2** and **-libverbs** switches to link your code against GPI-2. The GPI-2 requires the OFED infinband communication library ibverbs.
+
+### Compiling and linking with Intel compilers
+
+```bash
+    $ module load intel
+    $ module load gpi2
+    $ icc myprog.c -o myprog.x -Wl,-rpath=$LIBRARY_PATH -lGPI2 -libverbs
+```
+
+### Compiling and linking with GNU compilers
+
+```bash
+    $ module load gcc
+    $ module load gpi2
+    $ gcc myprog.c -o myprog.x -Wl,-rpath=$LIBRARY_PATH -lGPI2 -libverbs
+```
+
+Running the GPI-2 codes
+-----------------------
+
+!!! Note "Note"
+	gaspi_run starts the GPI-2 application
+
+The gaspi_run utility is used to start and run GPI-2 applications:
+
+```bash
+    $ gaspi_run -m machinefile ./myprog.x
+```
+
+A machine file (**machinefile**) with the hostnames of nodes where the application will run, must be provided. The machinefile lists all nodes on which to run, one entry per node per process. This file may be hand created or obtained from standard $PBS_NODEFILE:
+
+```bash
+    $ cut -f1 -d"." $PBS_NODEFILE > machinefile
+```
+
+machinefile:
+
+```bash
+    cn79
+    cn80
+```
+
+This machinefile will run 2 GPI-2 processes, one on node cn79 other on node cn80.
+
+machinefle:
+
+```bash
+    cn79
+    cn79
+    cn80
+    cn80
+```
+
+This machinefile will run 4 GPI-2 processes, 2 on node cn79 o 2 on node cn80.
+
+!!! Note "Note"
+	Use the **mpiprocs** to control how many GPI-2 processes will run per node
+
+Example:
+
+```bash
+    $ qsub -A OPEN-0-0 -q qexp -l select=2:ncpus=16:mpiprocs=16 -I
+```
+
+This example will produce $PBS_NODEFILE with 16 entries per node.
+
+### gaspi_logger
+
+!!! Note "Note"
+	gaspi_logger views the output form GPI-2 application ranks
+
+The gaspi_logger utility is used to view the output from all nodes except the master node (rank 0). The gaspi_logger is started, on another session, on the master node - the node where the gaspi_run is executed. The output of the application, when called with gaspi_printf(), will be redirected to the gaspi_logger. Other I/O routines (e.g. printf) will not.
+
+Example
+-------
+
+Following is an example GPI-2 enabled code:
+
+```cpp
+    #include <GASPI.h>
+    #include <stdlib.h>
+
+    void success_or_exit ( const char* file, const int line, const int ec)
+    {
+      if (ec != GASPI_SUCCESS)
+        {
+          gaspi_printf ("Assertion failed in %s[%i]:%dn", file, line, ec);
+          exit (1);
+        }
+    }
+
+    #define ASSERT(ec) success_or_exit (__FILE__, __LINE__, ec);
+
+    int main(int argc, char *argv[])
+    {
+      gaspi_rank_t rank, num;
+      gaspi_return_t ret;
+
+      /* Initialize GPI-2 */
+      ASSERT( gaspi_proc_init(GASPI_BLOCK) );
+
+      /* Get ranks information */
+      ASSERT( gaspi_proc_rank(&rank) );
+      ASSERT( gaspi_proc_num(&num) );
+
+      gaspi_printf("Hello from rank %d of %dn",
+               rank, num);
+
+      /* Terminate */
+      ASSERT( gaspi_proc_term(GASPI_BLOCK) );
+
+      return 0;
+    }
+```
+
+Load modules and compile:
+
+```bash
+    $ module load gcc gpi2
+    $ gcc helloworld_gpi.c -o helloworld_gpi.x -Wl,-rpath=$LIBRARY_PATH -lGPI2 -libverbs
+```
+
+Submit the job and run the GPI-2 application
+
+```bash
+    $ qsub -q qexp -l select=2:ncpus=1:mpiprocs=1,place=scatter,walltime=00:05:00 -I
+    qsub: waiting for job 171247.dm2 to start
+    qsub: job 171247.dm2 ready
+
+    cn79 $ module load gpi2
+    cn79 $ cut -f1 -d"." $PBS_NODEFILE > machinefile
+    cn79 $ gaspi_run -m machinefile ./helloworld_gpi.x
+    Hello from rank 0 of 2
+```
+
+At the same time, in another session, you may start the gaspi logger:
+
+```bash
+    $ ssh cn79
+    cn79 $ gaspi_logger
+    GASPI Logger (v1.1)
+    [cn80:0] Hello from rank 1 of 2
+```
+
+In this example, we compile the helloworld_gpi.c code using the **gnu compiler** (gcc) and link it to the GPI-2 and ibverbs library. The library search path is compiled in. For execution, we use the qexp queue, 2 nodes 1 core each. The GPI module must be loaded on the master compute node (in this example the cn79), gaspi_logger is used from different session to view the output of the second process.

docs.it4i/anselm-cluster-documentation/software/index.md

+Anselm Cluster Software
+===
+
+## [Modules](../../modules-anselm)
+* List of available modules
+## [COMSOL](comsol-multiphysics)
+* A finite element analysis, solver and Simulation software
+## [ParaView](paraview)
+* An open-source, multi-platform data analysis and visualization application
+## [Compilers](compilers)
+* Available compilers, including GNU, INTEL and UPC compilers
+## [nVidia CUDA](nvidia-cuda)
+* A guide to nVidia CUDA programming and GPU usage
+## [GPI-2](gpi2)
+* A library that implements the GASPI specification
+## [OpenFOAM](openfoam)
+* A free, open source CFD software package
+## [ISV Licenses](isv_licenses)
+* A guide to managing Independent Software Vendor licences
+## [Intel Xeon Phi](intel-xeon-phi)
+* A guide to Intel Xeon Phi usage
+## [Virtualization](kvirtualization)
+## [Java](java)
+* Java on ANSELM
+## [Operating System](operating-system)
+* The operating system, deployed on ANSELM
+##  Intel Suite
+* The Intel Parallel Studio XE
+### [Introduction](intel-suite/introduction)
+### [Intel MKL](intel-suite/intel-mkl)
+### [Intel Compilers](intel-suite/intel-compilers)
+### [Intel IPP](intel-suite/intel-integrated-performance-primitives)
+### [Intel TBB](intel-suite/intel-tbb)
+### [Intel Debugger](intel-suite/intel-debugger)
+##  MPI
+* Message Passing Interface libraries
+### [Introduction](mpi/mpi)
+### [MPI4Py (MPI for Python)](mpi/mpi4py-mpi-for-python)
+### [Running OpenMPI](mpi/Running_OpenMPI)
+### [Running MPICH2](mpi/running-mpich2)
+##  Numerical Libraries
+* Libraries for numerical computations
+### [Intel numerical libraries](numerical-libraries/intel-numerical-libraries)
+### [PETSc](numerical-libraries/petsc)
+### [Trilinos](numerical-libraries/trilinos)
+### [FFTW](numerical-libraries/fftw)
+### [GSL](numerical-libraries/gsl)
+### [MAGMA for Intel Xeon Phi](numerical-libraries/magma-for-intel-xeon-phi)
+### [HDF5](numerical-libraries/hdf5)
+##  Omics Master
+### [Diagnostic component (TEAM)](omics-master/diagnostic-component-team)
+### [Priorization component (BiERApp)](omics-master/priorization-component-bierapp)
+### [Overview](omics-master/overview)
+##  Debuggers
+* A collection of development tools
+### [Valgrind](debuggers/valgrind)
+### [PAPI](debuggers/papi)
+### [Allinea Forge (DDT,MAP)](debuggers/allinea-ddt)
+### [Total View](debuggers/total-view)
+### [CUBE](debuggers/cube)
+### [Intel VTune Amplifier](debuggers/intel-vtune-amplifier)
+### [VNC](debuggers/debuggers)
+### [Scalasca](debuggers/scalasca)
+### [Score-P](debuggers/score-p)
+### [Intel Performance Counter Monitor](debuggers/intel-performance-counter-monitor)
+### [Allinea Performance Reports](debuggers/allinea-performance-reports)
+### [Vampir](debuggers/vampir)
+##  Numerical Languages
+* Interpreted languages for numerical computations
+### [Introduction](numerical-languages/introduction)
+### [R](numerical-languages/r)
+### [Matlab 2013-2014](numerical-languages/matlab_1314)
+### [Matlab](numerical-languages/matlab)
+### [Octave](numerical-languages/octave)
+##  Chemistry
+* Tools for computational chemistry
+### [Molpro](chemistry/molpro)
+### [NWChem](chemistry/nwchem)
+##  Ansys
+* An engineering simulation software
+### [Introduction](ansys/ansys)
+### [ANSYS CFX](ansys/ansys-cfx)
+### [ANSYS LS-DYNA](ansys/ansys-ls-dyna)
+### [ANSYS MAPDL](ansys/ansys-mechanical-apdl)
+### [LS-DYNA](ansys/ls-dyna)
+### [ANSYS Fluent](ansys/ansys-fluent)

docs.it4i/anselm-cluster-documentation/software/intel-suite/intel-compilers.md

+Intel Compilers
+===============
+
+The Intel compilers version 13.1.1 are available, via module intel. The compilers include the icc C and C++ compiler and the ifort fortran 77/90/95 compiler.
+
+```bash
+    $ module load intel
+    $ icc -v
+    $ ifort -v
+```
+
+The intel compilers provide for vectorization of the code, via the AVX instructions and support threading parallelization via OpenMP
+
+For maximum performance on the Anselm cluster, compile your programs using the AVX instructions, with reporting where the vectorization was used. We recommend following compilation options for high performance
+
+```bash
+    $ icc   -ipo -O3 -vec -xAVX -vec-report1 myprog.c mysubroutines.c -o myprog.x
+    $ ifort -ipo -O3 -vec -xAVX -vec-report1 myprog.f mysubroutines.f -o myprog.x
+```
+
+In this example, we compile the program enabling interprocedural optimizations between source files (-ipo), aggresive loop optimizations (-O3) and vectorization (-vec -xAVX)
+
+The compiler recognizes the omp, simd, vector and ivdep pragmas for OpenMP parallelization and AVX vectorization. Enable the OpenMP parallelization by the **-openmp** compiler switch.
+
+```bash
+    $ icc -ipo -O3 -vec -xAVX -vec-report1 -openmp myprog.c mysubroutines.c -o myprog.x
+    $ ifort -ipo -O3 -vec -xAVX -vec-report1 -openmp myprog.f mysubroutines.f -o myprog.x
+```
+
+Read more at <http://software.intel.com/sites/products/documentation/doclib/stdxe/2013/composerxe/compiler/cpp-lin/index.htm>
+
+Sandy Bridge/Haswell binary compatibility
+-----------------------------------------
+Anselm nodes are currently equipped with Sandy Bridge CPUs, while Salomon will use Haswell architecture. >The new processors are backward compatible with the Sandy Bridge nodes, so all programs that ran on the Sandy Bridge processors, should also run on the new Haswell nodes. >To get optimal performance out of the Haswell processors a program should make use of the special AVX2 instructions for this processor. One can do this by recompiling codes with the compiler flags >designated to invoke these instructions. For the Intel compiler suite, there are two ways of doing this:
+
+-   Using compiler flag (both for Fortran and C): -xCORE-AVX2. This will create a binary with AVX2 instructions, specifically for the Haswell processors. Note that the executable will not run on Sandy Bridge nodes.
+-   Using compiler flags (both for Fortran and C): -xAVX -axCORE-AVX2. This will generate multiple, feature specific auto-dispatch code paths for Intel® processors, if there is a performance benefit. So this binary will run both on Sandy Bridge and Haswell processors. During runtime it will be decided which path to follow, dependent on which processor you are running on. In general this will result in larger binaries.

docs.it4i/anselm-cluster-documentation/software/intel-suite/intel-debugger.md

+Intel Debugger
+==============
+
+Debugging serial applications
+-----------------------------
+The intel debugger version 13.0 is available, via module intel. The debugger works for applications compiled with C and C++ compiler and the ifort fortran 77/90/95 compiler. The debugger provides java GUI environment. Use X display for running the GUI.
+
+```bash
+    $ module load intel
+    $ idb
+```
+
+The debugger may run in text mode. To debug in text mode, use
+
+```bash
+    $ idbc
+```
+
+To debug on the compute nodes, module intel must be loaded. The GUI on compute nodes may be accessed using the same way as in the GUI section
+
+Example:
+
+```bash
+    $ qsub -q qexp -l select=1:ncpus=16 -X -I
+    qsub: waiting for job 19654.srv11 to start
+    qsub: job 19654.srv11 ready
+
+    $ module load intel
+    $ module load java
+    $ icc -O0 -g myprog.c -o myprog.x
+    $ idb ./myprog.x
+```
+
+In this example, we allocate 1 full compute node, compile program myprog.c with debugging options -O0 -g and run the idb debugger interactively on the myprog.x executable. The GUI access is via X11 port forwarding provided by the PBS workload manager.
+
+Debugging parallel applications
+-------------------------------
+Intel debugger is capable of debugging multithreaded and MPI parallel programs as well.
+
+### Small number of MPI ranks
+
+For debugging small number of MPI ranks, you may execute and debug each rank in separate xterm terminal (do not forget the X display. Using Intel MPI, this may be done in following way:
+
+```bash
+    $ qsub -q qexp -l select=2:ncpus=16 -X -I
+    qsub: waiting for job 19654.srv11 to start
+    qsub: job 19655.srv11 ready
+
+    $ module load intel impi
+    $ mpirun -ppn 1 -hostfile $PBS_NODEFILE --enable-x xterm -e idbc ./mympiprog.x
+```
+
+In this example, we allocate 2 full compute node, run xterm on each node and start idb debugger in command line mode, debugging two ranks of mympiprog.x application. The xterm will pop up for each rank, with idb prompt ready. The example is not limited to use of Intel MPI
+
+### Large number of MPI ranks
+
+Run the idb debugger from within the MPI debug option. This will cause the debugger to bind to all ranks and provide aggregated outputs across the ranks, pausing execution automatically just after startup. You may then set break points and step the execution manually. Using Intel MPI:
+
+```bash
+    $ qsub -q qexp -l select=2:ncpus=16 -X -I
+    qsub: waiting for job 19654.srv11 to start
+    qsub: job 19655.srv11 ready
+
+    $ module load intel impi
+    $ mpirun -n 32 -idb ./mympiprog.x
+```
+
+### Debugging multithreaded application
+
+Run the idb debugger in GUI mode. The menu Parallel contains number of tools for debugging multiple threads. One of the most useful tools is the **Serialize Execution** tool, which serializes execution of concurrent threads for easy orientation and identification of concurrency related bugs.
+
+Further information
+-------------------
+Exhaustive manual on idb features and usage is published at [Intel website](http://software.intel.com/sites/products/documentation/doclib/stdxe/2013/composerxe/debugger/user_guide/index.htm)
+

docs.it4i/anselm-cluster-documentation/software/intel-suite/intel-integrated-performance-primitives.md

+Intel IPP
+=========
+
+Intel Integrated Performance Primitives
+---------------------------------------
+Intel Integrated Performance Primitives, version 7.1.1, compiled for AVX vector instructions is available, via module ipp. The IPP is a very rich library of highly optimized algorithmic building blocks for media and data applications. This includes signal, image and frame processing algorithms, such as FFT, FIR, Convolution, Optical Flow, Hough transform, Sum, MinMax, as well as cryptographic functions, linear algebra functions and many more.
+
+!!! Note "Note"
+	Check out IPP before implementing own math functions for data processing, it is likely already there.
+
+```bash
+    $ module load ipp
+```
+
+The module sets up environment variables, required for linking and running ipp enabled applications.
+
+IPP example
+-----------
+
+```cpp
+    #include "ipp.h"
+    #include <stdio.h>
+    int main(int argc, char* argv[])
+    {
+            const IppLibraryVersion *lib;
+            Ipp64u fm;
+            IppStatus status;
+
+            status= ippInit();            //IPP initialization with the best optimization layer
+            if( status != ippStsNoErr ) {
+                    printf("IppInit() Error:n");
+                    printf("%sn", ippGetStatusString(status) );
+                    return -1;
+            }
+
+            //Get version info
+            lib = ippiGetLibVersion();
+            printf("%s %sn", lib->Name, lib->Version);
+
+            //Get CPU features enabled with selected library level
+            fm=ippGetEnabledCpuFeatures();
+            printf("SSE    :%cn",(fm>1)&1?'Y':'N');
+            printf("SSE2   :%cn",(fm>2)&1?'Y':'N');
+            printf("SSE3   :%cn",(fm>3)&1?'Y':'N');
+            printf("SSSE3  :%cn",(fm>4)&1?'Y':'N');
+            printf("SSE41  :%cn",(fm>6)&1?'Y':'N');
+            printf("SSE42  :%cn",(fm>7)&1?'Y':'N');
+            printf("AVX    :%cn",(fm>8)&1 ?'Y':'N');
+            printf("AVX2   :%cn", (fm>15)&1 ?'Y':'N' );
+            printf("----------n");
+            printf("OS Enabled AVX :%cn", (fm>9)&1 ?'Y':'N');
+            printf("AES            :%cn", (fm>10)&1?'Y':'N');
+            printf("CLMUL          :%cn", (fm>11)&1?'Y':'N');
+            printf("RDRAND         :%cn", (fm>13)&1?'Y':'N');
+            printf("F16C           :%cn", (fm>14)&1?'Y':'N');
+
+            return 0;
+    }
+```
+
+Compile above example, using any compiler and the ipp module.
+
+```bash
+    $ module load intel
+    $ module load ipp
+
+    $ icc testipp.c -o testipp.x -lippi -lipps -lippcore
+```
+
+You will need the ipp module loaded to run the ipp enabled executable. This may be avoided, by compiling library search paths into the executable
+
+```bash
+    $ module load intel
+    $ module load ipp
+
+    $ icc testipp.c -o testipp.x -Wl,-rpath=$LIBRARY_PATH -lippi -lipps -lippcore
+```
+
+Code samples and documentation
+------------------------------
+Intel provides number of [Code Samples for IPP](https://software.intel.com/en-us/articles/code-samples-for-intel-integrated-performance-primitives-library), illustrating use of IPP.
+
+Read full documentation on IPP [on Intel website,](http://software.intel.com/sites/products/search/search.php?q=&x=15&y=6&product=ipp&version=7.1&docos=lin) in particular the [IPP Reference manual.](http://software.intel.com/sites/products/documentation/doclib/ipp_sa/71/ipp_manual/index.htm)

docs.it4i/anselm-cluster-documentation/software/intel-suite/intel-tbb.md

+Intel TBB
+=========
+
+Intel Threading Building Blocks
+-------------------------------
+Intel Threading Building Blocks (Intel TBB) is a library that supports scalable parallel programming using standard ISO C++ code. It does not require special languages or compilers.  To use the library, you specify tasks, not threads, and let the library map tasks onto threads in an efficient manner. The tasks are executed by a runtime scheduler and may
+be offloaded to [MIC accelerator](../intel-xeon-phi/).
+
+Intel TBB version 4.1 is available on Anselm
+
+```bash
+    $ module load tbb
+```
+
+The module sets up environment variables, required for linking and running tbb enabled applications.
+
+!!! Note "Note"
+	Link the tbb library, using -ltbb
+
+Examples
+--------
+Number of examples, demonstrating use of TBB and its built-in scheduler is available on Anselm, in the $TBB_EXAMPLES directory.
+
+```bash
+    $ module load intel
+    $ module load tbb
+    $ cp -a $TBB_EXAMPLES/common $TBB_EXAMPLES/parallel_reduce /tmp/
+    $ cd /tmp/parallel_reduce/primes
+    $ icc -O2 -DNDEBUG -o primes.x main.cpp primes.cpp -ltbb
+    $ ./primes.x
+```
+
+In this example, we compile, link and run the primes example, demonstrating use of parallel task-based reduce in computation of prime numbers.
+
+You will need the tbb module loaded to run the tbb enabled executable. This may be avoided, by compiling library search paths into the executable.
+
+```bash
+    $ icc -O2 -o primes.x main.cpp primes.cpp -Wl,-rpath=$LIBRARY_PATH -ltbb
+```
+
+Further reading
+---------------
+Read more on Intel website, <http://software.intel.com/sites/products/documentation/doclib/tbb_sa/help/index.htm>
+

docs.it4i/anselm-cluster-documentation/software/intel-suite/introduction.md

+Intel Parallel Studio
+=====================
+
+The Anselm cluster provides following elements of the Intel Parallel Studio XE
+
+|Intel Parallel Studio XE|
+|-------------------------------------------------|
+|Intel Compilers|
+|Intel Debugger|
+|Intel MKL Library|
+|Intel Integrated Performance Primitives Library|
+|Intel Threading Building Blocks Library|
+
+Intel compilers
+---------------
+The Intel compilers version 13.1.3 are available, via module intel. The compilers include the icc C and C++ compiler and the ifort fortran 77/90/95 compiler.
+
+```bash
+    $ module load intel
+    $ icc -v
+    $ ifort -v
+```
+
+Read more at the [Intel Compilers](intel-compilers/) page.
+
+Intel debugger
+--------------
+The intel debugger version 13.0 is available, via module intel. The debugger works for applications compiled with C and C++ compiler and the ifort fortran 77/90/95 compiler. The debugger provides java GUI environment. Use X display for running the GUI.
+
+```bash
+    $ module load intel
+    $ idb
+```
+
+Read more at the [Intel Debugger](intel-debugger/) page.
+
+Intel Math Kernel Library
+-------------------------
+Intel Math Kernel Library (Intel MKL) is a library of math kernel subroutines, extensively threaded and optimized for maximum performance. Intel MKL unites and provides these basic components: BLAS, LAPACK, ScaLapack, PARDISO, FFT, VML, VSL, Data fitting, Feast Eigensolver and many more.
+
+```bash
+    $ module load mkl
+```
+
+Read more at the [Intel MKL](intel-mkl/) page.
+
+Intel Integrated Performance Primitives
+---------------------------------------
+Intel Integrated Performance Primitives, version 7.1.1, compiled for AVX is available, via module ipp. The IPP is a library of highly optimized algorithmic building blocks for media and data applications. This includes signal, image and frame processing algorithms, such as FFT, FIR, Convolution, Optical Flow, Hough transform, Sum, MinMax and many more.
+
+```bash
+    $ module load ipp
+```
+
+Read more at the [Intel IPP](intel-integrated-performance-primitives/) page.
+
+Intel Threading Building Blocks
+-------------------------------
+Intel Threading Building Blocks (Intel TBB) is a library that supports scalable parallel programming using standard ISO C++ code. It does not require special languages or compilers. It is designed to promote scalable data parallel programming. Additionally, it fully supports nested parallelism, so you can build larger parallel components from smaller parallel components. To use the library, you specify tasks, not threads, and let the library map tasks onto threads in an efficient manner.
+
+```bash
+    $ module load tbb
+```
+
+Read more at the [Intel TBB](intel-tbb/) page.