docs.it4i/anselm-cluster-documentation/software/numerical-languages/octave.md

+Octave
+======
+
+Introduction
+------------
+GNU Octave is a high-level interpreted language, primarily intended for numerical computations. It provides capabilities for the numerical solution of linear and nonlinear problems, and for performing other numerical experiments. It also provides extensive graphics capabilities for data visualization and manipulation. Octave is normally used through its interactive command line interface, but it can also be used to write non-interactive programs. The Octave language is quite similar to Matlab so that most programs are easily portable. Read more on <http://www.gnu.org/software/octave/>
+
+Two versions of octave are available on Anselm, via module
+
+|Version|module|
+|---|---|
+|Octave 3.8.2, compiled with GCC and Multithreaded MKL|Octave/3.8.2-gimkl-2.11.5|
+|Octave 4.0.1, compiled with GCC and Multithreaded MKL|Octave/4.0.1-gimkl-2.11.5|
+|Octave 4.0.0, compiled with >GCC and OpenBLAS|Octave/4.0.0-foss-2015g|
+
+ Modules and execution
+----------------------
+
+    $ module load Octave
+
+The octave on Anselm is linked to highly optimized MKL mathematical library. This provides threaded parallelization to many octave kernels, notably the linear algebra subroutines. Octave runs these heavy calculation kernels without any penalty. By default, octave would parallelize to 16 threads. You may control the threads by setting the OMP_NUM_THREADS environment variable.
+
+To run octave interactively, log in with ssh -X parameter for X11 forwarding. Run octave:
+
+```bash
+    $ octave
+```
+
+To run octave in batch mode, write an octave script, then write a bash jobscript and execute via the qsub command. By default, octave will use 16 threads when running MKL kernels.
+
+```bash
+    #!/bin/bash
+
+    # change to local scratch directory
+    cd /lscratch/$PBS_JOBID || exit
+
+    # copy input file to scratch
+    cp $PBS_O_WORKDIR/octcode.m .
+
+    # load octave module
+    module load octave
+
+    # execute the calculation
+    octave -q --eval octcode > output.out
+
+    # copy output file to home
+    cp output.out $PBS_O_WORKDIR/.
+
+    #exit
+    exit
+```
+
+This script may be submitted directly to the PBS workload manager via the qsub command.  The inputs are in octcode.m file, outputs in output.out file. See the single node jobscript example in the [Job execution section](http://support.it4i.cz/docs/anselm-cluster-documentation/resource-allocation-and-job-execution).
+
+The octave c compiler mkoctfile calls the GNU gcc 4.8.1 for compiling native c code. This is very useful for running native c subroutines in octave environment.
+
+```bash
+    $ mkoctfile -v
+```
+
+Octave may use MPI for interprocess communication This functionality is currently not supported on Anselm cluster. In case you require the octave interface to MPI, please contact [Anselm support](https://support.it4i.cz/rt/).
+
+Xeon Phi Support
+----------------
+Octave may take advantage of the Xeon Phi accelerators. This will only work on the  [Intel Xeon Phi](../intel-xeon-phi/) [accelerated nodes](../../compute-nodes/).
+
+### Automatic offload support
+
+Octave can accelerate BLAS type operations (in particular the Matrix Matrix multiplications] on the Xeon Phi accelerator, via [Automatic Offload using the MKL library](../intel-xeon-phi/#section-3)
+
+Example
+
+```bash
+    $ export OFFLOAD_REPORT=2
+    $ export MKL_MIC_ENABLE=1
+    $ module load octave
+    $ octave -q
+    octave:1> A=rand(10000); B=rand(10000);
+    octave:2> tic; C=A*B; toc
+    [MKL] [MIC --] [AO Function]    DGEMM
+    [MKL] [MIC --] [AO DGEMM Workdivision]    0.32 0.68
+    [MKL] [MIC 00] [AO DGEMM CPU Time]    2.896003 seconds
+    [MKL] [MIC 00] [AO DGEMM MIC Time]    1.967384 seconds
+    [MKL] [MIC 00] [AO DGEMM CPU->MIC Data]    1347200000 bytes
+    [MKL] [MIC 00] [AO DGEMM MIC->CPU Data]    2188800000 bytes
+    Elapsed time is 2.93701 seconds.
+```
+
+In this example, the calculation was automatically divided among the CPU cores and the Xeon Phi MIC accelerator, reducing the total runtime from 6.3 secs down to 2.9 secs.
+
+### Native support
+
+A version of [native](../intel-xeon-phi/#section-4) Octave is compiled for Xeon Phi accelerators. Some limitations apply for this version:
+
+-   Only command line support. GUI, graph plotting etc. is not supported.
+-   Command history in interactive mode is not supported.
+
+Octave is linked with parallel Intel MKL, so it best suited for batch processing of tasks that utilize BLAS, LAPACK and FFT operations. By default, number of threads is set to 120, you can control this with > OMP_NUM_THREADS environment
+variable.
+
+!!! Note "Note"
+	Calculations that do not employ parallelism (either by using parallel MKL eg. via matrix operations,  fork() function, [parallel package](http://octave.sourceforge.net/parallel/) or other mechanism) will actually run slower than on host CPU.
+
+To use Octave on a node with Xeon Phi:
+
+```bash
+    $ ssh mic0                                               # login to the MIC card
+    $ source /apps/tools/octave/3.8.2-mic/bin/octave-env.sh # set up environment variables
+    $ octave -q /apps/tools/octave/3.8.2-mic/example/test0.m # run an example
+```

docs.it4i/anselm-cluster-documentation/software/numerical-languages/r.md

+R
+===
+
+Introduction
+------------
+The R is a language and environment for statistical computing and graphics.  R provides a wide variety of statistical (linear and nonlinear modelling, classical statistical tests, time-series analysis, classification, clustering, ...) and graphical techniques, and is highly extensible.
+
+One of R's strengths is the ease with which well-designed publication-quality plots can be produced, including mathematical symbols and formulae where needed. Great care has been taken over the defaults for the minor design choices in graphics, but the user retains full control.
+
+Another convenience is the ease with which the C code or third party libraries may be integrated within R.
+
+Extensive support for parallel computing is available within R.
+
+Read more on <http://www.r-project.org/>, <http://cran.r-project.org/doc/manuals/r-release/R-lang.html>
+
+Modules
+-------
+The R version 3.0.1 is available on Anselm, along with GUI interface Rstudio
+
+|Application|Version|module|
+|---|---|
+| **R**|R 3.0.1|R|
+|**Rstudio**|Rstudio 0.97|Rstudio|
+
+```bash
+    $ module load R
+```
+
+Execution
+---------
+The R on Anselm is linked to highly optimized MKL mathematical library. This provides threaded parallelization to many R kernels, notably the linear algebra subroutines. The R runs these heavy calculation kernels without any penalty. By default, the R would parallelize to 16 threads. You may control the threads by setting the OMP_NUM_THREADS environment variable.
+
+### Interactive execution
+
+To run R interactively, using Rstudio GUI, log in with ssh -X parameter for X11 forwarding. Run rstudio:
+
+```bash
+    $ module load Rstudio
+    $ rstudio
+```
+
+### Batch execution
+
+To run R in batch mode, write an R script, then write a bash jobscript and execute via the qsub command. By default, R will use 16 threads when running MKL kernels.
+
+Example jobscript:
+
+```bash
+    #!/bin/bash
+
+    # change to local scratch directory
+    cd /lscratch/$PBS_JOBID || exit
+
+    # copy input file to scratch
+    cp $PBS_O_WORKDIR/rscript.R .
+
+    # load R module
+    module load R
+
+    # execute the calculation
+    R CMD BATCH rscript.R routput.out
+
+    # copy output file to home
+    cp routput.out $PBS_O_WORKDIR/.
+
+    #exit
+    exit
+```
+
+This script may be submitted directly to the PBS workload manager via the qsub command.  The inputs are in rscript.R file, outputs in routput.out file. See the single node jobscript example in the [Job execution section](../../resource-allocation-and-job-execution/job-submission-and-execution/).
+
+Parallel R
+----------
+Parallel execution of R may be achieved in many ways. One approach is the implied parallelization due to linked libraries or specially enabled functions, as [described above](r/#interactive-execution). In the following sections, we focus on explicit parallelization, where  parallel constructs are directly stated within the R script.
+
+Package parallel
+--------------------
+The package parallel provides support for parallel computation, including by forking (taken from package multicore), by sockets (taken from package snow) and random-number generation.
+
+The package is activated this way:
+
+```bash
+    $ R
+    > library(parallel)
+```
+
+More information and examples may be obtained directly by reading the documentation available in R
+
+```bash
+    > ?parallel
+    > library(help = "parallel")
+    > vignette("parallel")
+```
+
+Download the package [parallell](package-parallel-vignette.pdf) vignette.
+
+The forking is the most simple to use. Forking family of functions provide parallelized, drop in replacement for the serial apply() family of functions.
+
+!!! Note "Note"
+	Forking via package parallel provides functionality similar to OpenMP construct
+
+	omp parallel for
+
+	Only cores of single node can be utilized this way!
+
+Forking example:
+
+```bash
+    library(parallel)
+
+    #integrand function
+    f <- function(i,h) {
+    x <- h*(i-0.5)
+    return (4/(1 + x*x))
+    }
+
+    #initialize
+    size <- detectCores()
+
+    while (TRUE)
+    {
+      #read number of intervals
+      cat("Enter the number of intervals: (0 quits) ")
+      fp<-file("stdin"); n<-scan(fp,nmax=1); close(fp)
+
+      if(n<=0) break
+
+      #run the calculation
+      n <- max(n,size)
+      h <-   1.0/n
+
+      i <- seq(1,n);
+      pi3 <- h*sum(simplify2array(mclapply(i,f,h,mc.cores=size)));
+
+      #print results
+      cat(sprintf("Value of PI %16.14f, diff= %16.14fn",pi3,pi3-pi))
+    }
+```
+
+The above example is the classic parallel example for calculating the number π. Note the **detectCores()** and **mclapply()** functions. Execute the example as:
+
+```bash
+    $ R --slave --no-save --no-restore -f pi3p.R
+```
+
+Every evaluation of the integrad function runs in parallel on different process.
+
+Package Rmpi
+------------
+!!! Note "Note"
+	package Rmpi provides an interface (wrapper) to MPI APIs.
+
+It also provides interactive R slave environment. On Anselm, Rmpi provides interface to the [OpenMPI](../mpi-1/Running_OpenMPI/).
+
+Read more on Rmpi at <http://cran.r-project.org/web/packages/Rmpi/>, reference manual is available at <http://cran.r-project.org/web/packages/Rmpi/Rmpi.pdf>
+
+When using package Rmpi, both openmpi and R modules must be loaded
+
+```bash
+    $ module load openmpi
+    $ module load R
+```
+Rmpi may be used in three basic ways. The static approach is identical to executing any other MPI programm. In addition, there is Rslaves dynamic MPI approach and the mpi.apply approach. In the following section, we will use the number π integration example, to illustrate all these concepts.
+
+### static Rmpi
+
+Static Rmpi programs are executed via mpiexec, as any other MPI programs. Number of processes is static - given at the launch time.
+
+Static Rmpi example:
+
+```cpp
+    library(Rmpi)
+
+    #integrand function
+    f <- function(i,h) {
+    x <- h*(i-0.5)
+    return (4/(1 + x*x))
+    }
+
+    #initialize
+    invisible(mpi.comm.dup(0,1))
+    rank <- mpi.comm.rank()
+    size <- mpi.comm.size()
+    n<-0
+
+    while (TRUE)
+    {
+      #read number of intervals
+      if (rank==0) {
+       cat("Enter the number of intervals: (0 quits) ")
+       fp<-file("stdin"); n<-scan(fp,nmax=1); close(fp)
+      }
+
+      #broadcat the intervals
+      n <- mpi.bcast(as.integer(n),type=1)
+
+      if(n<=0) break
+
+      #run the calculation
+      n <- max(n,size)
+      h <-   1.0/n
+
+      i <- seq(rank+1,n,size);
+      mypi <- h*sum(sapply(i,f,h));
+
+      pi3 <- mpi.reduce(mypi)
+
+      #print results
+      if (rank==0) cat(sprintf("Value of PI %16.14f, diff= %16.14fn",pi3,pi3-pi))
+    }
+
+    mpi.quit()
+```
+
+The above is the static MPI example for calculating the number π. Note the **library(Rmpi)** and **mpi.comm.dup()** function calls.
+
+Execute the example as:
+
+```bash
+    $ mpiexec R --slave --no-save --no-restore -f pi3.R
+```
+
+### dynamic Rmpi
+
+Dynamic Rmpi programs are executed by calling the R directly. openmpi module must be still loaded. The R slave processes will be spawned by a function call within the Rmpi program.
+
+Dynamic Rmpi example:
+
+```cpp
+    #integrand function
+    f <- function(i,h) {
+    x <- h*(i-0.5)
+    return (4/(1 + x*x))
+    }
+
+    #the worker function
+    workerpi <- function()
+    {
+    #initialize
+    rank <- mpi.comm.rank()
+    size <- mpi.comm.size()
+    n<-0
+
+    while (TRUE)
+    {
+      #read number of intervals
+      if (rank==0) {
+       cat("Enter the number of intervals: (0 quits) ")
+       fp<-file("stdin"); n<-scan(fp,nmax=1); close(fp)
+      }
+
+      #broadcat the intervals
+      n <- mpi.bcast(as.integer(n),type=1)
+
+      if(n<=0) break
+
+      #run the calculation
+      n <- max(n,size)
+      h <-   1.0/n
+
+      i <- seq(rank+1,n,size);
+      mypi <- h*sum(sapply(i,f,h));
+
+      pi3 <- mpi.reduce(mypi)
+
+      #print results
+      if (rank==0) cat(sprintf("Value of PI %16.14f, diff= %16.14fn",pi3,pi3-pi))
+    }
+    }
+
+    #main
+    library(Rmpi)
+
+    cat("Enter the number of slaves: ")
+    fp<-file("stdin"); ns<-scan(fp,nmax=1); close(fp)
+
+    mpi.spawn.Rslaves(nslaves=ns)
+    mpi.bcast.Robj2slave(f)
+    mpi.bcast.Robj2slave(workerpi)
+
+    mpi.bcast.cmd(workerpi())
+    workerpi()
+
+    mpi.quit()
+```
+
+The above example is the dynamic MPI example for calculating the number π. Both master and slave processes carry out the calculation. Note the mpi.spawn.Rslaves(), mpi.bcast.Robj2slave()** and the mpi.bcast.cmd()** function calls.
+
+Execute the example as:
+
+```bash
+    $ R --slave --no-save --no-restore -f pi3Rslaves.R
+```
+
+### mpi.apply Rmpi
+
+mpi.apply is a specific way of executing Dynamic Rmpi programs.
+
+!!! Note "Note"
+	mpi.apply() family of functions provide MPI parallelized, drop in replacement for the serial apply() family of functions.
+
+Execution is identical to other dynamic Rmpi programs.
+
+mpi.apply Rmpi example:
+
+```bash
+    #integrand function
+    f <- function(i,h) {
+    x <- h*(i-0.5)
+    return (4/(1 + x*x))
+    }
+
+    #the worker function
+    workerpi <- function(rank,size,n)
+    {
+      #run the calculation
+      n <- max(n,size)
+      h <- 1.0/n
+
+      i <- seq(rank,n,size);
+      mypi <- h*sum(sapply(i,f,h));
+
+      return(mypi)
+    }
+
+    #main
+    library(Rmpi)
+
+    cat("Enter the number of slaves: ")
+    fp<-file("stdin"); ns<-scan(fp,nmax=1); close(fp)
+
+    mpi.spawn.Rslaves(nslaves=ns)
+    mpi.bcast.Robj2slave(f)
+    mpi.bcast.Robj2slave(workerpi)
+
+    while (TRUE)
+    {
+      #read number of intervals
+      cat("Enter the number of intervals: (0 quits) ")
+      fp<-file("stdin"); n<-scan(fp,nmax=1); close(fp)
+      if(n<=0) break
+
+      #run workerpi
+      i=seq(1,2*ns)
+      pi3=sum(mpi.parSapply(i,workerpi,2*ns,n))
+
+      #print results
+      cat(sprintf("Value of PI %16.14f, diff= %16.14fn",pi3,pi3-pi))
+    }
+
+    mpi.quit()
+```
+
+The above is the mpi.apply MPI example for calculating the number π. Only the slave processes carry out the calculation. Note the **mpi.parSapply()**, function call. The package parallel [example](r/#package-parallel)[above](r/#package-parallel) may be trivially adapted (for much better performance) to this structure using the mclapply() in place of mpi.parSapply().
+
+Execute the example as:
+
+```bash
+    $ R --slave --no-save --no-restore -f pi3parSapply.R
+```
+
+Combining parallel and Rmpi
+---------------------------
+
+Currently, the two packages can not be combined for hybrid calculations.
+
+Parallel execution
+------------------
+
+The R parallel jobs are executed via the PBS queue system exactly as any other parallel jobs. User must create an appropriate jobscript and submit via the **qsub**
+
+Example jobscript for [static Rmpi](r/#static-rmpi) parallel R execution, running 1 process per core:
+
+```bash
+    #!/bin/bash
+    #PBS -q qprod
+    #PBS -N Rjob
+    #PBS -l select=100:ncpus=16:mpiprocs=16:ompthreads=1
+
+    # change to  scratch directory
+    SCRDIR=/scratch/$USER/myjob
+    cd $SCRDIR || exit
+
+    # copy input file to scratch
+    cp $PBS_O_WORKDIR/rscript.R .
+
+    # load R and openmpi module
+    module load R
+    module load openmpi
+
+    # execute the calculation
+    mpiexec -bycore -bind-to-core R --slave --no-save --no-restore -f rscript.R
+
+    # copy output file to home
+    cp routput.out $PBS_O_WORKDIR/.
+
+    #exit
+    exit
+```
+
+For more information about jobscripts and MPI execution refer to the [Job submission](../../resource-allocation-and-job-execution/job-submission-and-execution/) and general [MPI](../mpi/mpi/) sections.

docs.it4i/anselm-cluster-documentation/software/numerical-libraries/fftw.md

+FFTW
+====
+
+The discrete Fourier transform in one or more dimensions, MPI parallel
+
+FFTW is a C subroutine library for computing the discrete Fourier transform  in one or more dimensions, of arbitrary input size, and of both real and complex data (as well as of even/odd data, i.e. the discrete cosine/sine transforms or DCT/DST). The FFTW library allows for MPI parallel, in-place discrete Fourier transform, with data distributed over number of nodes.
+
+Two versions, **3.3.3** and **2.1.5** of FFTW are available on Anselm, each compiled for **Intel MPI** and **OpenMPI** using **intel** and **gnu** compilers. These are available via modules:
+
+ |Version |Parallelization |module |linker options |
+ | --- | --- |
+ |FFTW3 gcc3.3.3 |pthread, OpenMP |fftw3/3.3.3-gcc |-lfftw3, -lfftw3_threads-lfftw3_omp |
+ |FFTW3 icc3.3.3 |pthread, OpenMP |fftw3 |-lfftw3, -lfftw3_threads-lfftw3_omp |
+ |FFTW2 gcc2.1.5 |pthread |fftw2/2.1.5-gcc |-lfftw, -lfftw_threads |
+ |FFTW2 icc2.1.5 |pthread |fftw2 |-lfftw, -lfftw_threads |
+ |FFTW3 gcc3.3.3 |OpenMPI |fftw-mpi3/3.3.3-gcc |-lfftw3_mpi |
+ |FFTW3 icc3.3.3 |Intel MPI |fftw3-mpi |-lfftw3_mpi |
+ |FFTW2 gcc2.1.5 |OpenMPI |fftw2-mpi/2.1.5-gcc |-lfftw_mpi |
+ |FFTW2 gcc2.1.5 |IntelMPI |fftw2-mpi/2.1.5-gcc |-lfftw_mpi |
+
+```bash
+    $ module load fftw3
+```
+
+The module sets up environment variables, required for linking and running fftw enabled applications. Make sure that the choice of fftw module is consistent with your choice of MPI library. Mixing MPI of different implementations may have unpredictable results.
+
+Example
+-------
+
+```cpp
+    #include <fftw3-mpi.h>
+    int main(int argc, char **argv)
+    {
+        const ptrdiff_t N0 = 100, N1 = 1000;
+        fftw_plan plan;
+        fftw_complex *data;
+        ptrdiff_t alloc_local, local_n0, local_0_start, i, j;
+
+        MPI_Init(&argc, &argv);
+        fftw_mpi_init();
+
+        /* get local data size and allocate */
+        alloc_local = fftw_mpi_local_size_2d(N0, N1, MPI_COMM_WORLD,
+                                             &local_n0, &local_0_start);
+        data = fftw_alloc_complex(alloc_local);
+
+        /* create plan for in-place forward DFT */
+        plan = fftw_mpi_plan_dft_2d(N0, N1, data, data, MPI_COMM_WORLD,
+                                    FFTW_FORWARD, FFTW_ESTIMATE);
+
+        /* initialize data  */
+        for (i = 0; i < local_n0; ++i) for (j = 0; j < N1; ++j)
+        {   data[i*N1 + j][0] = i;
+            data[i*N1 + j][1] = j; }
+
+        /* compute transforms, in-place, as many times as desired */
+        fftw_execute(plan);
+
+        fftw_destroy_plan(plan);
+
+        MPI_Finalize();
+    }
+```
+
+Load modules and compile:
+
+```bash
+    $ module load impi intel
+    $ module load fftw3-mpi
+
+    $ mpicc testfftw3mpi.c -o testfftw3mpi.x -Wl,-rpath=$LIBRARY_PATH -lfftw3_mpi
+```
+
+Run the example as [Intel MPI program](../mpi/running-mpich2/).
+
+Read more on FFTW usage on the [FFTW website.](http://www.fftw.org/fftw3_doc/)

docs.it4i/anselm-cluster-documentation/software/numerical-libraries/gsl.md

+GSL
+===
+
+The GNU Scientific Library. Provides a wide range of mathematical routines.
+
+Introduction
+------------
+The GNU Scientific Library (GSL) provides a wide range of mathematical routines such as random number generators, special functions and least-squares fitting. There are over 1000 functions in total. The routines have been written from scratch in C, and present a modern Applications Programming Interface (API) for C programmers, allowing wrappers to be written for very high level languages.
+
+The library covers a wide range of topics in numerical computing. Routines are available for the following areas:
+
+
+                           Complex Numbers          	Roots of Polynomials
+
+                           Special Functions        	Vectors and Matrices
+
+                           Permutations            	Combinations
+
+                           Sorting                  	BLAS Support
+
+                           Linear Algebra           	CBLAS Library
+
+                           Fast Fourier Transforms  	Eigensystems
+
+                           Random Numbers           	Quadrature
+
+                           Random Distributions     	Quasi-Random Sequences
+
+                           Histograms               	Statistics
+
+                           Monte Carlo Integration  	N-Tuples
+
+                           Differential Equations   	Simulated Annealing
+
+                           Numerical Differentiation    Interpolation
+
+                           Series Acceleration      	Chebyshev Approximations
+
+                           Root-Finding             	Discrete Hankel Transforms
+
+                           Least-Squares Fitting    	Minimization
+
+                           IEEE Floating-Point      	Physical Constants
+
+                           Basis Splines            	Wavelets
+
+Modules
+-------
+
+The GSL 1.16 is available on Anselm, compiled for GNU and Intel compiler. These variants are available via modules:
+
+|Module|Compiler|
+|---|---|
+|  gsl/1.16-gcc|gcc 4.8.6|
+|gsl/1.16-icc(default)|icc|
+
+```bash
+     $ module load gsl
+```
+
+The module sets up environment variables, required for linking and running GSL enabled applications. This particular command loads the default module, which is gsl/1.16-icc
+
+Linking
+-------
+Load an appropriate gsl module. Link using **-lgsl** switch to link your code against GSL. The GSL depends on cblas API to BLAS library, which must be supplied for linking. The BLAS may be provided, for example from the MKL library, as well as from the BLAS GSL library (-lgslcblas). Using the MKL is recommended.
+
+### Compiling and linking with Intel compilers
+
+```bash
+    $ module load intel
+    $ module load gsl
+    $ icc myprog.c -o myprog.x -Wl,-rpath=$LIBRARY_PATH -mkl -lgsl
+```
+
+### Compiling and linking with GNU compilers
+
+```bash
+    $ module load gcc
+    $ module load mkl
+    $ module load gsl/1.16-gcc
+    $ gcc myprog.c -o myprog.x -Wl,-rpath=$LIBRARY_PATH -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_core -lgomp -lgsl
+```
+
+Example
+-------
+Following is an example of discrete wavelet transform implemented by GSL:
+
+```cpp
+    #include <stdio.h>
+    #include <math.h>
+    #include <gsl/gsl_sort.h>
+    #include <gsl/gsl_wavelet.h>
+
+    int
+    main (int argc, char **argv)
+    {
+      int i, n = 256, nc = 20;
+      double *data = malloc (n * sizeof (double));
+      double *abscoeff = malloc (n * sizeof (double));
+      size_t *p = malloc (n * sizeof (size_t));
+
+      gsl_wavelet *w;
+      gsl_wavelet_workspace *work;
+
+      w = gsl_wavelet_alloc (gsl_wavelet_daubechies, 4);
+      work = gsl_wavelet_workspace_alloc (n);
+
+      for (i=0; i<n; i++)
+      data[i] = sin (3.141592654*(double)i/256.0);
+
+      gsl_wavelet_transform_forward (w, data, 1, n, work);
+
+      for (i = 0; i < n; i++)
+        {
+          abscoeff[i] = fabs (data[i]);
+        }
+
+      gsl_sort_index (p, abscoeff, 1, n);
+
+      for (i = 0; (i + nc) < n; i++)
+        data[p[i]] = 0;
+
+      gsl_wavelet_transform_inverse (w, data, 1, n, work);
+
+      for (i = 0; i < n; i++)
+        {
+          printf ("%gn", data[i]);
+        }
+
+      gsl_wavelet_free (w);
+      gsl_wavelet_workspace_free (work);
+
+      free (data);
+      free (abscoeff);
+      free (p);
+      return 0;
+    }
+```
+
+Load modules and compile:
+
+```bash
+    $ module load intel gsl
+    icc dwt.c -o dwt.x -Wl,-rpath=$LIBRARY_PATH -mkl -lgsl
+```
+
+In this example, we compile the dwt.c code using the Intel compiler and link it to the MKL and GSL library, note the -mkl and -lgsl options. The library search path is compiled in, so that no modules are necessary to run the code.

docs.it4i/anselm-cluster-documentation/software/numerical-libraries/hdf5.md

+HDF5
+====
+
+Hierarchical Data Format library. Serial and MPI parallel version.
+
+[HDF5 (Hierarchical Data Format)](http://www.hdfgroup.org/HDF5/) is a general purpose library and file format for storing scientific data. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic objects, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids. You can also mix and match them in HDF5 files according to your needs.
+
+Versions **1.8.11** and **1.8.13** of HDF5 library are available on Anselm, compiled for **Intel MPI** and **OpenMPI** using **intel** and **gnu** compilers. These are available via modules:
+
+ |Version |Parallelization |module |C linker options|C++ linker options|Fortran linker options |
+ | --- | --- |
+ |HDF5 icc serial |pthread |hdf5/1.8.11 |$HDF5_INC $HDF5_SHLIB |$HDF5_INC $HDF5_CPP_LIB |$HDF5_INC $HDF5_F90_LIB |
+ |HDF5 icc parallel MPI |pthread, IntelMPI |hdf5-parallel/1.8.11 |$HDF5_INC $HDF5_SHLIB |Not supported |$HDF5_INC $HDF5_F90_LIB |
+ |HDF5 icc serial |pthread |hdf5/1.8.13 |$HDF5_INC $HDF5_SHLIB |$HDF5_INC $HDF5_CPP_LIB |$HDF5_INC $HDF5_F90_LIB |
+ |HDF5 icc parallel MPI |pthread, IntelMPI |hdf5-parallel/1.8.13 |$HDF5_INC $HDF5_SHLIB |Not supported |$HDF5_INC $HDF5_F90_LIB |
+ |HDF5 gcc parallel MPI |pthread, OpenMPI 1.6.5, gcc 4.8.1 |hdf5-parallel/1.8.11-gcc |$HDF5_INC $HDF5_SHLIB |Not supported |$HDF5_INC $HDF5_F90_LIB |
+ |HDF5 gcc parallel MPI|pthread, OpenMPI 1.6.5, gcc 4.8.1 |hdf5-parallel/1.8.13-gcc |$HDF5_INC $HDF5_SHLIB |Not supported |$HDF5_INC $HDF5_F90_LIB |
+ |HDF5 gcc parallel MPI |pthread, OpenMPI 1.8.1, gcc 4.9.0 |hdf5-parallel/1.8.13-gcc49 |$HDF5_INC $HDF5_SHLIB |Not supported |$HDF5_INC $HDF5_F90_LIB |
+
+```bash
+    $ module load hdf5-parallel
+```
+
+The module sets up environment variables, required for linking and running HDF5 enabled applications. Make sure that the choice of HDF5 module is consistent with your choice of MPI library. Mixing MPI of different implementations may have unpredictable results.
+
+!!! Note "Note"
+	Be aware, that GCC version of **HDF5 1.8.11** has serious performance issues, since it's compiled with -O0 optimization flag. This version is provided only for testing of code compiled only by GCC and IS NOT recommended for production computations. For more informations, please see: <http://www.hdfgroup.org/ftp/HDF5/prev-releases/ReleaseFiles/release5-1811>
+
+	All GCC versions of **HDF5 1.8.13** are not affected by the bug, are compiled with -O3 optimizations and are recommended for production computations.
+
+Example
+-------
+
+```cpp
+    #include "hdf5.h"
+    #define FILE "dset.h5"
+
+    int main() {
+
+       hid_t       file_id, dataset_id, dataspace_id;  /* identifiers */
+       hsize_t     dims[2];
+       herr_t      status;
+       int         i, j, dset_data[4][6];
+
+       /* Create a new file using default properties. */
+       file_id = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
+
+       /* Create the data space for the dataset. */
+       dims[0] = 4;
+       dims[1] = 6;
+       dataspace_id = H5Screate_simple(2, dims, NULL);
+
+       /* Initialize the dataset. */
+       for (i = 0; i < 4; i++)
+          for (j = 0; j < 6; j++)
+             dset_data[i][j] = i * 6 + j + 1;
+
+       /* Create the dataset. */
+       dataset_id = H5Dcreate2(file_id, "/dset", H5T_STD_I32BE, dataspace_id,
+                              H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
+
+       /* Write the dataset. */
+       status = H5Dwrite(dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT,
+                         dset_data);
+
+       status = H5Dread(dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT,
+                        dset_data);
+
+       /* End access to the dataset and release resources used by it. */
+       status = H5Dclose(dataset_id);
+
+       /* Terminate access to the data space. */
+       status = H5Sclose(dataspace_id);
+
+       /* Close the file. */
+       status = H5Fclose(file_id);
+    }
+```
+
+Load modules and compile:
+
+```bash
+    $ module load intel impi
+    $ module load hdf5-parallel
+
+    $ mpicc hdf5test.c -o hdf5test.x -Wl,-rpath=$LIBRARY_PATH $HDF5_INC $HDF5_SHLIB
+```
+
+Run the example as [Intel MPI program](../anselm-cluster-documentation/software/mpi/running-mpich2/).
+
+For further informations, please see the website: <http://www.hdfgroup.org/HDF5/>

docs.it4i/anselm-cluster-documentation/software/numerical-libraries/intel-numerical-libraries.md

+Intel numerical libraries
+=========================
+
+Intel libraries for high performance in numerical computing
+
+Intel Math Kernel Library
+-------------------------
+Intel Math Kernel Library (Intel MKL) is a library of math kernel subroutines, extensively threaded and optimized for maximum performance. Intel MKL unites and provides these basic components: BLAS, LAPACK, ScaLapack, PARDISO, FFT, VML, VSL, Data fitting, Feast Eigensolver and many more.
+
+```bash
+    $ module load mkl
+```
+
+Read more at the [Intel MKL](../intel-suite/intel-mkl/) page.
+
+Intel Integrated Performance Primitives
+---------------------------------------
+Intel Integrated Performance Primitives, version 7.1.1, compiled for AVX is available, via module ipp. The IPP is a library of highly optimized algorithmic building blocks for media and data applications. This includes signal, image and frame processing algorithms, such as FFT, FIR, Convolution, Optical Flow, Hough transform, Sum, MinMax and many more.
+
+```bash
+    $ module load ipp
+```
+
+Read more at the [Intel IPP](../intel-suite/intel-integrated-performance-primitives/) page.
+
+Intel Threading Building Blocks
+-------------------------------
+Intel Threading Building Blocks (Intel TBB) is a library that supports scalable parallel programming using standard ISO C++ code. It does not require special languages or compilers. It is designed to promote scalable data parallel programming. Additionally, it fully supports nested parallelism, so you can build larger parallel components from smaller parallel components. To use the library, you specify tasks, not threads, and let the library map tasks onto threads in an efficient manner.
+
+```bash
+    $ module load tbb
+```
+
+Read more at the [Intel TBB](../intel-suite/intel-tbb/) page.

docs.it4i/anselm-cluster-documentation/software/numerical-libraries/magma-for-intel-xeon-phi.md

+MAGMA for Intel Xeon Phi
+========================
+
+Next generation dense algebra library for heterogeneous systems with accelerators
+
+### Compiling and linking with MAGMA
+
+To be able to compile and link code with MAGMA library user has to load following module:
+
+```bash
+    $ module load magma/1.3.0-mic
+```
+
+To make compilation more user friendly module also sets these two environment variables:
+
+!!! Note "Note"
+	MAGMA_INC - contains paths to the MAGMA header files (to be used for compilation step)
+
+!!! Note "Note"
+	MAGMA_LIBS - contains paths to MAGMA libraries (to be used for linking step).
+
+Compilation example:
+
+```bash
+    $ icc -mkl -O3 -DHAVE_MIC -DADD_ -Wall $MAGMA_INC -c testing_dgetrf_mic.cpp -o testing_dgetrf_mic.o
+
+    $ icc -mkl -O3 -DHAVE_MIC -DADD_ -Wall -fPIC -Xlinker -zmuldefs -Wall -DNOCHANGE -DHOST  testing_dgetrf_mic.o  -o testing_dgetrf_mic $MAGMA_LIBS
+```
+
+### Running MAGMA code
+
+MAGMA implementation for Intel MIC requires a MAGMA server running on accelerator prior to executing the user application. The server can be started and stopped using following scripts:
+
+!!! Note "Note"
+	To start MAGMA server use:
+	**$MAGMAROOT/start_magma_server**
+
+!!! Note "Note"
+	To stop the server use:
+	**$MAGMAROOT/stop_magma_server**
+
+!!! Note "Note"
+	For deeper understanding how the MAGMA server is started, see the following script:
+	**$MAGMAROOT/launch_anselm_from_mic.sh**
+
+To test if the MAGMA server runs properly we can run one of examples that are part of the MAGMA installation:
+
+```bash
+    [user@cn204 ~]$ $MAGMAROOT/testing/testing_dgetrf_mic
+
+    [user@cn204 ~]$ export OMP_NUM_THREADS=16
+
+    [lriha@cn204 ~]$ $MAGMAROOT/testing/testing_dgetrf_mic
+    Usage: /apps/libs/magma-mic/magmamic-1.3.0/testing/testing_dgetrf_mic [options] [-h|--help]
+
+      M     N     CPU GFlop/s (sec)   MAGMA GFlop/s (sec)   ||PA-LU||/(||A||*N)
+    =========================================================================
+     1088  1088     ---   (  ---  )     13.93 (   0.06)     ---
+     2112  2112     ---   (  ---  )     77.85 (   0.08)     ---
+     3136  3136     ---   (  ---  )    183.21 (   0.11)     ---
+     4160  4160     ---   (  ---  )    227.52 (   0.21)     ---
+     5184  5184     ---   (  ---  )    258.61 (   0.36)     ---
+     6208  6208     ---   (  ---  )    333.12 (   0.48)     ---
+     7232  7232     ---   (  ---  )    416.52 (   0.61)     ---
+     8256  8256     ---   (  ---  )    446.97 (   0.84)     ---
+     9280  9280     ---   (  ---  )    461.15 (   1.16)     ---
+    10304 10304     ---   (  ---  )    500.70 (   1.46)     ---
+```
+
+!!! Note "Note"
+	Please note: MAGMA contains several benchmarks and examples that can be found in:
+	**$MAGMAROOT/testing/**
+
+!!! Note "Note"
+	MAGMA relies on the performance of all CPU cores as well as on the performance of the accelerator. Therefore on Anselm number of CPU OpenMP threads has to be set to 16:
+	**export OMP_NUM_THREADS=16**
+
+
+See more details at [MAGMA home page](http://icl.cs.utk.edu/magma/).
+
+References
+----------
+[1] MAGMA MIC: Linear Algebra Library for Intel Xeon Phi Coprocessors, Jack Dongarra et. al, [http://icl.utk.edu/projectsfiles/magma/pubs/24-MAGMA_MIC_03.pdf](http://icl.utk.edu/projectsfiles/magma/pubs/24-MAGMA_MIC_03.pdf)

docs.it4i/anselm-cluster-documentation/software/numerical-libraries/petsc.md

+PETSc
+=====
+
+PETSc is a suite of building blocks for the scalable solution of scientific and engineering applications modelled by partial differential equations. It supports MPI, shared memory, and GPUs through CUDA or OpenCL, as well as hybrid MPI-shared memory or MPI-GPU parallelism.
+
+Introduction
+------------
+PETSc (Portable, Extensible Toolkit for Scientific Computation) is a suite of building blocks (data structures and routines) for the scalable solution of scientific and engineering applications modelled by partial differential equations. It allows thinking in terms of high-level objects (matrices) instead of low-level objects (raw arrays). Written in C language but can also be called from FORTRAN, C++, Python and Java codes. It supports MPI, shared memory, and GPUs through CUDA or OpenCL, as well as hybrid MPI-shared memory or MPI-GPU parallelism.
+
+Resources
+---------
+-   [project webpage](http://www.mcs.anl.gov/petsc/)
+-   [documentation](http://www.mcs.anl.gov/petsc/documentation/)
+    -   [PETSc Users  Manual (PDF)](http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf)
+    -   [index of all manual pages](http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/singleindex.html)
+-   PRACE Video Tutorial [part1](http://www.youtube.com/watch?v=asVaFg1NDqY), [part2](http://www.youtube.com/watch?v=ubp_cSibb9I), [part3](http://www.youtube.com/watch?v=vJAAAQv-aaw), [part4](http://www.youtube.com/watch?v=BKVlqWNh8jY), [part5](http://www.youtube.com/watch?v=iXkbLEBFjlM)
+
+Modules
+-------
+
+You can start using PETSc on Anselm by loading the PETSc module. Module names obey this pattern:
+
+```bash
+    # module load petsc/version-compiler-mpi-blas-variant, e.g.
+      module load petsc/3.4.4-icc-impi-mkl-opt
+```
+
+where `variant` is replaced by one of `{dbg, opt, threads-dbg, threads-opt}`. The `opt` variant is compiled without debugging information (no `-g` option) and with aggressive compiler optimizations (`-O3 -xAVX`). This variant is suitable for performance measurements and production runs. In all other cases use the debug (`dbg`) variant, because it contains debugging information, performs validations and self-checks, and provides a clear stack trace and message in case of an error. The other two variants `threads-dbg` and `threads-opt` are `dbg` and `opt`, respectively, built with [OpenMP and pthreads threading support](http://www.mcs.anl.gov/petsc/features/threads.html).
+
+External libraries
+------------------
+PETSc needs at least MPI, BLAS and LAPACK. These dependencies are currently satisfied with Intel MPI and Intel MKL in Anselm `petsc` modules.
+
+PETSc can be linked with a plethora of [external numerical libraries](http://www.mcs.anl.gov/petsc/miscellaneous/external.html), extending PETSc functionality, e.g. direct linear system solvers, preconditioners or partitioners. See below a list of libraries currently included in Anselm `petsc` modules.
+
+All these libraries can be used also alone, without PETSc. Their static or shared program libraries are available in
+`$PETSC_DIR/$PETSC_ARCH/lib` and header files in `$PETSC_DIR/$PETSC_ARCH/include`. `PETSC_DIR` and `PETSC_ARCH` are environment variables pointing to a specific PETSc instance based on the petsc module loaded.
+
+### Libraries linked to PETSc on Anselm (as of 11 April 2015)
+
+-   dense linear algebra
+    -   [Elemental](http://libelemental.org/)
+-   sparse linear system solvers
+    -   [Intel MKL Pardiso](https://software.intel.com/en-us/node/470282)
+    -   [MUMPS](http://mumps.enseeiht.fr/)
+    -   [PaStiX](http://pastix.gforge.inria.fr/)
+    -   [SuiteSparse](http://faculty.cse.tamu.edu/davis/suitesparse.html)
+    -   [SuperLU](http://crd.lbl.gov/~xiaoye/SuperLU/#superlu)
+    -   [SuperLU_Dist](http://crd.lbl.gov/~xiaoye/SuperLU/#superlu_dist)
+-   input/output
+    -   [ExodusII](http://sourceforge.net/projects/exodusii/)
+    -   [HDF5](http://www.hdfgroup.org/HDF5/)
+    -   [NetCDF](http://www.unidata.ucar.edu/software/netcdf/)
+-   partitioning
+    -   [Chaco](http://www.cs.sandia.gov/CRF/chac.html)
+    -   [METIS](http://glaros.dtc.umn.edu/gkhome/metis/metis/overview)
+    -   [ParMETIS](http://glaros.dtc.umn.edu/gkhome/metis/parmetis/overview)
+    -   [PT-Scotch](http://www.labri.fr/perso/pelegrin/scotch/)
+-   preconditioners & multigrid
+    -   [Hypre](http://acts.nersc.gov/hypre/)
+    -   [Trilinos ML](http://trilinos.sandia.gov/packages/ml/)
+    -   [SPAI - Sparse Approximate Inverse](https://bitbucket.org/petsc/pkg-spai)

docs.it4i/anselm-cluster-documentation/software/numerical-libraries/trilinos.md

+Trilinos
+========
+
+Packages for large scale scientific and engineering problems. Provides MPI and hybrid parallelization.
+
+### Introduction
+
+Trilinos is a collection of software packages for the numerical solution of large scale scientific and engineering problems. It is based on C++ and feautures modern object-oriented design. Both serial as well as parallel computations based on MPI and hybrid parallelization are supported within Trilinos packages.
+
+### Installed packages
+
+Current Trilinos installation on ANSELM contains (among others) the following main packages
+
+-   **Epetra** - core linear algebra package containing classes for manipulation with serial and distributed vectors, matrices, and graphs. Dense linear solvers are supported via interface to BLAS and LAPACK (Intel MKL on ANSELM). Its extension **EpetraExt** contains e.g. methods for matrix-matrix multiplication.
+-   **Tpetra** - next-generation linear algebra package. Supports 64bit indexing and arbitrary data type using C++ templates.
+-   **Belos** - library of various iterative solvers (CG, block CG, GMRES, block GMRES etc.).
+-   **Amesos** - interface to direct sparse solvers.
+-   **Anasazi** - framework for large-scale eigenvalue algorithms.
+-   **IFPACK** - distributed algebraic preconditioner (includes e.g. incomplete LU factorization)
+-   **Teuchos** - common tools packages. This package contains classes for memory management, output, performance monitoring, BLAS and LAPACK wrappers etc.
+
+For the full list of Trilinos packages, descriptions of their capabilities, and user manuals see [http://trilinos.sandia.gov.](http://trilinos.sandia.gov)
+
+### Installed version
+
+Currently, Trilinos in version 11.2.3 compiled with Intel Compiler is installed on ANSELM.
+
+### Compilling against Trilinos
+
+First, load the appropriate module:
+
+```bash
+    $ module load trilinos
+```
+
+For the compilation of CMake-aware project, Trilinos provides the FIND_PACKAGE( Trilinos ) capability, which makes it easy to build against Trilinos, including linking against the correct list of libraries. For details, see <http://trilinos.sandia.gov/Finding_Trilinos.txt>
+
+For compiling using simple makefiles, Trilinos provides Makefile.export system, which allows users to include important Trilinos variables directly into their makefiles. This can be done simply by inserting the following line into the makefile:
+
+```bash
+    include Makefile.export.Trilinos
+```
+
+or
+
+```bash
+    include Makefile.export.<package>
+```
+
+if you are interested only in a specific Trilinos package. This will give you access to the variables such as Trilinos_CXX_COMPILER, Trilinos_INCLUDE_DIRS, Trilinos_LIBRARY_DIRS etc. For the detailed description and example makefile see <http://trilinos.sandia.gov/Export_Makefile.txt>.

docs.it4i/anselm-cluster-documentation/software/nvidia-cuda.md

+nVidia CUDA
+===========
+
+##A guide to nVidia CUDA programming and GPU usage
+
+CUDA Programming on Anselm
+--------------------------
+The default programming model for GPU accelerators on Anselm is Nvidia CUDA. To set up the environment for CUDA use
+
+```bash
+    $ module load cuda
+```
+
+If the user code is hybrid and uses both CUDA and MPI, the MPI environment has to be set up as well. One way to do this is to use the PrgEnv-gnu module, which sets up correct combination of GNU compiler and MPI library.
+
+```bash
+    $ module load PrgEnv-gnu
+```
+
+CUDA code can be compiled directly on login1 or login2 nodes. User does not have to use compute nodes with GPU accelerator for compilation. To compile a CUDA source code, use nvcc compiler.
+
+```bash
+    $ nvcc --version
+```
+
+CUDA Toolkit comes with large number of examples, that can be helpful to start with. To compile and test these examples user should copy them to its home directory
+
+```bash
+    $ cd ~
+    $ mkdir cuda-samples
+    $ cp -R /apps/nvidia/cuda/6.5.14/samples/* ~/cuda-samples/
+```
+
+To compile an examples, change directory to the particular example (here the example used is deviceQuery) and run "make" to start the compilation
+
+```bash
+    $ cd ~/cuda-samples/1_Utilities/deviceQuery
+    $ make
+```
+
+To run the code user can use PBS interactive session to get access to a node from qnvidia queue (note: use your project name with parameter -A in the qsub command) and execute the binary file
+
+```bash
+    $ qsub -I -q qnvidia -A OPEN-0-0
+    $ module load cuda
+    $ ~/cuda-samples/1_Utilities/deviceQuery/deviceQuery
+```
+
+Expected output of the deviceQuery example executed on a node with Tesla K20m is
+
+```bash
+    CUDA Device Query (Runtime API) version (CUDART static linking)
+
+    Detected 1 CUDA Capable device(s)
+
+    Device 0: "Tesla K20m"
+    CUDA Driver Version / Runtime Version 5.0 / 5.0
+    CUDA Capability Major/Minor version number: 3.5
+    Total amount of global memory: 4800 MBytes (5032706048 bytes)
+    (13) Multiprocessors x (192) CUDA Cores/MP: 2496 CUDA Cores
+    GPU Clock rate: 706 MHz (0.71 GHz)
+    Memory Clock rate: 2600 Mhz
+    Memory Bus Width: 320-bit
+    L2 Cache Size: 1310720 bytes
+    Max Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536,65536), 3D=(4096,4096,4096)
+    Max Layered Texture Size (dim) x layers 1D=(16384) x 2048, 2D=(16384,16384) x 2048
+    Total amount of constant memory: 65536 bytes
+    Total amount of shared memory per block: 49152 bytes
+    Total number of registers available per block: 65536
+    Warp size: 32
+    Maximum number of threads per multiprocessor: 2048
+    Maximum number of threads per block: 1024
+    Maximum sizes of each dimension of a block: 1024 x 1024 x 64
+    Maximum sizes of each dimension of a grid: 2147483647 x 65535 x 65535
+    Maximum memory pitch: 2147483647 bytes
+    Texture alignment: 512 bytes
+    Concurrent copy and kernel execution: Yes with 2 copy engine(s)
+    Run time limit on kernels: No
+    Integrated GPU sharing Host Memory: No
+    Support host page-locked memory mapping: Yes
+    Alignment requirement for Surfaces: Yes
+    Device has ECC support: Enabled
+    Device supports Unified Addressing (UVA): Yes
+    Device PCI Bus ID / PCI location ID: 2 / 0
+    Compute Mode:
+    < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
+    deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 5.0, CUDA Runtime Version = 5.0, NumDevs = 1, Device0 = Tesla K20m
+```
+
+### Code example
+
+In this section we provide a basic CUDA based vector addition code example. You can directly copy and paste the code to test it.
+
+```bash
+    $ vim test.cu
+
+    #define N (2048*2048)
+    #define THREADS_PER_BLOCK 512
+
+    #include <stdio.h>
+    #include <stdlib.h>
+
+    // GPU kernel function to add two vectors
+    __global__ void add_gpu( int *a, int *b, int *c, int n){
+      int index = threadIdx.x + blockIdx.x * blockDim.x;
+      if (index < n)
+        c[index] = a[index] + b[index];
+    }
+
+    // CPU function to add two vectors
+    void add_cpu (int *a, int *b, int *c, int n) {
+      for (int i=0; i < n; i++)
+        c[i] = a[i] + b[i];
+    }
+
+    // CPU function to generate a vector of random integers
+    void random_ints (int *a, int n) {
+      for (int i = 0; i < n; i++)
+      a[i] = rand() % 10000; // random number between 0 and 9999
+    }
+
+    // CPU function to compare two vectors
+    int compare_ints( int *a, int *b, int n ){
+      int pass = 0;
+      for (int i = 0; i < N; i++){
+        if (a[i] != b[i]) {
+          printf("Value mismatch at location %d, values %d and %dn",i, a[i], b[i]);
+          pass = 1;
+        }
+      }
+      if (pass == 0) printf ("Test passedn"); else printf ("Test Failedn");
+      return pass;
+    }
+
+    int main( void ) {
+
+      int *a, *b, *c; // host copies of a, b, c
+      int *dev_a, *dev_b, *dev_c; // device copies of a, b, c
+      int size = N * sizeof( int ); // we need space for N integers
+
+      // Allocate GPU/device copies of dev_a, dev_b, dev_c
+      cudaMalloc( (void**)&dev_a, size );
+      cudaMalloc( (void**)&dev_b, size );
+      cudaMalloc( (void**)&dev_c, size );
+
+      // Allocate CPU/host copies of a, b, c
+      a = (int*)malloc( size );
+      b = (int*)malloc( size );
+      c = (int*)malloc( size );
+
+      // Fill input vectors with random integer numbers
+      random_ints( a, N );
+      random_ints( b, N );
+
+      // copy inputs to device
+      cudaMemcpy( dev_a, a, size, cudaMemcpyHostToDevice );
+      cudaMemcpy( dev_b, b, size, cudaMemcpyHostToDevice );
+
+      // launch add_gpu() kernel with blocks and threads
+      add_gpu<<< N/THREADS_PER_BLOCK, THREADS_PER_BLOCK >>( dev_a, dev_b, dev_c, N );
+
+      // copy device result back to host copy of c
+      cudaMemcpy( c, dev_c, size, cudaMemcpyDeviceToHost );
+
+      //Check the results with CPU implementation
+      int *c_h; c_h = (int*)malloc( size );
+      add_cpu (a, b, c_h, N);
+      compare_ints(c, c_h, N);
+
+      // Clean CPU memory allocations
+      free( a ); free( b ); free( c ); free (c_h);
+
+      // Clean GPU memory allocations
+      cudaFree( dev_a );
+      cudaFree( dev_b );
+      cudaFree( dev_c );
+
+      return 0;
+    }
+```
+
+This code can be compiled using following command
+
+```bash
+    $ nvcc test.cu -o test_cuda
+```
+
+To run the code use interactive PBS session to get access to one of the GPU accelerated nodes
+
+```bash
+    $ qsub -I -q qnvidia -A OPEN-0-0
+    $ module load cuda
+    $ ./test.cuda
+```
+
+CUDA Libraries
+--------------
+
+### CuBLAS
+
+The NVIDIA CUDA Basic Linear Algebra Subroutines (cuBLAS) library is a GPU-accelerated version of the complete standard BLAS library with 152 standard BLAS routines. Basic description of the library together with basic performance comparison with MKL can be found [here](https://developer.nvidia.com/cublas "Nvidia cuBLAS").
+
+**CuBLAS example: SAXPY**
+
+SAXPY function multiplies the vector x by the scalar alpha and adds it to the vector y overwriting the latest vector with the result. The description of the cuBLAS function can be found in [NVIDIA CUDA documentation](http://docs.nvidia.com/cuda/cublas/index.html#cublas-lt-t-gt-axpy "Nvidia CUDA documentation "). Code can be pasted in the file and compiled without any modification.
+
+```cpp
+    /* Includes, system */
+    #include <stdio.h>
+    #include <stdlib.h>
+
+    /* Includes, cuda */
+    #include <cuda_runtime.h>
+    #include <cublas_v2.h>
+
+    /* Vector size */
+    #define N  (32)
+
+    /* Host implementation of a simple version of saxpi */
+    void saxpy(int n, float alpha, const float *x, float *y)
+    {
+        for (int i = 0; i < n; ++i)
+        y[i] = alpha*x[i] + y[i];
+    }
+
+    /* Main */
+    int main(int argc, char **argv)
+    {
+        float *h_X, *h_Y, *h_Y_ref;
+        float *d_X = 0;
+        float *d_Y = 0;
+
+        const float alpha = 1.0f;
+        int i;
+
+        cublasHandle_t handle;
+
+        /* Initialize CUBLAS */
+        printf("simpleCUBLAS test running..n");
+        cublasCreate(&handle);
+
+        /* Allocate host memory for the matrices */
+        h_X = (float *)malloc(N * sizeof(h_X[0]));
+        h_Y = (float *)malloc(N * sizeof(h_Y[0]));
+        h_Y_ref = (float *)malloc(N * sizeof(h_Y_ref[0]));
+
+        /* Fill the matrices with test data */
+        for (i = 0; i < N; i++)
+        {
+            h_X[i] = rand() / (float)RAND_MAX;
+            h_Y[i] = rand() / (float)RAND_MAX;
+            h_Y_ref[i] = h_Y[i];
+        }
+
+        /* Allocate device memory for the matrices */
+        cudaMalloc((void **)&d_X, N * sizeof(d_X[0]));
+        cudaMalloc((void **)&d_Y, N * sizeof(d_Y[0]));
+
+        /* Initialize the device matrices with the host matrices */
+        cublasSetVector(N, sizeof(h_X[0]), h_X, 1, d_X, 1);
+        cublasSetVector(N, sizeof(h_Y[0]), h_Y, 1, d_Y, 1);
+
+        /* Performs operation using plain C code */
+        saxpy(N, alpha, h_X, h_Y_ref);
+
+        /* Performs operation using cublas */
+        cublasSaxpy(handle, N, &alpha, d_X, 1, d_Y, 1);
+
+        /* Read the result back */
+        cublasGetVector(N, sizeof(h_Y[0]), d_Y, 1, h_Y, 1);
+
+        /* Check result against reference */
+        for (i = 0; i < N; ++i)
+            printf("CPU res = %f t GPU res = %f t diff = %f n", h_Y_ref[i], h_Y[i], h_Y_ref[i] - h_Y[i]);
+
+        /* Memory clean up */
+        free(h_X); free(h_Y); free(h_Y_ref);
+        cudaFree(d_X); cudaFree(d_Y);
+
+        /* Shutdown */
+        cublasDestroy(handle);
+    }
+```
+
+!!! Note "Note"
+	Please note: cuBLAS has its own function for data transfers between CPU and GPU memory:
+
+	- [cublasSetVector](http://docs.nvidia.com/cuda/cublas/index.html#cublassetvector) 		- transfers data from CPU to GPU memory
+	- [cublasGetVector](http://docs.nvidia.com/cuda/cublas/index.html#cublasgetvector) - transfers data from GPU to CPU memory
+
+To compile the code using NVCC compiler a "-lcublas" compiler flag has to be specified:
+
+```bash
+    $ module load cuda
+    $ nvcc -lcublas test_cublas.cu -o test_cublas_nvcc
+```
+
+To compile the same code with GCC:
+
+```bash
+    $ module load cuda
+    $ gcc -std=c99 test_cublas.c -o test_cublas_icc -lcublas -lcudart
+```
+
+To compile the same code with Intel compiler:
+
+```bash
+    $ module load cuda intel
+    $ icc -std=c99 test_cublas.c -o test_cublas_icc -lcublas -lcudart
+```

docs.it4i/anselm-cluster-documentation/software/omics-master/diagnostic-component-team.md

+Diagnostic component (TEAM)
+===========================
+
+### Access
+
+TEAM is available at the following address: <http://omics.it4i.cz/team/>
+
+!!! Note "Note"
+	The address is accessible only via [VPN. ](../../accessing-the-cluster/vpn-access/)
+
+### Diagnostic component (TEAM)
+
+VCF files are scanned by this diagnostic tool for known diagnostic disease-associated variants. When no diagnostic mutation is found, the file can be sent to the disease-causing gene discovery tool to see wheter new disease associated variants can be found.
+
+TEAM (27) is an intuitive and easy-to-use web tool that fills the gap between the predicted mutations and the final diagnostic in targeted enrichment sequencing analysis. The tool searches for known diagnostic mutations, corresponding to a disease panel, among the predicted patient’s variants. Diagnostic variants for the disease are taken from four databases of disease-related variants (HGMD-public, HUMSAVAR , ClinVar and COSMIC) If no primary diagnostic variant is found, then a list of secondary findings that can help to establish a diagnostic is produced. TEAM also provides with an interface for the definition of and customization of panels, by means of which, genes and mutations can be added or discarded to adjust panel definitions.
+
+![Interface of the application. Panels for defining targeted regions of interest can be set up by just drag and drop known disease genes or disease definitions from the lists. Thus, virtual panels can be interactively improved as the knowledge of the disease increases.](../../../img/fig5.png)
+
+**Figure 5.** Interface of the application. Panels for defining targeted regions of interest can be set up by just drag and drop known disease genes or disease definitions from the lists. Thus, virtual panels can be interactively improved as the knowledge of the disease increases.

docs.it4i/anselm-cluster-documentation/software/omics-master/overview.md

+Overview
+========
+
+The human NGS data processing solution
+
+Introduction
+------------
+The scope of this OMICS MASTER solution is restricted to human genomics research (disease causing gene discovery in whole human genome or exome) or diagnosis (panel sequencing), although it could be extended in the future to other usages.
+
+The pipeline inputs the raw data produced by the sequencing machines and undergoes a processing procedure that consists on a quality control, the mapping and variant calling steps that result in a file containing the set of variants in the sample. From this point, the prioritization component or the diagnostic component can be launched.
+
+![OMICS MASTER solution overview. Data is produced in the external labs and comes to IT4I (represented by the blue dashed line). The data pre-processor converts raw data into a list of variants and annotations for each sequenced patient. These lists files together with primary and secondary (alignment) data files are stored in IT4I sequence DB and uploaded to the discovery (candidate priorization) or diagnostic component where they can be analysed directly by the user that produced
+them, depending of the experimental design carried out.](../../../img/fig1.png)
+
+**Figure 1.** OMICS MASTER solution overview. Data is produced in the external labs and comes to IT4I (represented by the blue dashed line). The data pre-processor converts raw data into a list of variants and annotations for each sequenced patient. These lists files together with primary and secondary (alignment) data files are stored in IT4I sequence DB and uploaded to the discovery (candidate prioritization) or diagnostic component where they can be analyzed directly by the user that produced them, depending of the experimental design carried out.
+
+Typical genomics pipelines are composed by several components that need to be launched manually. The advantage of OMICS MASTER pipeline is that all these components are invoked sequentially in an automated way.
+
+OMICS MASTER pipeline inputs a FASTQ file and outputs an enriched VCF file. This pipeline is able to queue all the jobs to PBS by only launching a process taking all the necessary input files and creates the intermediate and final folders
+
+Let’s see each of the OMICS MASTER solution components:
+
+Components
+----------
+
+### Processing
+
+This component is composed by a set of programs that carry out quality controls, alignment, realignment, variant calling and variant annotation. It turns raw data from the sequencing machine into files containing lists of variants (VCF) that once annotated, can be used by the following components (discovery and diagnosis).
+
+We distinguish three types of sequencing instruments: bench sequencers (MySeq, IonTorrent, and Roche Junior, although this last one is about being discontinued), which produce relatively Genomes in the clinic
+
+low throughput (tens of million reads), and high end sequencers, which produce high throughput (hundreds of million reads) among which we have Illumina HiSeq 2000 (and new models) and SOLiD. All of them but SOLiD produce data in sequence format. SOLiD produces data in a special format called colour space that require of specific software for the mapping process. Once the mapping has been done, the rest of the pipeline is identical. Anyway, SOLiD is a technology which is also about being discontinued by the manufacturer so, this type of data will be scarce in the future.
+
+#### Quality control, preprocessing and statistics for FASTQ
+
+FastQC& FastQC.
+
+These steps are carried out over the original FASTQ file with optimized scripts and includes the following steps: sequence cleansing, estimation of base quality scores, elimination of duplicates and statistics.
+
+Input: **FASTQ file.**
+
+Output: **FASTQ file plus an HTML file containing statistics on the data.**
+
+FASTQ format It represents the nucleotide sequence and its corresponding quality scores.
+
+![FASTQ file.](../../../img/fig2.png "fig2.png")
+**Figure 2.**FASTQ file.
+
+#### Mapping
+
+Component:** Hpg-aligner.**
+
+Sequence reads are mapped over the human reference genome. SOLiD reads are not covered by this solution; they should be mapped with specific software (among the few available options, SHRiMP seems to be the best one). For the rest of NGS machine outputs we use HPG Aligner. HPG-Aligner is an innovative solution, based on a combination of mapping with BWT and local alignment with Smith-Waterman (SW), that drastically increases mapping accuracy (97% versus 62-70% by current mappers, in the most common scenarios). This proposal provides a simple and fast solution that maps almost all the reads, even those containing a high number of mismatches or indels.
+
+Input: **FASTQ file.**
+
+Output:** Aligned file in BAM format.**
+
+**Sequence Alignment/Map (SAM)**
+
+It is a human readable tab-delimited format in which each read and its alignment is represented on a single line. The format can represent unmapped reads, reads that are mapped to unique locations, and reads that are mapped to multiple locations.
+
+The SAM format (1) consists of one header section and one alignment section. The lines in the header section start with character ‘@’, and lines in the alignment section do not. All lines are TAB delimited.
+
+In SAM, each alignment line has 11 mandatory fields and a variable number of optional fields. The mandatory fields are briefly described in Table 1. They must be present but their value can be a ‘*’ or a zero (depending on the field) if the
+corresponding information is unavailable.
+
+ |**No.** |**Name** |**Description**|
+ |--|--|
+ |1 |QNAME |Query NAME of the read or the read pai |
+ |2 |FLAG |Bitwise FLAG (pairing,strand,mate strand,etc.) |
+ |3 |RNAME |<p>Reference sequence NAME |
+ |4 |POS  |<p>1-Based  leftmost POSition of clipped alignment |
+ |5 |MAPQ  |<p>MAPping Quality (Phred-scaled) |
+ |6 |CIGAR |<p>Extended CIGAR string (operations:MIDNSHP) |
+ |7 |MRNM  |<p>Mate REference NaMe ('=' if same RNAME) |
+ |8 |MPOS  |<p>1-Based leftmost Mate POSition |
+ |9 |ISIZE |<p>Inferred Insert SIZE |
+ |10 |SEQ  |<p>Query SEQuence on the same strand as the reference |
+ |11 |QUAL |<p>Query QUALity (ASCII-33=Phred base quality) |
+
+**Table 1.** Mandatory fields in the SAM format.
+
+The standard CIGAR description of pairwise alignment defines three operations: ‘M’ for match/mismatch, ‘I’ for insertion compared with the reference and ‘D’ for deletion. The extended CIGAR proposed in SAM added four more operations: ‘N’ for skipped bases on the reference, ‘S’ for soft clipping, ‘H’ for hard clipping and ‘P’ for padding. These support splicing, clipping, multi-part and padded alignments. Figure 3 shows examples of CIGAR strings for different types of alignments.
+
+![SAM format file. The ‘@SQ’ line in the header section gives the order of reference sequences. Notably, r001 is the name of a read pair. According to FLAG 163 (=1+2+32+128), the read mapped to position 7 is the second read in the pair (128) and regarded as properly paired (1 + 2); its mate is mapped to 37 on the reverse strand (32). Read r002 has three soft-clipped (unaligned) bases. The coordinate shown in SAM is the position of the first aligned base. The CIGAR string for this alignment contains a P (padding) operation which correctly aligns the inserted sequences. Padding operations can be absent when an aligner does not support multiple sequence alignment. The last six bases of read r003 map to position 9, and the first five to position 29 on the reverse strand. The hard clipping operation H indicates that the clipped sequence is not present in the sequence field. The NM tag gives the number of mismatches. Read r004 is aligned across an intron, indicated by the N operation.](../../../img/fig3.png)
+
+**Figure 3.** SAM format file. The ‘@SQ’ line in the header section gives the order of reference sequences. Notably, r001 is the name of a read pair. According to FLAG 163 (=1+2+32+128), the read mapped to position 7 is the second read in the pair (128) and regarded as properly paired (1 + 2); its mate is mapped to 37 on the reverse strand (32). Read r002 has three soft-clipped (unaligned) bases. The coordinate shown in SAM is the position of the first aligned base. The CIGAR string for this alignment contains a P (padding) operation which correctly aligns the inserted sequences. Padding operations can be absent when an aligner does not support multiple sequence alignment. The last six bases of read r003 map to position 9, and the first five to position 29 on the reverse strand. The hard clipping operation H indicates that the clipped sequence is not present in the sequence field. The NM tag gives the number of mismatches. Read r004 is aligned across an intron, indicated by the N operation.
+
+**Binary Alignment/Map (BAM)**
+
+BAM is the binary representation of SAM and keeps exactly the same information as SAM. BAM uses lossless compression to reduce the size of the data by about 75% and provides an indexing system that allows reads that overlap a region of the genome to be retrieved and rapidly traversed.
+
+#### Quality control, preprocessing and statistics for BAM
+
+**Component:** Hpg-Fastq & FastQC. Some features:
+
+-   Quality control: % reads with N errors, % reads with multiple mappings, strand bias, paired-end insert, ...
+-   Filtering: by number of errors, number of hits, …
+    -   Comparator: stats, intersection, ...
+
+**Input:** BAM file.
+
+**Output:** BAM file plus an HTML file containing statistics.
+
+#### Variant Calling
+
+Component:** GATK.**
+
+Identification of single nucleotide variants and indels on the alignments is performed using the Genome Analysis Toolkit (GATK). GATK (2) is a software package developed at the Broad Institute to analyze high-throughput sequencing data. The toolkit offers a wide variety of tools, with a primary focus on variant discovery and genotyping as well as strong emphasis on data quality assurance.
+
+**Input:** BAM
+
+**Output:** VCF
+
+**Variant Call Format (VCF)**
+
+VCF (3) is a standardized format for storing the most prevalent types of sequence variation, including SNPs, indels and larger structural variants, together with rich annotations. The format was developed with the primary intention to represent human genetic variation, but its use is not restricted >to diploid genomes and can be used in different contexts as well. Its flexibility and user extensibility allows representation of a wide variety of genomic variation with respect to a single reference sequence.
+
+A VCF file consists of a header section and a data section. The header contains an arbitrary number of metainformation lines, each starting with characters ‘##’, and a TAB delimited field definition line, starting with a single ‘#’ character. The meta-information header lines provide a standardized description of tags and annotations used in the data section. The use of meta-information allows the information stored within a VCF file to be tailored to the dataset in question. It can be also used to provide information about the means of file creation, date of creation, version of the reference sequence, software used and any other information relevant to the history of the file. The field definition line names eight mandatory columns, corresponding to data columns representing the chromosome (CHROM), a 1-based position of the start of the variant (POS), unique identifiers of the variant (ID), the reference allele (REF), a comma separated list of  alternate non-reference alleles (ALT), a phred-scaled quality score (QUAL), site filtering information (FILTER) and a semicolon separated list of additional, user extensible annotation (INFO). In addition, if samples are present in the file, the mandatory header columns are followed by a FORMAT column and an arbitrary number of sample IDs that define the samples included in the VCF file. The FORMAT column is used to define  the information contained within each subsequent genotype column, which consists of a colon separated list of fields. For example, the FORMAT field GT:GQ:DP in the fourth data entry of Figure 1a indicates that the subsequent entries contain information regarding the genotype, genotype quality and  read depth for each sample. All data lines are TAB delimited and the number of fields in each data line must match the number of fields in the header line. It is strongly recommended that all annotation tags used are declared in the VCF header section.
+
+![a) Example of valid VCF. The header lines ##fileformat and #CHROM are mandatory, the rest is optional but strongly recommended. Each line of the body describes variants present in the sampled population at one genomic position or region. All alternate alleles are listed in the ALT column and referenced from the genotype fields as 1-based indexes to
+this list; the reference haplotype is designated as 0. For multiploid data, the separator indicates whether the data are phased (|) or unphased (/). Thus, the two alleles C and G at the positions 2 and 5 in this figure occur on the same chromosome in SAMPLE1. The first data line shows an example of a deletion (present in SAMPLE1) and a replacement of
+two bases by another base (SAMPLE2); the second line shows a SNP and an insertion; the third a SNP; the fourth a large structural variant described by the annotation in the INFO column, the coordinate is that of the base before the variant. (b–f ) Alignments and VCF representations of different sequence variants: SNP, insertion, deletion, replacement, and a large deletion. The REF columns shows the reference bases replaced by the haplotype in the ALT column. The coordinate refers to the first reference base. (g) Users are advised to use simplest representation possible and lowest coordinate in cases where the position is ambiguous.](../../../img/fig4.png)
+
+**Figure 4.** (a) Example of valid VCF. The header lines ##fileformat and #CHROM are mandatory, the rest is optional but strongly recommended. Each line of the body describes variants present in the sampled population at one genomic position or region. All alternate alleles are listed in the ALT column and referenced from the genotype fields as 1-based indexes to this list; the reference haplotype is designated as 0. For multiploid data, the separator indicates whether the data are phased (|) or unphased (/). Thus, the two alleles C and G at the positions 2 and 5 in this figure occur on the same chromosome in SAMPLE1. The first data line shows an example of a deletion (present in SAMPLE1) and a replacement of two bases by another base (SAMPLE2); the second line shows a SNP and an insertion; the third a SNP; the fourth a large structural variant described by the annotation in the INFO column, the coordinate is that of the base before the variant. (b–f ) Alignments and VCF representations of different sequence variants: SNP, insertion, deletion, replacement, and a large deletion. The REF columns shows the reference bases replaced by the haplotype in the ALT column. The coordinate refers to the first reference base. (g) Users are advised to use simplest representation possible and lowest coordinate in cases where the position is ambiguous.
+
+###Annotating
+
+**Component:** HPG-Variant
+
+The functional consequences of every variant found are then annotated using the HPG-Variant software, which extracts from CellBase**,** the Knowledge database, all the information relevant on the predicted pathologic effect of the variants.
+
+VARIANT (VARIant Analysis Tool) (4) reports information on the variants found that include consequence type and annotations taken from different databases and repositories (SNPs and variants from dbSNP and 1000 genomes, and disease-related variants from the Genome-Wide Association Study (GWAS) catalog, Online Mendelian Inheritance in Man (OMIM), Catalog of Somatic Mutations in Cancer (COSMIC) mutations, etc. VARIANT also produces a rich variety of annotations that include information on the regulatory (transcription factor or miRNAbinding sites, etc.) or structural roles, or on the selective pressures on the sites affected by the variation. This information allows extending the conventional reports beyond the coding regions and expands the knowledge on the contribution of non-coding or synonymous variants to the phenotype studied.
+
+**Input:** VCF
+
+**Output:** The output of this step is the Variant Calling Format (VCF) file, which contains changes with respect to the reference genome with the corresponding QC and functional annotations.
+
+#### CellBase
+
+CellBase(5) is a relational database integrates biological information from different sources and includes:
+
+**Core features:**
+
+We took genome sequences, genes, transcripts, exons, cytobands or cross references (xrefs) identifiers (IDs) from Ensembl (6). Protein information including sequences, xrefs or protein features (natural variants, mutagenesis sites, post-translational modifications, etc.) were imported from UniProt (7).
+
+**Regulatory:**
+
+CellBase imports miRNA from miRBase (8); curated and non-curated miRNA targets from miRecords (9), miRTarBase (10),
+TargetScan(11) and microRNA.org (12) and CpG islands and conserved regions from the UCSC database (13).
+
+**Functional annotation**
+
+OBO Foundry (14) develops many biomedical ontologies that are implemented in OBO format. We designed a SQL schema to store these OBO ontologies and &gt;30 ontologies were imported. OBO ontology term annotations were taken from Ensembl (6). InterPro (15) annotations were also imported.
+
+**Variation**
+
+CellBase includes SNPs from dbSNP (16)^; SNP population frequencies from HapMap (17), 1000 genomes project (18) and Ensembl (6); phenotypically annotated SNPs were imported from NHRI GWAS Catalog (19),HGMD (20), Open Access GWAS Database (21), UniProt (7) and OMIM (22); mutations from COSMIC (23) and structural variations from Ensembl (6).
+
+**Systems biology**
+
+We also import systems biology information like interactome information from IntAct (24). Reactome (25) stores pathway and interaction information in BioPAX (26) format. BioPAX data exchange format enables the integration of diverse pathway
+resources. We successfully solved the problem of storing data released in BioPAX format into a SQL relational schema, which allowed us importing Reactome in CellBase.
+
+### [Diagnostic component (TEAM)](diagnostic-component-team/)
+
+### [Priorization component (BiERApp)](priorization-component-bierapp/)
+
+Usage
+-----
+First of all, we should load  ngsPipeline module:
+
+```bash
+    $ module load ngsPipeline
+```
+
+This command will load  python/2.7.5 module and all the required modules (hpg-aligner, gatk, etc)
+
+If we launch ngsPipeline with ‘-h’, we will get the usage help:
+
+```bash
+    $ ngsPipeline -h
+    Usage: ngsPipeline.py [-h] -i INPUT -o OUTPUT -p PED --project PROJECT --queue
+                          QUEUE [--stages-path STAGES_PATH] [--email EMAIL]
+     [--prefix PREFIX] [-s START] [-e END] --log
+
+    Python pipeline
+
+    optional arguments:
+      -h, --help            show this help message and exit
+      -i INPUT, --input INPUT
+      -o OUTPUT, --output OUTPUT
+                            Output Data directory
+      -p PED, --ped PED     Ped file with all individuals
+      --project PROJECT     Project Id
+      --queue QUEUE         Queue Id
+      --stages-path STAGES_PATH
+                            Custom Stages path
+      --email EMAIL         Email
+      --prefix PREFIX       Prefix name for Queue Jobs name
+      -s START, --start START
+                            Initial stage
+      -e END, --end END     Final stage
+      --log                 Log to file
+
+```
+
+Let us see a brief description of the arguments:
+
+```bash
+      *-h --help*. Show the help.
+
+      *-i, --input.* The input data directory. This directory must to have a special structure. We have to create one folder per sample (with the same name). These folders will host the fastq files. These fastq files must have the following pattern “sampleName” + “_” + “1 or 2” + “.fq”. 1 for the first pair (in paired-end sequences), and 2 for the
+second one.
+
+      *-o , --output.* The output folder. This folder will contain all the intermediate and final folders. When the pipeline will be executed completely, we could remove the intermediate folders and keep only the final one (with the VCF file containing all the variants)
+
+      *-p , --ped*. The ped file with the pedigree. This file contains all the sample names. These names must coincide with the names of the input folders. If our input folder contains more samples than the .ped file, the pipeline will use only the samples from the .ped file.
+
+      *--email.* Email for PBS notifications.
+
+      *--prefix.* Prefix for PBS Job names.
+
+      *-s, --start & -e, --end.*  Initial and final stage. If we want to launch the pipeline in a specific stage we must use -s. If we want to end the pipeline in a specific stage we must use -e.
+
+      *--log*. Using log argument NGSpipeline will prompt all the logs to this file.
+
+      *--project*>. Project ID of your supercomputer allocation.
+
+      *--queue*. [Queue](../../resource-allocation-and-job-execution/introduction.html) to run the jobs in.
+```
+
+Input, output and ped arguments are mandatory. If the output folder does not exist, the pipeline will create it.
+
+Examples
+---------------------
+This is an example usage of NGSpipeline:
+
+We have a folder with the following structure in
+
+```bash
+/apps/bio/omics/1.0/sample_data/ >:
+
+    /apps/bio/omics/1.0/sample_data
+    └── data
+        ├── file.ped
+        ├── sample1
+        │   ├── sample1_1.fq
+        │   └── sample1_2.fq
+        └── sample2
+            ├── sample2_1.fq
+            └── sample2_2.fq
+```
+
+The ped file ( file.ped) contains the following info:
+
+```bash
+    #family_ID sample_ID parental_ID maternal_ID sex phenotype
+    FAM sample_A 0 0 1 1
+    FAM sample_B 0 0 2 2
+```
+
+Now, lets load the NGSPipeline module and copy the sample data to a [scratch directory](../../storage/storage/):
+
+```bash
+    $ module load ngsPipeline
+    $ mkdir -p /scratch/$USER/omics/results
+    $ cp -r /apps/bio/omics/1.0/sample_data /scratch/$USER/omics/
+```
+
+Now, we can launch the pipeline (replace OPEN-0-0 with your Project ID):
+
+```bash
+    $ ngsPipeline -i /scratch/$USER/omics/sample_data/data -o /scratch/$USER/omics/results -p /scratch/$USER/omics/sample_data/data/file.ped --project OPEN-0-0 --queue qprod
+```
+
+This command submits the processing [jobs to the queue](../../resource-allocation-and-job-execution/job-submission-and-execution.html).
+
+If we want to re-launch the pipeline from stage 4 until stage 20 we should use the next command:
+
+```bash
+    $ ngsPipeline -i /scratch/$USER/omics/sample_data/data -o /scratch/$USER/omics/results -p /scratch/$USER/omics/sample_data/data/file.ped -s 4 -e 20 --project OPEN-0-0 --queue qprod
+```
+
+Details on the pipeline
+------------------------------------
+
+The pipeline calls the following tools:
+-   [fastqc](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/), quality control tool for high throughput
+    sequence data.
+-   [gatk](https://www.broadinstitute.org/gatk/), The Genome Analysis Toolkit or GATK is a software package developed at
+    the Broad Institute to analyze high-throughput sequencing data. The toolkit offers a wide variety of tools, with a primary focus on variant discovery and genotyping as well as strong emphasis on data quality assurance. Its robust architecture, powerful processing engine and high-performance computing features make it capable of taking on projects of any size.
+-   [hpg-aligner](http://wiki.opencb.org/projects/hpg/doku.php?id=aligner:downloads), HPG Aligner has been designed to align short and long reads with high sensitivity, therefore any number of mismatches or indels are allowed. HPG Aligner implements and combines two well known algorithms: *Burrows-Wheeler Transform* (BWT) to speed-up mapping high-quality reads, and *Smith-Waterman*> (SW) to increase sensitivity when reads cannot be mapped using BWT.
+-   [hpg-fastq](http://docs.bioinfo.cipf.es/projects/fastqhpc/wiki), a quality control tool for high throughput sequence data.
+-   [hpg-variant](http://wiki.opencb.org/projects/hpg/doku.php?id=variant:downloads), The HPG Variant suite is an ambitious project aimed to provide a complete suite of tools to work with genomic variation data, from VCF tools to variant profiling or genomic statistics. It is being implemented using High Performance Computing technologies to provide the best performance possible.
+-   [picard](http://picard.sourceforge.net/), Picard comprises Java-based command-line utilities that manipulate SAM files, and a Java API (HTSJDK) for creating new programs that read and write SAM files. Both SAM text format and SAM binary (BAM) format are supported.
+-   [samtools](http://samtools.sourceforge.net/samtools-c.shtml), SAM Tools provide various utilities for manipulating alignments in the SAM format, including sorting, merging, indexing and generating alignments in a per-position format.
+-   [snpEff](http://snpeff.sourceforge.net/), Genetic variant annotation and effect prediction toolbox.
+
+This listing show which tools are used in each step of the pipeline :
+
+-   stage-00: fastqc
+-   stage-01: hpg_fastq
+-   stage-02: fastqc
+-   stage-03: hpg_aligner and samtools
+-   stage-04: samtools
+-   stage-05: samtools
+-   stage-06: fastqc
+-   stage-07: picard
+-   stage-08: fastqc
+-   stage-09: picard
+-   stage-10: gatk
+-   stage-11: gatk
+-   stage-12: gatk
+-   stage-13: gatk
+-   stage-14: gatk
+-   stage-15: gatk
+-   stage-16: samtools
+-   stage-17: samtools
+-   stage-18: fastqc
+-   stage-19: gatk
+-   stage-20: gatk
+-   stage-21: gatk
+-   stage-22: gatk
+-   stage-23: gatk
+-   stage-24: hpg-variant
+-   stage-25: hpg-variant
+-   stage-26: snpEff
+-   stage-27: snpEff
+-   stage-28: hpg-variant
+
+Interpretation
+---------------------------
+
+The output folder contains all the subfolders with the intermediate data. This folder contains the final VCF with all the variants. This file can be uploaded into [TEAM](diagnostic-component-team.html) by using the VCF file button. It is important to note here that the entire management of the VCF file is local: no patient’s sequence data is sent over the Internet thus avoiding any problem of data privacy or confidentiality.
+
+![TEAM upload panel. Once the file has been uploaded, a panel must be chosen from the Panel list. Then, pressing the Run button the diagnostic process starts.]((../../../img/fig7.png)
+
+**Figure 7**. *TEAM upload panel.* *Once the file has been uploaded, a panel must be chosen from the Panel* list. Then, pressing the Run button the diagnostic process starts.
+
+Once the file has been uploaded, a panel must be chosen from the Panel list. Then, pressing the Run button the diagnostic process starts. TEAM searches first for known diagnostic mutation(s) taken from four databases: HGMD-public (20), [HUMSAVAR](http://www.uniprot.org/docs/humsavar), ClinVar (29)^ and COSMIC (23).
+
+![The panel manager. The elements used to define a panel are (A) disease terms, (B) diagnostic mutations and (C) genes. Arrows represent actions that can be taken in the panel manager. Panels can be defined by using the known mutations and genes of a particular disease. This can be done by dragging them to the Primary Diagnostic box (action D). This action, in addition to defining the diseases in the Primary Diagnostic box, automatically adds the corresponding genes to the Genes box. The panels can be customized by adding new genes (action F) or removing undesired genes (action G). New disease mutations can be added independently or associated to an already existing disease term (action E). Disease terms can be removed by simply dragging themback (action H).](../../../img/fig7x.png)
+
+**Figure 7.** *The panel manager. The elements used to define a panel are (**A**) disease terms, (**B**) diagnostic mutations and (**C**) genes. Arrows represent actions that can be taken in the panel manager. Panels can be defined by using the known mutations and genes of a particular disease. This can be done by dragging them to the **Primary Diagnostic** box (action **D**). This action, in addition to defining the diseases in the **Primary Diagnostic** box, automatically adds the corresponding genes to the **Genes** box. The panels can be customized by adding new genes (action **F**) or removing undesired genes (action **G**). New disease mutations can be added independently or associated to an already existing disease term (action **E**). Disease terms can be removed by simply dragging them back (action **H**).*
+
+For variant discovering/filtering we should upload the VCF file into BierApp by using the following form:
+
+*![BierApp VCF upload panel. It is recommended to choose a name for the job as well as a description.](../../../img/fig8.png)*
+
+**Figure 8.** *BierApp VCF upload panel. It is recommended to choose a name for the job as well as a description.**
+
+Each prioritization (‘job’) has three associated screens that facilitate the filtering steps. The first one, the ‘Summary’ tab, displays a statistic of the data set analyzed, containing the samples analyzed, the number and types of variants found and its distribution according to consequence types. The second screen, in the ‘Variants and effect’ tab, is the actual filtering tool, and the third one, the ‘Genome view’ tab, offers a representation of the selected variants within the genomic context provided by an embedded version of the Genome Maps Tool (30).
+
+![This picture shows all the information associated to the variants. If a variant has an associated phenotype we could see it in the last column. In this case, the variant 7:132481242 C&gt;T is associated to the phenotype: large intestine tumor.](../../../img/fig9.png)
+
+**Figure 9.** This picture shows all the information associated to the variants. If a variant has an associated phenotype we could see it in the last column. In this case, the variant 7:132481242 C&gt;T is associated to the phenotype: large intestine tumor.
+
+References
+-----------------------
+
+1.  Heng Li, Bob Handsaker, Alec Wysoker, Tim Fennell, Jue Ruan, Nils Homer, Gabor Marth5, Goncalo Abecasis6, Richard Durbin and 1000 Genome Project Data Processing Subgroup: The Sequence Alignment/Map format and SAMtools. Bioinformatics 2009, 25: 2078-2079.
+2.  McKenna A, Hanna M, Banks E, Sivachenko A, Cibulskis K, Kernytsky A, Garimella K, Altshuler D, Gabriel S, Daly M, DePristo MA: The Genome Analysis Toolkit: a MapReduce framework for analyzing next-generation DNA sequencing data.   *Genome Res* >2010, 20:1297-1303.
+3.  Petr Danecek, Adam Auton, Goncalo Abecasis, Cornelis A. Albers, Eric Banks, Mark A. DePristo, Robert E. Handsaker, Gerton Lunter, Gabor T. Marth, Stephen T. Sherry, Gilean McVean, Richard Durbin, and 1000 Genomes Project Analysis Group. The variant call format and VCFtools. Bioinformatics 2011, 27: 2156-2158.
+4.  Medina I, De Maria A, Bleda M, Salavert F, Alonso R, Gonzalez CY, Dopazo J: VARIANT: Command Line, Web service and Web interface for fast and accurate functional characterization of variants found by Next-Generation Sequencing. Nucleic Acids Res 2012, 40:W54-58.
+5.  Bleda M, Tarraga J, de Maria A, Salavert F, Garcia-Alonso L, Celma M, Martin A, Dopazo J, Medina I: CellBase, a  comprehensive collection of RESTful web services for retrieving relevant biological information from heterogeneous sources. Nucleic Acids Res 2012, 40:W609-614.
+6.  Flicek,P., Amode,M.R., Barrell,D., Beal,K., Brent,S., Carvalho-Silva,D., Clapham,P., Coates,G., Fairley,S., Fitzgerald,S. et al. (2012) Ensembl 2012. Nucleic Acids Res., 40, D84–D90.
+7.  UniProt Consortium. (2012) Reorganizing the protein space at the Universal Protein Resource (UniProt). Nucleic   Acids Res., 40, D71–D75.
+8.  Kozomara,A. and Griffiths-Jones,S. (2011) miRBase: integrating microRNA annotation and deep-sequencing data.    Nucleic Acids Res., 39, D152–D157.
+9.  Xiao,F., Zuo,Z., Cai,G., Kang,S., Gao,X. and Li,T. (2009) miRecords: an integrated resource for microRNA-target interactions. Nucleic Acids Res., 37, D105–D110.
+10. Hsu,S.D., Lin,F.M., Wu,W.Y., Liang,C., Huang,W.C., Chan,W.L., Tsai,W.T., Chen,G.Z., Lee,C.J., Chiu,C.M. et al. (2011) miRTarBase: a database curates experimentally validated microRNA-target interactions. Nucleic Acids Res., 39, D163–D169.
+11. Friedman,R.C., Farh,K.K., Burge,C.B. and Bartel,D.P. (2009) Most mammalian mRNAs are conserved targets of microRNAs. Genome Res., 19, 92–105. 12. Betel,D., Wilson,M., Gabow,A., Marks,D.S. and Sander,C. (2008) The microRNA.org resource: targets and expression. Nucleic Acids Res., 36, D149–D153.
+13. Dreszer,T.R., Karolchik,D., Zweig,A.S., Hinrichs,A.S., Raney,B.J., Kuhn,R.M., Meyer,L.R., Wong,M., Sloan,C.A., Rosenbloom,K.R. et al. (2012) The UCSC genome browser database: extensions and updates 2011. Nucleic Acids Res.,40, D918–D923.
+14. Smith,B., Ashburner,M., Rosse,C., Bard,J., Bug,W., Ceusters,W., Goldberg,L.J., Eilbeck,K., Ireland,A., Mungall,C.J. et al. (2007) The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration. Nat.    Biotechnol., 25, 1251–1255.
+15.  Hunter,S., Jones,P., Mitchell,A., Apweiler,R., Attwood,T.K.,Bateman,A., Bernard,T., Binns,D., Bork,P., Burge,S. et al. (2012) InterPro in 2011: new developments in the family and domain prediction database. Nucleic Acids Res.,40, D306–D312.
+16.  Sherry,S.T., Ward,M.H., Kholodov,M., Baker,J., Phan,L., Smigielski,E.M. and Sirotkin,K. (2001) dbSNP: the NCBI database of genetic variation. Nucleic Acids Res., 29, 308–311.
+17.  Altshuler,D.M., Gibbs,R.A., Peltonen,L., Dermitzakis,E., Schaffner,S.F., Yu,F., Bonnen,P.E., de Bakker,P.I.,  Deloukas,P., Gabriel,S.B. et al. (2010) Integrating common and rare genetic variation in diverse human populations. Nature, 467, 52–58.
+18.  1000 Genomes Project Consortium. (2010) A map of human genome variation from population-scale sequencing. Nature,    467, 1061–1073.
+19.  Hindorff,L.A., Sethupathy,P., Junkins,H.A., Ramos,E.M., Mehta,J.P., Collins,F.S. and Manolio,T.A. (2009)   Potential etiologic and functional implications of genome-wide association loci for human diseases and traits. Proc. Natl Acad.    Sci. USA, 106, 9362–9367.
+20.  Stenson,P.D., Ball,E.V., Mort,M., Phillips,A.D., Shiel,J.A., Thomas,N.S., Abeysinghe,S., Krawczak,M.   and Cooper,D.N. (2003) Human gene mutation database (HGMD):    2003 update. Hum. Mutat., 21, 577–581.
+21.  Johnson,A.D. and O’Donnell,C.J. (2009) An open access database of genome-wide association results. BMC Med.    Genet, 10, 6.
+22.  McKusick,V. (1998) A Catalog of Human Genes  and Genetic  Disorders, 12th edn. John Hopkins University    Press,Baltimore, MD.
+23.  Forbes,S.A., Bindal,N., Bamford,S., Cole,C.,    Kok,C.Y., Beare,D., Jia,M., Shepherd,R., Leung,K., Menzies,A. et al.    (2011) COSMIC: mining complete cancer genomes in the catalogue of    somatic mutations in cancer. Nucleic Acids Res.,    39, D945–D950.
+24.  Kerrien,S., Aranda,B., Breuza,L., Bridge,A.,    Broackes-Carter,F., Chen,C., Duesbury,M., Dumousseau,M.,    Feuermann,M., Hinz,U. et al. (2012) The Intact molecular interaction    database in 2012. Nucleic Acids Res., 40, D841–D846.
+25.  Croft,D., O’Kelly,G., Wu,G., Haw,R.,    Gillespie,M., Matthews,L., Caudy,M., Garapati,P.,    Gopinath,G., Jassal,B. et al. (2011) Reactome: a database of    reactions, pathways and biological processes. Nucleic Acids Res.,    39, D691–D697.
+26.  Demir,E., Cary,M.P., Paley,S., Fukuda,K.,    Lemer,C., Vastrik,I.,Wu,G., D’Eustachio,P., Schaefer,C., Luciano,J.    et al. (2010) The BioPAX community standard for pathway    data sharing. Nature Biotechnol., 28, 935–942.
+27.  Alemán Z, García-García F, Medina I, Dopazo J    (2014): A web tool for the design and management of panels of genes    for targeted enrichment and massive sequencing for    clinical applications. Nucleic Acids Res 42: W83-7.
+28.  [Alemán    A](http://www.ncbi.nlm.nih.gov/pubmed?term=Alem%C3%A1n%20A%5BAuthor%5D&cauthor=true&cauthor_uid=24803668)>, [Garcia-Garcia    F](http://www.ncbi.nlm.nih.gov/pubmed?term=Garcia-Garcia%20F%5BAuthor%5D&cauthor=true&cauthor_uid=24803668)>, [Salavert    F](http://www.ncbi.nlm.nih.gov/pubmed?term=Salavert%20F%5BAuthor%5D&cauthor=true&cauthor_uid=24803668)>, [Medina    I](http://www.ncbi.nlm.nih.gov/pubmed?term=Medina%20I%5BAuthor%5D&cauthor=true&cauthor_uid=24803668)>, [Dopazo    J](http://www.ncbi.nlm.nih.gov/pubmed?term=Dopazo%20J%5BAuthor%5D&cauthor=true&cauthor_uid=24803668)> (2014).    A web-based interactive framework to assist in the prioritization of    disease candidate genes in whole-exome sequencing studies.    [Nucleic    Acids Res.](http://www.ncbi.nlm.nih.gov/pubmed/?term=BiERapp "Nucleic acids research.")>42 :W88-93.
+29.  Landrum,M.J., Lee,J.M., Riley,G.R., Jang,W.,    Rubinstein,W.S., Church,D.M. and Maglott,D.R. (2014) ClinVar: public    archive of relationships among sequence variation and    human phenotype. Nucleic Acids Res., 42, D980–D985.
+30.  Medina I, Salavert F, Sanchez R, de Maria A,    Alonso R, Escobar P, Bleda M, Dopazo J: Genome Maps, a new    generation genome browser. Nucleic Acids Res 2013, 41:W41-46.

docs.it4i/anselm-cluster-documentation/software/omics-master/priorization-component-bierapp.md

+Priorization component (BiERApp)
+================================
+
+### Access
+
+BiERApp is available at the following address: <http://omics.it4i.cz/bierapp/>
+
+!!! Note "Note"
+	The address is accessible onlyvia [VPN. ](../../accessing-the-cluster/vpn-access/)
+
+###BiERApp
+
+**This tool is aimed to discover new disease genes or variants by studying affected families or cases and controls. It carries out a filtering process to sequentially remove: (i) variants which are not no compatible with the disease because are not expected to have impact on the protein function; (ii) variants that exist at frequencies incompatible with the disease; (iii) variants that do not segregate with the disease. The result is a reduced set of disease gene candidates that should be further validated experimentally.**
+
+BiERapp (28) efficiently helps in the identification of causative variants in family and sporadic genetic diseases. The program reads lists of predicted variants (nucleotide substitutions and indels) in affected individuals or tumor samples and controls. In family studies, different modes of inheritance can easily be defined to filter out variants that do not segregate with the disease along the family. Moreover, BiERapp integrates additional information such as allelic frequencies in the general population and the most popular damaging scores to further narrow down the number of putative variants in successive filtering steps. BiERapp provides an interactive and user-friendly interface that implements the filtering strategy used in the context of a large-scale genomic project carried out by the Spanish Network for Research, in Rare Diseases (CIBERER) and the Medical Genome Project. in which more than 800 exomes have been analyzed.
+
+![Web interface to the prioritization tool. This figure shows the interface of the web tool for candidate gene prioritization with the filters available. The tool includes a genomic viewer (Genome Maps 30) that enables the representation of the variants in the corresponding genomic coordinates.](../../../img/fig6.png)
+
+**Figure 6**. Web interface to the prioritization tool.This figure shows the interface of the web tool for candidate gene
+prioritization with the filters available. The tool includes a genomic viewer (Genome Maps 30) that enables the representation of the variants in the corresponding genomic coordinates.
+

docs.it4i/anselm-cluster-documentation/software/openfoam.md

+OpenFOAM
+========
+
+##A free, open source CFD software package
+
+Introduction
+----------------
+OpenFOAM is a free, open source CFD software package developed by [**OpenCFD Ltd**](http://www.openfoam.com/about) at [**ESI Group**](http://www.esi-group.com/) and distributed by the [**OpenFOAM Foundation **](http://www.openfoam.org/). It has a large user base across most areas of engineering and science, from both commercial and academic organisations.
+
+Homepage: <http://www.openfoam.com/>
+
+###Installed version
+
+Currently, several version compiled by GCC/ICC compilers in single/double precision with several version of openmpi are available on Anselm.
+
+For example syntax of available OpenFOAM module is:
+
+&lt; openfoam/2.2.1-icc-openmpi1.6.5-DP &gt;
+
+this means openfoam version 2.2.1 compiled by ICC compiler with openmpi1.6.5 in double precision.
+
+Naming convection of the installed versions is following:
+
+openfoam/&lt;>VERSION>&gt;-&lt;>COMPILER<span>&gt;-&lt;</span><span>openmpiVERSION</span><span>&gt;-&lt;</span><span>PRECISION</span><span>&gt;</span>
+
+-   &lt;VERSION>&gt; - version of openfoam
+-   &lt;COMPILER&gt; - version of used compiler
+-   &lt;openmpiVERSION&gt; - version of used openmpi/impi
+-   &lt;PRECISION&gt; - DP/SP – double/single precision
+
+###Available OpenFOAM modules
+
+To check available modules use
+
+```bash
+    $ module avail
+```
+
+In /opt/modules/modulefiles/engineering you can see installed engineering softwares:
+
+```bash
+    ------------------------------------ /opt/modules/modulefiles/engineering -------------------------------------------------------------
+    ansys/14.5.x               matlab/R2013a-COM                                openfoam/2.2.1-icc-impi4.1.1.036-DP
+    comsol/43b-COM             matlab/R2013a-EDU                                openfoam/2.2.1-icc-openmpi1.6.5-DP
+    comsol/43b-EDU             openfoam/2.2.1-gcc481-openmpi1.6.5-DP            paraview/4.0.1-gcc481-bullxmpi1.2.4.1-osmesa10.0
+    lsdyna/7.x.x               openfoam/2.2.1-gcc481-openmpi1.6.5-SP
+```
+
+For information how to use modules please [look here](../environment-and-modules/ "Environment and Modules ").
+
+Getting Started
+-------------------
+
+To create OpenFOAM environment on ANSELM give the commands:
+
+```bash
+    $ module load openfoam/2.2.1-icc-openmpi1.6.5-DP
+
+    $ source $FOAM_BASHRC
+```
+
+!!! Note "Note"
+	Please load correct module with your requirements “compiler - GCC/ICC, precision - DP/SP”.
+
+Create a project directory within the $HOME/OpenFOAM directory named >&lt;USER&gt;-&lt;OFversion&gt; and create a directory named run within it, e.g. by typing:
+
+```bash
+    $ mkdir -p $FOAM_RUN
+```
+
+Project directory is now available by typing:
+
+```bash
+    $ cd /home/<USER>/OpenFOAM/<USER>-<OFversion>/run
+```
+
+&lt;OFversion&gt; - for example &lt;2.2.1&gt;
+
+or
+
+```bash
+    $ cd $FOAM_RUN
+```
+
+Copy the tutorial examples directory in the OpenFOAM distribution to the run directory:
+
+```bash
+    $ cp -r $FOAM_TUTORIALS $FOAM_RUN
+```
+
+Now you can run the first case for example incompressible laminar flow in a cavity.
+
+Running Serial Applications
+-------------------------------
+
+Create a Bash script >test.sh
+
+```bash
+    #!/bin/bash
+    module load openfoam/2.2.1-icc-openmpi1.6.5-DP
+    source $FOAM_BASHRC
+
+    # source to run functions
+    . $WM_PROJECT_DIR/bin/tools/RunFunctions
+
+    cd $FOAM_RUN/tutorials/incompressible/icoFoam/cavity
+
+    runApplication blockMesh
+    runApplication icoFoam
+```
+
+Job submission
+
+```bash
+    $ qsub -A OPEN-0-0 -q qprod -l select=1:ncpus=16,walltime=03:00:00 test.sh
+```
+For information about job submission please [look here](../resource-allocation-and-job-execution/job-submission-and-execution/ "Job submission").
+
+Running applications in parallel
+-------------------------------------------------
+Run the second case for example external incompressible turbulent flow - case - motorBike.
+
+First we must run serial application bockMesh and decomposePar for preparation of parallel computation.
+
+!!! Note "Note"
+	Create a Bash scrip test.sh:
+
+```bash
+    #!/bin/bash
+    module load openfoam/2.2.1-icc-openmpi1.6.5-DP
+    source $FOAM_BASHRC
+
+    # source to run functions
+    . $WM_PROJECT_DIR/bin/tools/RunFunctions
+
+    cd $FOAM_RUN/tutorials/incompressible/simpleFoam/motorBike
+
+    runApplication blockMesh
+    runApplication decomposePar
+```
+
+Job submission
+
+```bash
+    $ qsub -A OPEN-0-0 -q qprod -l select=1:ncpus=16,walltime=03:00:00 test.sh
+```
+
+This job create simple block mesh and domain decomposition. Check your decomposition, and submit parallel computation:
+
+!!! Note "Note"
+	Create a PBS script testParallel.pbs:
+
+```bash
+    #!/bin/bash
+    #PBS -N motorBike
+    #PBS -l select=2:ncpus=16
+    #PBS -l walltime=01:00:00
+    #PBS -q qprod
+    #PBS -A OPEN-0-0
+
+    module load openfoam/2.2.1-icc-openmpi1.6.5-DP
+    source $FOAM_BASHRC
+
+    cd $FOAM_RUN/tutorials/incompressible/simpleFoam/motorBike
+
+    nproc = 32
+
+    mpirun -hostfile ${PBS_NODEFILE} -np $nproc snappyHexMesh -overwrite -parallel | tee snappyHexMesh.log
+
+    mpirun -hostfile ${PBS_NODEFILE} -np $nproc potentialFoam -noFunctionObject-writep -parallel | tee potentialFoam.log
+
+    mpirun -hostfile ${PBS_NODEFILE} -np $nproc simpleFoam -parallel | tee simpleFoam.log
+```
+
+nproc – number of subdomains
+
+Job submission
+
+```bash
+    $ qsub testParallel.pbs
+```
+Compile your own solver
+----------------------------------------
+
+Initialize OpenFOAM environment before compiling your solver
+
+```bash
+    $ module load openfoam/2.2.1-icc-openmpi1.6.5-DP
+    $ source $FOAM_BASHRC
+    $ cd $FOAM_RUN/
+```
+
+Create directory applications/solvers in user directory
+
+```bash
+    $ mkdir -p applications/solvers
+    $ cd applications/solvers
+```
+
+Copy icoFoam solver’s source files
+
+```bash
+    $ cp -r $FOAM_SOLVERS/incompressible/icoFoam/ My_icoFoam
+    $ cd My_icoFoam
+```
+
+Rename icoFoam.C to My_icoFOAM.C
+
+```bash
+    $ mv icoFoam.C My_icoFoam.C
+```
+
+Edit >*files* file in *Make* directory:
+
+```bash
+    icoFoam.C
+    EXE = $(FOAM_APPBIN)/icoFoam
+```
+
+and change to:
+
+```bash
+    My_icoFoam.C
+    EXE = $(FOAM_USER_APPBIN)/My_icoFoam
+```
+
+In directory My_icoFoam give the compilation command:
+
+```bash
+    $ wmake
+```
+
+------------------------------------------------------------------------
+ **Have a fun with OpenFOAM :)**

docs.it4i/anselm-cluster-documentation/software/operating-system.md

+Operating System
+===============
+
+The operating system on Anselm is Linux - **bullx Linux Server release 6.X**
+
+bullx Linux is based on Red Hat Enterprise Linux. bullx Linux is a Linux distribution provided by Bull and dedicated to HPC applications.
+

docs.it4i/anselm-cluster-documentation/software/paraview.md

+ParaView
+========
+
+##An open-source, multi-platform data analysis and visualization application
+
+Introduction
+------------
+
+**ParaView** is an open-source, multi-platform data analysis and visualization application. ParaView users can quickly build visualizations to analyze their data using qualitative and quantitative techniques. The data exploration can be done interactively in 3D or programmatically using ParaView's batch processing capabilities.
+
+ParaView was developed to analyze extremely large datasets using distributed memory computing resources. It can be run on supercomputers to analyze datasets of exascale size as well as on laptops for smaller data.
+
+Homepage : <http://www.paraview.org/>
+
+Installed version
+-----------------
+Currently, version 4.0.1 compiled with GCC 4.8.1 against Bull MPI library and OSMesa 10.0 is installed on Anselm.
+
+Usage
+-----
+On Anselm, ParaView is to be used in client-server mode. A parallel ParaView server is launched on compute nodes by the user, and client is launched on your desktop PC to control and view the visualization. Download ParaView client application for your OS here: <http://paraview.org/paraview/resources/software.php>. Important : **your version must match the version number installed on Anselm** ! (currently v4.0.1)
+
+### Launching server
+
+To launch the server, you must first allocate compute nodes, for example
+
+```bash
+    $ qsub -I -q qprod -A OPEN-0-0 -l select=2
+```
+
+to launch an interactive session on 2 nodes. Refer to [Resource Allocation and Job Execution](../resource-allocation-and-job-execution/introduction/) for details.
+
+After the interactive session is opened, load the ParaView module :
+
+```bash
+    $ module add paraview
+```
+
+Now launch the parallel server, with number of nodes times 16 processes:
+
+```bash
+    $ mpirun -np 32 pvserver --use-offscreen-rendering
+    Waiting for client...
+    Connection URL: cs://cn77:11111
+    Accepting connection(s): cn77:11111
+```
+
+Note the that the server is listening on compute node cn77 in this case, we shall use this information later.
+
+### Client connection
+
+Because a direct connection is not allowed to compute nodes on Anselm, you must establish a SSH tunnel to connect to the server. Choose a port number on your PC to be forwarded to ParaView server, for example 12345. If your PC is running Linux, use this command to estabilish a SSH tunnel:
+
+```bash
+    ssh -TN -L 12345:cn77:11111 username@anselm.it4i.cz
+```
+
+replace  username with your login and cn77 with the name of compute node your ParaView server is running on (see previous step). If you use PuTTY on Windows, load Anselm connection configuration, t>hen go to Connection-&gt; SSH>-&gt;Tunnels to set up the port forwarding. Click Remote radio button. Insert 12345 to Source port textbox. Insert cn77:11111. Click Add button, then Open.
+
+Now launch ParaView client installed on your desktop PC. Select File-&gt;Connect..., click Add Server. Fill in the following :
+
+Name : Anselm tunnel
+Server Type : Client/Server
+Host : localhost
+Port : 12345
+
+Click Configure, Save, the configuration is now saved for later use. Now click Connect to connect to the ParaView server. In your terminal where you have interactive session with ParaView server launched, you should see:
+
+```bash
+    Client connected.
+```
+
+You can now use Parallel ParaView.
+
+### Close server
+
+Remember to close the interactive session after you finish working with ParaView server, as it will remain launched even after your client is disconnected and will continue to consume resources.
+
+GPU support
+-----------
+Currently, GPU acceleration is not supported in the server and ParaView will not take advantage of accelerated nodes on Anselm. Support for GPU acceleration might be added in the future.

docs.it4i/anselm-cluster-documentation/storage/cesnet-data-storage.md

+CESNET Data Storage
+===================
+
+Introduction
+------------
+Do not use shared filesystems at IT4Innovations as a backup for large amount of data or long-term archiving purposes.
+
+!!! Note "Note"
+	The IT4Innovations does not provide storage capacity for data archiving. Academic staff and students of research institutions in the Czech Republic can use [CESNET Storage service](https://du.cesnet.cz/).
+
+The CESNET Storage service can be used for research purposes, mainly by academic staff and students of research institutions in the Czech Republic.
+
+User of data storage CESNET (DU) association can become organizations or an individual person who is either in the current employment relationship (employees) or the current study relationship (students) to a legal entity (organization) that meets the “Principles for access to CESNET Large infrastructure (Access Policy)”.
+
+User may only use data storage CESNET for data transfer and storage which are associated with activities in science, research, development, the spread of education, culture and prosperity. In detail see “Acceptable Use Policy CESNET Large Infrastructure (Acceptable Use Policy, AUP)”.
+
+The service is documented at <https://du.cesnet.cz/wiki/doku.php/en/start>. For special requirements please contact directly CESNET Storage Department via e-mail [du-support(at)cesnet.cz](mailto:du-support@cesnet.cz).
+
+The procedure to obtain the CESNET access is quick and trouble-free.
+
+(source [https://du.cesnet.cz/](https://du.cesnet.cz/wiki/doku.php/en/start "CESNET Data Storage"))
+
+CESNET storage access
+---------------------
+
+### Understanding Cesnet storage
+
+!!! Note "Note"
+	It is very important to understand the Cesnet storage before uploading data. Please read <https://du.cesnet.cz/en/navody/home-migrace-plzen/start> first.
+
+Once registered for CESNET Storage, you may [access the storage](https://du.cesnet.cz/en/navody/faq/start) in number of ways. We recommend the SSHFS and RSYNC methods.
+
+### SSHFS Access
+
+!!! Note "Note"
+	SSHFS: The storage will be mounted like a local hard drive
+
+The SSHFS  provides a very convenient way to access the CESNET Storage. The storage will be mounted onto a local directory, exposing the vast CESNET Storage as if it was a local removable harddrive. Files can be than copied in and out in a usual fashion.
+
+First, create the mountpoint
+
+```bash
+    $ mkdir cesnet
+```
+
+Mount the storage. Note that you can choose among the ssh.du1.cesnet.cz (Plzen), ssh.du2.cesnet.cz (Jihlava), ssh.du3.cesnet.cz (Brno) Mount tier1_home **(only 5120M !)**:
+
+```bash
+    $ sshfs username@ssh.du1.cesnet.cz:. cesnet/
+```
+
+For easy future access from Anselm, install your public key
+
+```bash
+    $ cp .ssh/id_rsa.pub cesnet/.ssh/authorized_keys
+```
+
+Mount tier1_cache_tape for the Storage VO:
+
+```bash
+    $ sshfs username@ssh.du1.cesnet.cz:/cache_tape/VO_storage/home/username cesnet/
+```
+
+View the archive, copy the files and directories in and out
+
+```bash
+    $ ls cesnet/
+    $ cp -a mydir cesnet/.
+    $ cp cesnet/myfile .
+```
+
+Once done, please remember to unmount the storage
+
+```bash
+    $ fusermount -u cesnet
+```
+
+### Rsync access
+
+!!! Note "Note"
+	Rsync provides delta transfer for best performance, can resume interrupted transfers
+
+Rsync is a fast and extraordinarily versatile file copying tool. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination.  Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
+
+Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time.  Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated.
+
+More about Rsync at <https://du.cesnet.cz/en/navody/rsync/start#pro_bezne_uzivatele>
+
+Transfer large files to/from Cesnet storage, assuming membership in the Storage VO
+
+```bash
+    $ rsync --progress datafile username@ssh.du1.cesnet.cz:VO_storage-cache_tape/.
+    $ rsync --progress username@ssh.du1.cesnet.cz:VO_storage-cache_tape/datafile .
+```
+
+Transfer large directories to/from Cesnet storage, assuming membership in the Storage VO
+
+```bash
+    $ rsync --progress -av datafolder username@ssh.du1.cesnet.cz:VO_storage-cache_tape/.
+    $ rsync --progress -av username@ssh.du1.cesnet.cz:VO_storage-cache_tape/datafolder .
+```
+
+Transfer rates of about 28MB/s can be expected.

docs.it4i/anselm-cluster-documentation/storage/storage.md

+Storage
+=======
+
+There are two main shared file systems on Anselm cluster, the [HOME](../storage/#home) and [SCRATCH](../storage/#scratch). All login and compute nodes may access same data on shared filesystems. Compute nodes are also equipped with local (non-shared) scratch, ramdisk and tmp filesystems.
+
+Archiving
+---------
+
+Please don't use shared filesystems as a backup for large amount of data or long-term archiving mean. The academic staff and students of research institutions in the Czech Republic can use [CESNET storage service](cesnet-data-storage/), which is available via SSHFS.
+
+Shared Filesystems
+------------------
+
+Anselm computer provides two main shared filesystems, the [HOME filesystem](../storage.html#home) and the [SCRATCH filesystem](../storage/#scratch). Both HOME and SCRATCH filesystems are realized as a parallel Lustre filesystem. Both shared file systems are accessible via the Infiniband network. Extended ACLs are provided on both Lustre filesystems for the purpose of sharing data with other users using fine-grained control.
+
+### Understanding the Lustre Filesystems
+
+(source <http://www.nas.nasa.gov>)
+
+A user file on the Lustre filesystem can be divided into multiple chunks (stripes) and stored across a subset of the object storage targets (OSTs) (disks). The stripes are distributed among the OSTs in a round-robin fashion to ensure load balancing.
+
+When a client (a  compute node from your job) needs to create or access a file, the client queries the metadata server ( MDS) and the metadata target ( MDT) for the layout and location of the [file's stripes](http://www.nas.nasa.gov/hecc/support/kb/Lustre_Basics_224.html#striping). Once the file is opened and the client obtains the striping information, the  MDS is no longer involved in the file I/O process. The client interacts directly with the object storage servers (OSSes) and OSTs to perform I/O operations such as locking, disk allocation, storage, and retrieval.
+
+If multiple clients try to read and write the same part of a file at the same time, the Lustre distributed lock manager enforces coherency so that all clients see consistent results.
+
+There is default stripe configuration for Anselm Lustre filesystems. However, users can set the following stripe parameters for their own directories or files to get optimum I/O performance:
+
+1.  stripe_size: the size of the chunk in bytes; specify with k, m, or g to use units of KB, MB, or GB, respectively; the size must be an even multiple of 65,536 bytes; default is 1MB for all Anselm Lustre filesystems
+2.  stripe_count the number of OSTs to stripe across; default is 1 for Anselm Lustre filesystems  one can specify -1 to use all OSTs in the filesystem.
+3.  stripe_offset The index of the OST where the first stripe is to be placed; default is -1 which results in random selection; using a non-default value is NOT recommended.
+
+!!! Note "Note"
+	Setting stripe size and stripe count correctly for your needs may significantly impact the I/O performance you experience.
+
+Use the lfs getstripe for getting the stripe parameters. Use the lfs setstripe command for setting the stripe parameters to get optimal I/O performance The correct stripe setting depends on your needs and file access patterns. 
+
+```bash
+$ lfs getstripe dir|filename
+$ lfs setstripe -s stripe_size -c stripe_count -o stripe_offset dir|filename 
+```
+
+Example:
+
+```bash
+$ lfs getstripe /scratch/username/
+/scratch/username/
+stripe_count:   1 stripe_size:    1048576 stripe_offset:  -1
+
+$ lfs setstripe -c -1 /scratch/username/
+$ lfs getstripe /scratch/username/
+/scratch/username/
+stripe_count:  10 stripe_size:    1048576 stripe_offset:  -1
+```
+
+In this example, we view current stripe setting of the /scratch/username/ directory. The stripe count is changed to all OSTs, and verified. All files written to this directory will be striped over 10 OSTs
+
+Use lfs check OSTs to see the number and status of active OSTs for each filesystem on Anselm. Learn more by reading the man page
+
+```bash
+$ lfs check osts
+$ man lfs
+```
+
+### Hints on Lustre Stripping
+
+!!! Note "Note"
+	Increase the stripe_count for parallel I/O to the same file.
+
+When multiple processes are writing blocks of data to the same file in parallel, the I/O performance for large files will improve when the stripe_count is set to a larger value. The stripe count sets the number of OSTs the file will be written to. By default, the stripe count is set to 1. While this default setting provides for efficient access of metadata (for example to support the ls -l command), large files should use stripe counts of greater than 1. This will increase the aggregate I/O bandwidth by using multiple OSTs in parallel instead of just one. A rule of thumb is to use a stripe count approximately equal to the number of gigabytes in the file.
+
+Another good practice is to make the stripe count be an integral factor of the number of processes performing the write in parallel, so that you achieve load balance among the OSTs. For example, set the stripe count to 16 instead of 15 when you have 64 processes performing the writes.
+
+!!! Note "Note"
+	Using a large stripe size can improve performance when accessing very large files
+
+Large stripe size allows each client to have exclusive access to its own part of a file. However, it can be counterproductive in some cases if it does not match your I/O pattern. The choice of stripe size has no effect on a single-stripe file.
+
+Read more on <http://wiki.lustre.org/manual/LustreManual20_HTML/ManagingStripingFreeSpace.html>
+
+### Lustre on Anselm
+
+The  architecture of Lustre on Anselm is composed of two metadata servers (MDS) and four data/object storage servers (OSS). Two object storage servers are used for file system HOME and another two object storage servers are used for file system SCRATCH.
+
+ Configuration of the storages
+
+-    HOME Lustre object storage
+    -   One disk array NetApp E5400
+    -   22 OSTs
+    -   227 2TB NL-SAS 7.2krpm disks
+    -   22 groups of 10 disks in RAID6 (8+2)
+    -   7 hot-spare disks
+-    SCRATCH Lustre object storage
+    -   Two disk arrays NetApp E5400
+    -   10 OSTs
+    -   106 2TB NL-SAS 7.2krpm disks
+    -   10 groups of 10 disks in RAID6 (8+2)
+    -   6 hot-spare disks
+-    Lustre metadata storage
+    -   One disk array NetApp E2600
+    -   12 300GB SAS 15krpm disks
+    -   2 groups of 5 disks in RAID5
+    -   2 hot-spare disks
+
+###HOME
+
+The HOME filesystem is mounted in directory /home. Users home directories /home/username reside on this filesystem. Accessible capacity is 320TB, shared among all users. Individual users are restricted by filesystem usage quotas, set to 250GB per user. If 250GB should prove as insufficient for particular user, please contact [support](https://support.it4i.cz/rt), the quota may be lifted upon request.
+
+!!! Note "Note"
+	The HOME filesystem is intended for preparation, evaluation, processing and storage of data generated by active Projects.
+
+The HOME filesystem should not be used to archive data of past Projects or other unrelated data.
+
+The files on HOME filesystem will not be deleted until end of the [users lifecycle](../../get-started-with-it4innovations/obtaining-login-credentials/obtaining-login-credentials/).
+
+The filesystem is backed up, such that it can be restored in case of catasthropic failure resulting in significant data loss. This backup however is not intended to restore old versions of user data or to restore (accidentaly) deleted files.
+
+The HOME filesystem is realized as Lustre parallel filesystem and is available on all login and computational nodes.
+Default stripe size is 1MB, stripe count is 1. There are 22 OSTs dedicated for the HOME filesystem.
+
+!!! Note "Note"
+	Setting stripe size and stripe count correctly for your needs may significantly impact the I/O performance you experience.
+
+|HOME filesystem||
+|---|---|
+|Mountpoint|/home|
+|Capacity|320TB|
+|Throughput|2GB/s|
+|User quota|250GB|
+|Default stripe size|1MB|
+|Default stripe count|1|
+|Number of OSTs|22|
+
+###SCRATCH
+
+The SCRATCH filesystem is mounted in directory /scratch. Users may freely create subdirectories and files on the filesystem. Accessible capacity is 146TB, shared among all users. Individual users are restricted by filesystem usage quotas, set to 100TB per user. The purpose of this quota is to prevent runaway programs from filling the entire filesystem and deny service to other users. If 100TB should prove as insufficient for particular user, please contact [support](https://support.it4i.cz/rt), the quota may be lifted upon request.
+
+!!! Note "Note"
+	The Scratch filesystem is intended  for temporary scratch data generated during the calculation as well as for high performance access to input and output files. All I/O intensive jobs must use the SCRATCH filesystem as their working directory.
+
+    >Users are advised to save the necessary data from the SCRATCH filesystem to HOME filesystem after the calculations and clean up the scratch files.
+
+	Files on the SCRATCH filesystem that are **not accessed for more than 90 days** will be automatically **deleted**.
+
+The SCRATCH filesystem is realized as Lustre parallel filesystem and is available from all login and computational nodes. Default stripe size is 1MB, stripe count is 1. There are 10 OSTs dedicated for the SCRATCH filesystem.
+
+!!! Note "Note"
+	Setting stripe size and stripe count correctly for your needs may significantly impact the I/O performance you experience.
+
+|SCRATCH filesystem||
+|---|---|
+|Mountpoint|/scratch|
+|Capacity|146TB|
+|Throughput|6GB/s|
+|User quota|100TB|
+|Default stripe size|1MB|
+|Default stripe count|1|
+|Number of OSTs|10|
+
+### Disk usage and quota commands
+
+User quotas on the file systems can be checked and reviewed using following command:
+
+```bash
+$ lfs quota dir
+```
+
+Example for Lustre HOME directory:
+
+```bash
+$ lfs quota /home
+Disk quotas for user user001 (uid 1234):
+    Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
+         /home  300096       0 250000000       -    2102       0  500000    -
+Disk quotas for group user001 (gid 1234):
+    Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
+        /home  300096       0       0       -    2102       0       0       -
+```
+
+In this example, we view current quota size limit of 250GB and 300MB currently used by user001.
+
+Example for Lustre SCRATCH directory:
+
+```bash
+$ lfs quota /scratch
+Disk quotas for user user001 (uid 1234):
+     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
+          /scratch       8       0 100000000000       -       3       0       0       -
+Disk quotas for group user001 (gid 1234):
+ Filesystem kbytes quota limit grace files quota limit grace
+ /scratch       8       0       0       -       3       0       0       -
+```
+
+In this example, we view current quota size limit of 100TB and 8KB currently used by user001.
+
+To have a better understanding of where the space is exactly used, you can use following command to find out.
+
+```bash
+$ du -hs dir
+```
+
+Example for your HOME directory:
+
+```bash
+$ cd /home
+$ du -hs * .[a-zA-z0-9]* | grep -E "[0-9]*G|[0-9]*M" | sort -hr
+258M     cuda-samples
+15M      .cache
+13M      .mozilla
+5,5M     .eclipse
+2,7M     .idb_13.0_linux_intel64_app
+```
+
+This will list all directories which are having MegaBytes or GigaBytes of consumed space in your actual (in this example HOME) directory. List is sorted in descending order from largest to smallest files/directories.
+
+To have a better understanding of previous commands, you can read manpages.
+
+```bash
+$ man lfs
+```
+
+```bash
+$ man du
+```
+
+### Extended ACLs
+
+Extended ACLs provide another security mechanism beside the standard POSIX ACLs which are defined by three entries (for owner/group/others). Extended ACLs have more than the three basic entries. In addition, they also contain a mask entry and may contain any number of named user and named group entries.
+
+ACLs on a Lustre file system work exactly like ACLs on any Linux file system. They are manipulated with the standard tools in the standard manner. Below, we create a directory and allow a specific user access.
+
+```bash
+[vop999@login1.anselm ~]$ umask 027
+[vop999@login1.anselm ~]$ mkdir test
+[vop999@login1.anselm ~]$ ls -ld test
+drwxr-x--- 2 vop999 vop999 4096 Nov  5 14:17 test
+[vop999@login1.anselm ~]$ getfacl test
+# file: test
+# owner: vop999
+# group: vop999
+user::rwx
+group::r-x
+other::---
+
+[vop999@login1.anselm ~]$ setfacl -m user:johnsm:rwx test
+[vop999@login1.anselm ~]$ ls -ld test
+drwxrwx---+ 2 vop999 vop999 4096 Nov  5 14:17 test
+[vop999@login1.anselm ~]$ getfacl test
+# file: test
+# owner: vop999
+# group: vop999
+user::rwx
+user:johnsm:rwx
+group::r-x
+mask::rwx
+other::---
+```
+
+Default ACL mechanism can be used to replace setuid/setgid permissions on directories. Setting a default ACL on a directory (-d flag to setfacl) will cause the ACL permissions to be inherited by any newly created file or subdirectory within the directory. Refer to this page for more information on Linux ACL:
+
+[http://www.vanemery.com/Linux/ACL/POSIX_ACL_on_Linux.html ](http://www.vanemery.com/Linux/ACL/POSIX_ACL_on_Linux.html)
+
+Local Filesystems
+-----------------
+
+### Local Scratch
+
+!!! Note "Note"
+	Every computational node is equipped with 330GB local scratch disk.
+
+Use local scratch in case you need to access large amount of small files during your calculation.
+
+The local scratch disk is mounted as /lscratch and is accessible to user at /lscratch/$PBS_JOBID directory.
+
+The local scratch filesystem is intended  for temporary scratch data generated during the calculation as well as for high performance access to input and output files. All I/O intensive jobs that access large number of small files within the calculation must use the local scratch filesystem as their working directory. This is required for performance reasons, as frequent access to number of small files may overload the metadata servers (MDS) of the Lustre filesystem.
+
+!!! Note "Note"
+	The local scratch directory /lscratch/$PBS_JOBID will be deleted immediately after the calculation end. Users should take care to save the output data from within the jobscript.
+
+|local SCRATCH filesystem||
+|---|---|
+|Mountpoint|/lscratch|
+|Accesspoint|/lscratch/$PBS_JOBID|
+|Capacity|330GB|
+|Throughput|100MB/s|
+|User quota|none|
+
+### RAM disk
+
+Every computational node is equipped with filesystem realized in memory, so called RAM disk.
+
+!!! Note "Note"
+	Use RAM disk in case you need really fast access to your data of limited size during your calculation. Be very careful, use of RAM disk filesystem is at the expense of operational memory.
+
+The local RAM disk is mounted as /ramdisk and is accessible to user at /ramdisk/$PBS_JOBID directory.
+
+The local RAM disk filesystem is intended for temporary scratch data generated during the calculation as well as for high performance access to input and output files. Size of RAM disk filesystem is limited. Be very careful, use of RAM disk filesystem is at the expense of operational memory.  It is not recommended to allocate large amount of memory and use large amount of data in RAM disk filesystem at the same time.
+
+!!! Note "Note"
+	The local RAM disk directory /ramdisk/$PBS_JOBID will be deleted immediately after the calculation end. Users should take care to save the output data from within the jobscript.
+
+|RAM disk||
+|---|---|
+|Mountpoint| /ramdisk|
+|Accesspoint| /ramdisk/$PBS_JOBID|
+|Capacity|60GB at compute nodes without accelerator, 90GB at compute nodes with accelerator, 500GB at fat nodes|
+|Throughput|over 1.5 GB/s write, over 5 GB/s read, single thread, over 10 GB/s write, over 50 GB/s read, 16 threads|
+|User quota|none|
+
+### tmp
+
+Each node is equipped with local /tmp directory of few GB capacity. The /tmp directory should be used to work with small temporary files. Old files in /tmp directory are automatically purged.
+
+Summary
+----------
+
+|Mountpoint|Usage|Protocol|Net Capacity|Throughput|Limitations|Access|Services|
+|---|---|
+|/home|home directory|Lustre|320 TiB|2 GB/s|Quota 250GB|Compute and login nodes|backed up|
+|/scratch|cluster shared jobs' data|Lustre|146 TiB|6 GB/s|Quota 100TB|Compute and login nodes|files older 90 days removed|
+|/lscratch|node local jobs' data|local|330 GB|100 MB/s|none|Compute nodes|purged after job ends|
+|/ramdisk|node local jobs' data|local|60, 90, 500 GB|5-50 GB/s|none|Compute nodes|purged after job ends|
+|/tmp|local temporary files|local|9.5 GB|100 MB/s|none|Compute and login nodes|auto| purged

docs.it4i/downtimes_history.md

+History of Downtimes
+====================
+
+Full history of important announcements related to IT4I infrastructure, planned downtimes, outages etc.
+
+ |Date and time |Title and description |
+ | --- | --- |
+ |2016-08-29 00:00:00 |**Salomon Maintenance** The Salomon supercomputer will be down for the updates from 2016-09-19 11:00 CEST to 2016-09-21 11:00 CEST. |
+ |2016-07-29 00:00:00 |**Anselm outage** There was an unplanned outage of Anselm cluster due to network problems. Anselm is now back in production. |
+ |2016-07-27 13:28:51 |**Back in Production** The Salomon and Anselm supercomputers are online! The flooding in the computer room caused by the failed coolant water pipe is now cleared, damage to infrastructure is virtually none.The 8th Open Access Call deadline is extended till Friday 2016-07-29 to compensate the inaccessibility of the extranet.it4i.cz portal. |
+ |2016-07-20 08:45:00 |**Anselm planned downtime** There's a planned maintenance window from 2016-08-16 07:00 till 2016-08-18 16:00 CEST. |
+ |2016-06-29 13:50:00 |**Salomon cluster maintenance outage prolonged** Important! Salomon cluster maintenance outage will be prolonged till 2016-06-29 20:00 CEST. |
+ |2016-06-16 00:00:00 |**Salomon planned downtime** There's a planned maintenance window from 2016-06-28 09:00 till 2016-06-29 20:00 CEST.Thank you for understanding,the IT4Innovations team |
+ |2016-05-26 10:31:44 |**Salomon planned downtime** There's a planned maintenance window from 2016-06-08 09:00 till 2016-06-09 09:00 CEST.Thank you for understanding,the IT4Innovations team |
+ |2016-04-27 15:57:28 |**Salomon cluster maintenance outage prolonged** Important! Salomon cluster maintenance outage will be prolonged till 2016-04-28 14:00 CEST |
+ |2016-03-31 19:03:25 |**Failure on Salomon Cooling System** We have very serious issue with Salomon cooling system since 2016-03-31 10:00. We are working to resolve the issue. |
+ |2016-03-31 18:59:04 |**Salomon Back in Production** As of 2016-03-31 19:30 CET, the Salomon is back in production. The outage was caused by an issue in cooling system. |
+ |2016-03-30 15:57:57 |**PBS malfunction** We've had several issues with PBS scheduler since 2016-03-30 13:00 CEST. We are still working on it. |
+ |2016-03-26 09:52:41 |**Salomon back to production** We have recovered all the issues with the Salomon cluster. |
+ |2016-03-26 09:51:16 |**Failure on Salomon Cooling Infrastructure** We had an issue with the cooling infra of Salomon. This issue led to InfiniBand and storage outage. We are working to resolve the issue. |
+ |2016-03-14 14:57:51 |**Infrastructure Maintenance** The Salomon supercomputer will be down for the maintenance from 2016-03-22 10:00 CEST to 2016-03-22 17:00 CEST. |
+ |2016-02-24 12:30:00 |**Anselm Upgrade** The Anselm supercomputer will be down for the updates from 2016-02-01 to 2016-02-29. |
+ |2016-02-23 08:45:45 |**Anselm Upgrade** The Anselm supercomputer will be down for the updates from 2016-02-01 to 2016-02-29. |
+ |2016-02-14 08:07:46 |**Failure on Salomon Infiniband Network** We have very serious issue with Salomon Infiniband network since 2016-02-11 10:18. We are working to resolve the issue as quickly as possible and apologize for any inconvenience. |
+ |2016-02-04 11:53:30 |**Short network outage** We need to apply some changes in network devices settings which may cause short network outage for Anselm Login nodes. This work will start around 6 am 2016-01-26.Thanks for your understanding. |
+ |2016-02-04 11:52:37 |**Salomon Upgrade** The Salomon supercomputer will be down for the updates from 2016-02-16 09:00 CEST to 2016-02-16 13:00 CEST. |
+ |2016-02-04 11:52:25 |**Salomon Upgrade** The Salomon supercomputer will be down for the updates from 2016-02-16 09:00 CEST to 2016-02-16 13:00 CEST. |
+ |2015-11-26 08:11:25 |**/home downtime** Dear HPC usersThere's a /home downtime on the Salomon supercomputer planned for 25th November. The reason is a maintenance of the underlying CXFS filesystem. Your jobs will be scheduled with respect to this maintenance window.Thank you for understanding,the IT4I team |
+ |2015-11-24 09:10:15 |**The /home filesystem was down** On 23.11.2015, 13:55 - 14:55, the /home filesystem was down due to acute technical problems.>We apologize for inconvenience. |
+ |2015-09-04 16:14:12 |**SCRATCH downtime** Dear IT4I usersSalomon's SCRATCH will *not* be accessible on Thursday (10th September 2015) from 13:00 till 18:00 CEST.Thank you for understanding,the IT4Innovations team |
+ |2015-08-27 00:00:00 |**Today's SCRATCH downtime** Dear IT4I usersWe are sorry for today's (27th August) inaccessibility of SCRATCH filesystem due to a broken service which normally provides mapping for user/group IDs (UIDs/GIDs). The issue has been fixed. No data were lost.Thank you for understanding,the IT4Innovations team |
+ |2015-08-12 00:00:00 |**Unplanned downtime** Dear Salomon users,there was an unplanned downtime of the non-accelerated nodes. At this moment, systems are booting and we are revising consequences. Temporarily inaccessible SCRATCH filesystem is one of them.We're sorry for the inconvenience,the IT4I team |
+ |2015-08-06 00:00:00 |**SCRATCH downtime** Dear IT4I usersSalomon's SCRATCH will not be accessible tomorrow (7th August 2015) from 08:30 till 11:00 CEST.Thank you for understanding,the IT4Innovations team |
+ |2014-11-14 10:27:51 |**Unplanned PBS Downtime** Dear Anselm users,we apologize for the unavailability of our PBS scheduler during the last weekend. However, running jobs shouldn't have been affected at that time.Thank you for understanding,Anselm Admins |
+ |2014-11-14 10:27:50 |**Login1 troubles** Login1 had a short unplanned downtime. Sorry for the troubles. |
+ |2014-10-14 20:30:00 |**Unexpected power failure** Dear Anselm users,>>on Tuesday 14th approximately at 17:20 CEST we encountered power failure during service operation on backup diesel generator. The system shut down. Additional checks after the shutdown took more time than what would expect. The system was back on-line with all services approximately at 21:00 CEST. We are very sorry for any troubles, this matter may caused you.  If some of your jobs ended in incorrect state, please feel free to reclaim your core hours.>>Thank you for understanding, Anselm Administrators |
+ |2014-07-17 13:50:00 |**Login2(!) downtime** Dear Anselm users,there's an upgrade planned on Friday, 18th July from 13:00 till 16:00 CEST. Please, take in mind that login2.anselm.it4i.cz will be unavailable at the given time-frame. We are sorry for the inconvenience.Thank you for understanding,Anselm Admins |
+ |2014-07-16 13:11:34 |**Login1 downtime** Dear Anselm users,there's an upgrade planned on Thursday, 17th July from 13:00 till 16:00 CEST. Please, take in mind that login1.anselm.it4i.cz will be unavailable at the given time-frame. We are sorry for the inconvenience.Thank you for understanding,Anselm Admins |
+ |2014-06-18 10:51:56 |**Login2 downtime** Dear Anselm users,there's an upgrade planned on Wednesday, 18th June from 11:20 till 14:20 CEST. Please, take in mind that login2.anselm.it4i.cz will be unavailable at the given time-frame. We are sorry for the inconvenience.Thank you for understanding,Anselm Admins |
+ |2014-05-22 00:00:00 |**Outage** Dear Anselm users.As of today (20140523 10:45) we had an unmanaged outage of a few nodes. Affected nodes were cn[117-126,193-195].Sorry for the inconveniences,Anselm admins |
+ |2014-04-11 11:30:00 |**Heartbleed bug** Dear users of the Anselm cluster,A serious bug in the OpenSSL library, known as the "Heartbleed bug" has been recently discovered. >We would like to ensure you that IT4I has taken all necessary steps to fix the OpenSSL library on all the systems.The bug in OpenSSL library affected many sites worldwide for nearly two years. At this moment, there is no evidence that any abuse of data took place at IT4I.In order to ensure the security and integrity of IT4I systems, all users will be issued new login credentials, including password and ssh keys. For more informations about the Heartbleed bug, please see: [https://docs.it4i.cz/heartbleed-bug](https://docs.it4i.cz/heartbleed-bug) Thank you for your understanding.IT4Innovations team |
+ |2014-04-02 13:05:00 |**Scheduler is Down** We are sorry for the current scheduler issues which are caused by an inconsistency of the internal PBS database. Thus it's not possible to interact with the scheduler now. In addition, some jobs may be affected and some job outputs may not be retrieved at this moment. |
+ |2014-03-26 15:50:00 |**Temporary Scratch Mount on Login1** Because of the Lustre issues (mentioned in previous announcements) there's a temporary mount point for Scratch filesystem on login1 node. Please, follow this path to access your data:/scratch_nfs/ |
+ |2014-03-26 13:10:00 |**Both Login Nodes Inaccessible** We are sorry for the inaccessibility of both login nodes. We are co-operating with our supplier and trying hard to solve this problem as soon as possible.Thank you for understanding. |
+ |2014-03-25 22:05:00 |**Login1(!) Not Responding** Currently, if you're having trouble while accessing Anselm, please use address login2.anselm.it4i.cz instead of anselm.it4i.cz. There's a Lustre issue with the login1 node, which causes it to not respond.We'll let you know by MOTD when login1 comes back online. |
+ |2013-12-03 00:00:00 |**Planned Downtime** On 17th December 08:00 to 18:00 CET Anselm will be down for maintenance. Power supply upgrade will take place, as well as system maintenance and software updates.Prior to the period:- Jobs will be scheduled for running with a respect to the downtime.During the period:- No Anselm HPC service will be available.- Following web applications will not be accessible: Request Tracker, Anselm cluster documentation, Anselm Allocation.- Submitting tickets through the e-mail address will be delayed.After the period:- All service will be brought back to normal.- Jobs in a 'Q' state will be scheduled for running.We are sorry for the inconvenience. |
+ |2013-10-14 00:00:00 |**Cooling system unstable** Dear Anselm users,there was an unplanned downtime due to severe issues with the cold doors today. We are trying hard to bring all services up. We assume that we will finish the maintenance at about 13:30 CEST.Thank you for understanding.Sincerely yours,Anselm admins |
+ |2013-09-17 15:50:00 |**A Fair Amount of Nodes Down** Dear Anselm users,We had an outage on the Anselm Cluster. A fair amount of the nodes was unavailable to production.Consider terminated job resubmission.We are sorry for the troubles,Anselm admins |
+ |2013-08-23 15:25:00 |**Infiniband Maintenance Window** Dear Anselm users,We would like to inform you about a planned Infiniband maintenance window, on Wednesday, 28th August from 09:00 till 16:30 CEST.No Anselm service will be available during this outage.New batch jobs will not be scheduled for running during this time. Consider altering the job walltime, to achieve job execution prior to the downtime (See Job Submission in the Anselm Documentation).Sincerely yours,Anselm Admins |
+ |2013-08-14 17:10:00 |**Planned Upgrade / Scheduler Downtime** Dear Anselm users,there's an upgrade planned on Thursday, 15th August from 18:00 till 22:00 CEST. Please, take in mind that PBS scheduler won't accept your jobs at the given timeframe. We are sorry for the inconvenience.Thank you for understanding,Anselm Admins |
+ |2013-07-16 13:45:00 |**Anselm Cluster Upgrade - July 23rd** Dear Anselm users.We would like to inform you that Anselm cluster will be unavailable due to upgrades on Tuesday, July 23rd from 07:00 to 20:00 CEST.Sorry for the inconvenience,Anselm admins--Anselm cluster documentation can be found at:[http://support.it4i.cz/docs/anselm-cluster-documentation/"](http://support.it4i.cz/docs/anselm-cluster-documentation/) |
+ |2013-06-27 09:35:00 |**SSH Password Authentication** Dear Anselm users.If you are experiencing some troubles when using SSH PasswordAuthentication on the client side, please switch to PubkeyAuthentication instead.We are trying hard to resolve this issue.Sincerely yours,Anselm admins |
\ No newline at end of file

docs.it4i/get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/cygwin-and-x11-forwarding.md

+Cygwin and X11 forwarding
+=========================
+
+### If  no able to forward X11 using PuTTY to CygwinX
+
+```bash
+[usename@login1.anselm ~]$ gnome-session &
+[1] 23691
+[usename@login1.anselm ~]$ PuTTY X11 proxy: unable to connect to forwarded X server: Network error: Connection refused
+PuTTY X11 proxy: unable to connect to forwarded X server: Network error: Connection refused
+
+  (gnome-session:23691): WARNING **: Cannot open display:**
+```
+
+1. Locate and modify Cygwin shortcut that uses [startxwin](http://x.cygwin.com/docs/man1/startxwin.1.html)
+    locate
+    C:cygwin64binXWin.exe
+    change it
+    to
+    C:*cygwin64binXWin.exe -listen tcp*
+
+![XWin-listen-tcp.png](../../../img/XWinlistentcp.png "XWin-listen-tcp.png")
+
+2. Check Putty settings:
+     Enable X11 forwarding
+
+    ![](../../../img/cygwinX11forwarding.png)