Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • sccs/docs.it4i.cz
  • soj0018/docs.it4i.cz
  • lszustak/docs.it4i.cz
  • jarosjir/docs.it4i.cz
  • strakpe/docs.it4i.cz
  • beranekj/docs.it4i.cz
  • tab0039/docs.it4i.cz
  • davidciz/docs.it4i.cz
  • gui0013/docs.it4i.cz
  • mrazek/docs.it4i.cz
  • lriha/docs.it4i.cz
  • it4i-vhapla/docs.it4i.cz
  • hol0598/docs.it4i.cz
  • sccs/docs-it-4-i-cz-fumadocs
  • siw019/docs-it-4-i-cz-fumadocs
15 results
Show changes
Hide whitespace changes
Inline Side-by-side
Showing
with 2807 additions and 0 deletions
docs.it4i/anselm-cluster-documentation/software/numerical-languages/r.md 0 → 100644
View file @ 14676d74
# R
## Introduction
The R is a language and environment for statistical computing and graphics. R provides a wide variety of statistical (linear and nonlinear modelling, classical statistical tests, time-series analysis, classification, clustering, ...) and graphical techniques, and is highly extensible.
One of R's strengths is the ease with which well-designed publication-quality plots can be produced, including mathematical symbols and formulae where needed. Great care has been taken over the defaults for the minor design choices in graphics, but the user retains full control.
Another convenience is the ease with which the C code or third party libraries may be integrated within R.
Extensive support for parallel computing is available within R.
Read more on <http://www.r-project.org/>, <http://cran.r-project.org/doc/manuals/r-release/R-lang.html>
## Modules
The R version 3.0.1 is available on Anselm, along with GUI interface Rstudio
| Application | Version | module |
| ----------- | ------------ | ------- |
| **R** | R 3.0.1 | R |
| **Rstudio** | Rstudio 0.97 | Rstudio |
```bash
$ module load R
```
## Execution
The R on Anselm is linked to highly optimized MKL mathematical library. This provides threaded parallelization to many R kernels, notably the linear algebra subroutines. The R runs these heavy calculation kernels without any penalty. By default, the R would parallelize to 16 threads. You may control the threads by setting the OMP_NUM_THREADS environment variable.
### Interactive execution
To run R interactively, using Rstudio GUI, log in with ssh -X parameter for X11 forwarding. Run rstudio:
```bash
$ module load Rstudio
$ rstudio
```
### Batch execution
To run R in batch mode, write an R script, then write a bash jobscript and execute via the qsub command. By default, R will use 16 threads when running MKL kernels.
Example jobscript:
```bash
#!/bin/bash
# change to local scratch directory
cd /lscratch/$PBS_JOBID || exit
# copy input file to scratch
cp $PBS_O_WORKDIR/rscript.R .
# load R module
module load R
# execute the calculation
R CMD BATCH rscript.R routput.out
# copy output file to home
cp routput.out $PBS_O_WORKDIR/.
#exit
exit
```
This script may be submitted directly to the PBS workload manager via the qsub command. The inputs are in rscript.R file, outputs in routput.out file. See the single node jobscript example in the [Job execution section](../../resource-allocation-and-job-execution/job-submission-and-execution/).
## Parallel R
Parallel execution of R may be achieved in many ways. One approach is the implied parallelization due to linked libraries or specially enabled functions, as [described above](r/#interactive-execution). In the following sections, we focus on explicit parallelization, where parallel constructs are directly stated within the R script.
## Package parallel
The package parallel provides support for parallel computation, including by forking (taken from package multicore), by sockets (taken from package snow) and random-number generation.
The package is activated this way:
```bash
$ R
> library(parallel)
```
More information and examples may be obtained directly by reading the documentation available in R
```bash
> ?parallel
> library(help = "parallel")
> vignette("parallel")
```
Download the package [parallell](package-parallel-vignette.pdf) vignette.
The forking is the most simple to use. Forking family of functions provide parallelized, drop in replacement for the serial apply() family of functions.
!!! Note "Note"
Forking via package parallel provides functionality similar to OpenMP construct
omp parallel for
Only cores of single node can be utilized this way!
Forking example:
```bash
library(parallel)
#integrand function
f <- function(i,h) {
x <- h*(i-0.5)
return (4/(1 + x*x))
}
#initialize
size <- detectCores()
while (TRUE)
{
#read number of intervals
cat("Enter the number of intervals: (0 quits) ")
fp<-file("stdin"); n<-scan(fp,nmax=1); close(fp)
if(n<=0) break
#run the calculation
n <- max(n,size)
h <- 1.0/n
i <- seq(1,n);
pi3 <- h*sum(simplify2array(mclapply(i,f,h,mc.cores=size)));
#print results
cat(sprintf("Value of PI %16.14f, diff= %16.14fn",pi3,pi3-pi))
}
```
The above example is the classic parallel example for calculating the number π. Note the **detectCores()** and **mclapply()** functions. Execute the example as:
```bash
$ R --slave --no-save --no-restore -f pi3p.R
```
Every evaluation of the integrad function runs in parallel on different process.
## Package Rmpi
!!! Note "Note"
package Rmpi provides an interface (wrapper) to MPI APIs.
It also provides interactive R slave environment. On Anselm, Rmpi provides interface to the [OpenMPI](../mpi-1/Running_OpenMPI/).
Read more on Rmpi at <http://cran.r-project.org/web/packages/Rmpi/>, reference manual is available at <http://cran.r-project.org/web/packages/Rmpi/Rmpi.pdf>
When using package Rmpi, both openmpi and R modules must be loaded
```bash
$ module load openmpi
$ module load R
```
Rmpi may be used in three basic ways. The static approach is identical to executing any other MPI programm. In addition, there is Rslaves dynamic MPI approach and the mpi.apply approach. In the following section, we will use the number π integration example, to illustrate all these concepts.
### static Rmpi
Static Rmpi programs are executed via mpiexec, as any other MPI programs. Number of processes is static - given at the launch time.
Static Rmpi example:
```cpp
library(Rmpi)
#integrand function
f <- function(i,h) {
x <- h*(i-0.5)
return (4/(1 + x*x))
}
#initialize
invisible(mpi.comm.dup(0,1))
rank <- mpi.comm.rank()
size <- mpi.comm.size()
n<-0
while (TRUE)
{
#read number of intervals
if (rank==0) {
cat("Enter the number of intervals: (0 quits) ")
fp<-file("stdin"); n<-scan(fp,nmax=1); close(fp)
}
#broadcat the intervals
n <- mpi.bcast(as.integer(n),type=1)
if(n<=0) break
#run the calculation
n <- max(n,size)
h <- 1.0/n
i <- seq(rank+1,n,size);
mypi <- h*sum(sapply(i,f,h));
pi3 <- mpi.reduce(mypi)
#print results
if (rank==0) cat(sprintf("Value of PI %16.14f, diff= %16.14fn",pi3,pi3-pi))
}
mpi.quit()
```
The above is the static MPI example for calculating the number π. Note the **library(Rmpi)** and **mpi.comm.dup()** function calls.
Execute the example as:
```bash
$ mpiexec R --slave --no-save --no-restore -f pi3.R
```
### dynamic Rmpi
Dynamic Rmpi programs are executed by calling the R directly. openmpi module must be still loaded. The R slave processes will be spawned by a function call within the Rmpi program.
Dynamic Rmpi example:
```cpp
#integrand function
f <- function(i,h) {
x <- h*(i-0.5)
return (4/(1 + x*x))
}
#the worker function
workerpi <- function()
{
#initialize
rank <- mpi.comm.rank()
size <- mpi.comm.size()
n<-0
while (TRUE)
{
#read number of intervals
if (rank==0) {
cat("Enter the number of intervals: (0 quits) ")
fp<-file("stdin"); n<-scan(fp,nmax=1); close(fp)
}
#broadcat the intervals
n <- mpi.bcast(as.integer(n),type=1)
if(n<=0) break
#run the calculation
n <- max(n,size)
h <- 1.0/n
i <- seq(rank+1,n,size);
mypi <- h*sum(sapply(i,f,h));
pi3 <- mpi.reduce(mypi)
#print results
if (rank==0) cat(sprintf("Value of PI %16.14f, diff= %16.14fn",pi3,pi3-pi))
}
}
#main
library(Rmpi)
cat("Enter the number of slaves: ")
fp<-file("stdin"); ns<-scan(fp,nmax=1); close(fp)
mpi.spawn.Rslaves(nslaves=ns)
mpi.bcast.Robj2slave(f)
mpi.bcast.Robj2slave(workerpi)
mpi.bcast.cmd(workerpi())
workerpi()
mpi.quit()
```
The above example is the dynamic MPI example for calculating the number π. Both master and slave processes carry out the calculation. Note the mpi.spawn.Rslaves(), mpi.bcast.Robj2slave()** and the mpi.bcast.cmd()** function calls.
Execute the example as:
```bash
$ R --slave --no-save --no-restore -f pi3Rslaves.R
```
### mpi.apply Rmpi
mpi.apply is a specific way of executing Dynamic Rmpi programs.
!!! Note "Note"
mpi.apply() family of functions provide MPI parallelized, drop in replacement for the serial apply() family of functions.
Execution is identical to other dynamic Rmpi programs.
mpi.apply Rmpi example:
```bash
#integrand function
f <- function(i,h) {
x <- h*(i-0.5)
return (4/(1 + x*x))
}
#the worker function
workerpi <- function(rank,size,n)
{
#run the calculation
n <- max(n,size)
h <- 1.0/n
i <- seq(rank,n,size);
mypi <- h*sum(sapply(i,f,h));
return(mypi)
}
#main
library(Rmpi)
cat("Enter the number of slaves: ")
fp<-file("stdin"); ns<-scan(fp,nmax=1); close(fp)
mpi.spawn.Rslaves(nslaves=ns)
mpi.bcast.Robj2slave(f)
mpi.bcast.Robj2slave(workerpi)
while (TRUE)
{
#read number of intervals
cat("Enter the number of intervals: (0 quits) ")
fp<-file("stdin"); n<-scan(fp,nmax=1); close(fp)
if(n<=0) break
#run workerpi
i=seq(1,2*ns)
pi3=sum(mpi.parSapply(i,workerpi,2*ns,n))
#print results
cat(sprintf("Value of PI %16.14f, diff= %16.14fn",pi3,pi3-pi))
}
mpi.quit()
```
The above is the mpi.apply MPI example for calculating the number π. Only the slave processes carry out the calculation. Note the **mpi.parSapply()**, function call. The package parallel [example](r/#package-parallel)[above](r/#package-parallel) may be trivially adapted (for much better performance) to this structure using the mclapply() in place of mpi.parSapply().
Execute the example as:
```bash
$ R --slave --no-save --no-restore -f pi3parSapply.R
```
## Combining parallel and Rmpi
Currently, the two packages can not be combined for hybrid calculations.
## Parallel execution
The R parallel jobs are executed via the PBS queue system exactly as any other parallel jobs. User must create an appropriate jobscript and submit via the **qsub**
Example jobscript for [static Rmpi](r/#static-rmpi) parallel R execution, running 1 process per core:
```bash
#!/bin/bash
#PBS -q qprod
#PBS -N Rjob
#PBS -l select=100:ncpus=16:mpiprocs=16:ompthreads=1
# change to scratch directory
SCRDIR=/scratch/$USER/myjob
cd $SCRDIR || exit
# copy input file to scratch
cp $PBS_O_WORKDIR/rscript.R .
# load R and openmpi module
module load R
module load openmpi
# execute the calculation
mpiexec -bycore -bind-to-core R --slave --no-save --no-restore -f rscript.R
# copy output file to home
cp routput.out $PBS_O_WORKDIR/.
#exit
exit
```
For more information about jobscript and MPI execution refer to the [Job submission](../../resource-allocation-and-job-execution/job-submission-and-execution/) and general [MPI](../mpi/mpi/) sections.
docs.it4i/anselm-cluster-documentation/software/numerical-libraries/fftw.md 0 → 100644
View file @ 14676d74
# FFTW
The discrete Fourier transform in one or more dimensions, MPI parallel
FFTW is a C subroutine library for computing the discrete Fourier transform in one or more dimensions, of arbitrary input size, and of both real and complex data (as well as of even/odd data, e.g. the discrete cosine/sine transforms or DCT/DST). The FFTW library allows for MPI parallel, in-place discrete Fourier transform, with data distributed over number of nodes.
Two versions, **3.3.3** and **2.1.5** of FFTW are available on Anselm, each compiled for **Intel MPI** and **OpenMPI** using **intel** and **gnu** compilers. These are available via modules:
| Version | Parallelization | module | linker options |
| -------------- | --------------- | ------------------- | ----------------------------------- |
| FFTW3 gcc3.3.3 | pthread, OpenMP | fftw3/3.3.3-gcc | -lfftw3, -lfftw3_threads-lfftw3_omp |
| FFTW3 icc3.3.3 | pthread, OpenMP | fftw3 | -lfftw3, -lfftw3_threads-lfftw3_omp |
| FFTW2 gcc2.1.5 | pthread | fftw2/2.1.5-gcc | -lfftw, -lfftw_threads |
| FFTW2 icc2.1.5 | pthread | fftw2 | -lfftw, -lfftw_threads |
| FFTW3 gcc3.3.3 | OpenMPI | fftw-mpi3/3.3.3-gcc | -lfftw3_mpi |
| FFTW3 icc3.3.3 | Intel MPI | fftw3-mpi | -lfftw3_mpi |
| FFTW2 gcc2.1.5 | OpenMPI | fftw2-mpi/2.1.5-gcc | -lfftw_mpi |
| FFTW2 gcc2.1.5 | IntelMPI | fftw2-mpi/2.1.5-gcc | -lfftw_mpi |
```bash
$ module load fftw3
```
The module sets up environment variables, required for linking and running FFTW enabled applications. Make sure that the choice of FFTW module is consistent with your choice of MPI library. Mixing MPI of different implementations may have unpredictable results.
## Example
```cpp
#include <fftw3-mpi.h>
int main(int argc, char **argv)
{
const ptrdiff_t N0 = 100, N1 = 1000;
fftw_plan plan;
fftw_complex *data;
ptrdiff_t alloc_local, local_n0, local_0_start, i, j;
MPI_Init(&argc, &argv);
fftw_mpi_init();
/* get local data size and allocate */
alloc_local = fftw_mpi_local_size_2d(N0, N1, MPI_COMM_WORLD,
&local_n0, &local_0_start);
data = fftw_alloc_complex(alloc_local);
/* create plan for in-place forward DFT */
plan = fftw_mpi_plan_dft_2d(N0, N1, data, data, MPI_COMM_WORLD,
FFTW_FORWARD, FFTW_ESTIMATE);
/* initialize data */
for (i = 0; i < local_n0; ++i) for (j = 0; j < N1; ++j)
{ data[i*N1 + j][0] = i;
data[i*N1 + j][1] = j; }
/* compute transforms, in-place, as many times as desired */
fftw_execute(plan);
fftw_destroy_plan(plan);
MPI_Finalize();
}
```
Load modules and compile:
```bash
$ module load impi intel
$ module load fftw3-mpi
$ mpicc testfftw3mpi.c -o testfftw3mpi.x -Wl,-rpath=$LIBRARY_PATH -lfftw3_mpi
```
Run the example as [Intel MPI program](../mpi/running-mpich2/).
Read more on FFTW usage on the [FFTW website.](http://www.fftw.org/fftw3_doc/)
docs.it4i/anselm-cluster-documentation/software/numerical-libraries/gsl.md 0 → 100644
View file @ 14676d74
# GSL
The GNU Scientific Library. Provides a wide range of mathematical routines.
## Introduction
The GNU Scientific Library (GSL) provides a wide range of mathematical routines such as random number generators, special functions and least-squares fitting. There are over 1000 functions in total. The routines have been written from scratch in C, and present a modern Applications Programming Interface (API) for C programmers, allowing wrappers to be written for very high level languages.
The library covers a wide range of topics in numerical computing. Routines are available for the following areas:
Complex Numbers Roots of Polynomials
Special Functions Vectors and Matrices
Permutations Combinations
Sorting BLAS Support
Linear Algebra CBLAS Library
Fast Fourier Transforms Eigensystems
Random Numbers Quadrature
Random Distributions Quasi-Random Sequences
Histograms Statistics
Monte Carlo Integration N-Tuples
Differential Equations Simulated Annealing
Numerical Differentiation Interpolation
Series Acceleration Chebyshev Approximations
Root-Finding Discrete Hankel Transforms
Least-Squares Fitting Minimization
IEEE Floating-Point Physical Constants
Basis Splines Wavelets
## Modules
The GSL 1.16 is available on Anselm, compiled for GNU and Intel compiler. These variants are available via modules:
| Module | Compiler |
| --------------------- | --------- |
| gsl/1.16-gcc | gcc 4.8.6 |
| gsl/1.16-icc(default) | icc |
```bash
$ module load gsl
```
The module sets up environment variables, required for linking and running GSL enabled applications. This particular command loads the default module, which is gsl/1.16-icc
## Linking
Load an appropriate gsl module. Link using **-lgsl** switch to link your code against GSL. The GSL depends on cblas API to BLAS library, which must be supplied for linking. The BLAS may be provided, for example from the MKL library, as well as from the BLAS GSL library (-lgslcblas). Using the MKL is recommended.
### Compiling and linking with Intel compilers
```bash
$ module load intel
$ module load gsl
$ icc myprog.c -o myprog.x -Wl,-rpath=$LIBRARY_PATH -mkl -lgsl
```
### Compiling and linking with GNU compilers
```bash
$ module load gcc
$ module load mkl
$ module load gsl/1.16-gcc
$ gcc myprog.c -o myprog.x -Wl,-rpath=$LIBRARY_PATH -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_core -lgomp -lgsl
```
## Example
Following is an example of discrete wavelet transform implemented by GSL:
```cpp
#include <stdio.h>
#include <math.h>
#include <gsl/gsl_sort.h>
#include <gsl/gsl_wavelet.h>
int
main (int argc, char **argv)
{
int i, n = 256, nc = 20;
double *data = malloc (n * sizeof (double));
double *abscoeff = malloc (n * sizeof (double));
size_t *p = malloc (n * sizeof (size_t));
gsl_wavelet *w;
gsl_wavelet_workspace *work;
w = gsl_wavelet_alloc (gsl_wavelet_daubechies, 4);
work = gsl_wavelet_workspace_alloc (n);
for (i=0; i<n; i++)
data[i] = sin (3.141592654*(double)i/256.0);
gsl_wavelet_transform_forward (w, data, 1, n, work);
for (i = 0; i < n; i++)
{
abscoeff[i] = fabs (data[i]);
}
gsl_sort_index (p, abscoeff, 1, n);
for (i = 0; (i + nc) < n; i++)
data[p[i]] = 0;
gsl_wavelet_transform_inverse (w, data, 1, n, work);
for (i = 0; i < n; i++)
{
printf ("%gn", data[i]);
}
gsl_wavelet_free (w);
gsl_wavelet_workspace_free (work);
free (data);
free (abscoeff);
free (p);
return 0;
}
```
Load modules and compile:
```bash
$ module load intel gsl
icc dwt.c -o dwt.x -Wl,-rpath=$LIBRARY_PATH -mkl -lgsl
```
In this example, we compile the dwt.c code using the Intel compiler and link it to the MKL and GSL library, note the -mkl and -lgsl options. The library search path is compiled in, so that no modules are necessary to run the code.
docs.it4i/anselm-cluster-documentation/software/numerical-libraries/hdf5.md 0 → 100644
View file @ 14676d74
# HDF5
Hierarchical Data Format library. Serial and MPI parallel version.
[HDF5 (Hierarchical Data Format)](http://www.hdfgroup.org/HDF5/) is a general purpose library and file format for storing scientific data. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic objects, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids. You can also mix and match them in HDF5 files according to your needs.
Versions **1.8.11** and **1.8.13** of HDF5 library are available on Anselm, compiled for **Intel MPI** and **OpenMPI** using **intel** and **gnu** compilers. These are available via modules:
| Version | Parallelization | module | C linker options | C++ linker options | Fortran linker options |
| --------------------- | --------------------------------- | -------------------------- | --------------------- | ----------------------- | ----------------------- |
| HDF5 icc serial | pthread | hdf5/1.8.11 | $HDF5_INC $HDF5_SHLIB | $HDF5_INC $HDF5_CPP_LIB | $HDF5_INC $HDF5_F90_LIB |
| HDF5 icc parallel MPI | pthread, IntelMPI | hdf5-parallel/1.8.11 | $HDF5_INC $HDF5_SHLIB | Not supported | $HDF5_INC $HDF5_F90_LIB |
| HDF5 icc serial | pthread | hdf5/1.8.13 | $HDF5_INC $HDF5_SHLIB | $HDF5_INC $HDF5_CPP_LIB | $HDF5_INC $HDF5_F90_LIB |
| HDF5 icc parallel MPI | pthread, IntelMPI | hdf5-parallel/1.8.13 | $HDF5_INC $HDF5_SHLIB | Not supported | $HDF5_INC $HDF5_F90_LIB |
| HDF5 gcc parallel MPI | pthread, OpenMPI 1.6.5, gcc 4.8.1 | hdf5-parallel/1.8.11-gcc | $HDF5_INC $HDF5_SHLIB | Not supported | $HDF5_INC $HDF5_F90_LIB |
| HDF5 gcc parallel MPI | pthread, OpenMPI 1.6.5, gcc 4.8.1 | hdf5-parallel/1.8.13-gcc | $HDF5_INC $HDF5_SHLIB | Not supported | $HDF5_INC $HDF5_F90_LIB |
| HDF5 gcc parallel MPI | pthread, OpenMPI 1.8.1, gcc 4.9.0 | hdf5-parallel/1.8.13-gcc49 | $HDF5_INC $HDF5_SHLIB | Not supported | $HDF5_INC $HDF5_F90_LIB |
```bash
$ module load hdf5-parallel
```
The module sets up environment variables, required for linking and running HDF5 enabled applications. Make sure that the choice of HDF5 module is consistent with your choice of MPI library. Mixing MPI of different implementations may have unpredictable results.
!!! Note "Note"
Be aware, that GCC version of **HDF5 1.8.11** has serious performance issues, since it's compiled with -O0 optimization flag. This version is provided only for testing of code compiled only by GCC and IS NOT recommended for production computations. For more information, please see: <http://www.hdfgroup.org/ftp/HDF5/prev-releases/ReleaseFiles/release5-1811>
All GCC versions of **HDF5 1.8.13** are not affected by the bug, are compiled with -O3 optimizations and are recommended for production computations.
## Example
```cpp
#include "hdf5.h"
#define FILE "dset.h5"
int main() {
hid_t file_id, dataset_id, dataspace_id; /* identifiers */
hsize_t dims[2];
herr_t status;
int i, j, dset_data[4][6];
/* Create a new file using default properties. */
file_id = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
/* Create the data space for the dataset. */
dims[0] = 4;
dims[1] = 6;
dataspace_id = H5Screate_simple(2, dims, NULL);
/* Initialize the dataset. */
for (i = 0; i < 4; i++)
for (j = 0; j < 6; j++)
dset_data[i][j] = i * 6 + j + 1;
/* Create the dataset. */
dataset_id = H5Dcreate2(file_id, "/dset", H5T_STD_I32BE, dataspace_id,
H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
/* Write the dataset. */
status = H5Dwrite(dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT,
dset_data);
status = H5Dread(dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT,
dset_data);
/* End access to the dataset and release resources used by it. */
status = H5Dclose(dataset_id);
/* Terminate access to the data space. */
status = H5Sclose(dataspace_id);
/* Close the file. */
status = H5Fclose(file_id);
}
```
Load modules and compile:
```bash
$ module load intel impi
$ module load hdf5-parallel
$ mpicc hdf5test.c -o hdf5test.x -Wl,-rpath=$LIBRARY_PATH $HDF5_INC $HDF5_SHLIB
```
Run the example as [Intel MPI program](../anselm-cluster-documentation/software/mpi/running-mpich2/).
For further information, please see the website: <http://www.hdfgroup.org/HDF5/>
docs.it4i/anselm-cluster-documentation/software/numerical-libraries/intel-numerical-libraries.md 0 → 100644
View file @ 14676d74
# Intel numerical libraries
Intel libraries for high performance in numerical computing
## Intel Math Kernel Library
Intel Math Kernel Library (Intel MKL) is a library of math kernel subroutines, extensively threaded and optimized for maximum performance. Intel MKL unites and provides these basic components: BLAS, LAPACK, ScaLapack, PARDISO, FFT, VML, VSL, Data fitting, Feast Eigensolver and many more.
```bash
$ module load mkl
```
Read more at the [Intel MKL](../intel-suite/intel-mkl/) page.
## Intel Integrated Performance Primitives
Intel Integrated Performance Primitives, version 7.1.1, compiled for AVX is available, via module ipp. The IPP is a library of highly optimized algorithmic building blocks for media and data applications. This includes signal, image and frame processing algorithms, such as FFT, FIR, Convolution, Optical Flow, Hough transform, Sum, MinMax and many more.
```bash
$ module load ipp
```
Read more at the [Intel IPP](../intel-suite/intel-integrated-performance-primitives/) page.
## Intel Threading Building Blocks
Intel Threading Building Blocks (Intel TBB) is a library that supports scalable parallel programming using standard ISO C++ code. It does not require special languages or compilers. It is designed to promote scalable data parallel programming. Additionally, it fully supports nested parallelism, so you can build larger parallel components from smaller parallel components. To use the library, you specify tasks, not threads, and let the library map tasks onto threads in an efficient manner.
```bash
$ module load tbb
```
Read more at the [Intel TBB](../intel-suite/intel-tbb/) page.
docs.it4i/anselm-cluster-documentation/software/numerical-libraries/magma-for-intel-xeon-phi.md 0 → 100644
View file @ 14676d74
# MAGMA for Intel Xeon Phi
Next generation dense algebra library for heterogeneous systems with accelerators
### Compiling and linking with MAGMA
To be able to compile and link code with MAGMA library user has to load following module:
```bash
$ module load magma/1.3.0-mic
```
To make compilation more user friendly module also sets these two environment variables:
!!! Note "Note"
MAGMA_INC - contains paths to the MAGMA header files (to be used for compilation step)
!!! Note "Note"
MAGMA_LIBS - contains paths to MAGMA libraries (to be used for linking step).
Compilation example:
```bash
$ icc -mkl -O3 -DHAVE_MIC -DADD_ -Wall $MAGMA_INC -c testing_dgetrf_mic.cpp -o testing_dgetrf_mic.o
$ icc -mkl -O3 -DHAVE_MIC -DADD_ -Wall -fPIC -Xlinker -zmuldefs -Wall -DNOCHANGE -DHOST testing_dgetrf_mic.o -o testing_dgetrf_mic $MAGMA_LIBS
```
### Running MAGMA code
MAGMA implementation for Intel MIC requires a MAGMA server running on accelerator prior to executing the user application. The server can be started and stopped using following scripts:
!!! Note "Note"
To start MAGMA server use:
**$MAGMAROOT/start_magma_server**
!!! Note "Note"
To stop the server use:
**$MAGMAROOT/stop_magma_server**
!!! Note "Note"
For deeper understanding how the MAGMA server is started, see the following script:
**$MAGMAROOT/launch_anselm_from_mic.sh**
To test if the MAGMA server runs properly we can run one of examples that are part of the MAGMA installation:
```bash
[user@cn204 ~]$ $MAGMAROOT/testing/testing_dgetrf_mic
[user@cn204 ~]$ export OMP_NUM_THREADS=16
[lriha@cn204 ~]$ $MAGMAROOT/testing/testing_dgetrf_mic
Usage: /apps/libs/magma-mic/magmamic-1.3.0/testing/testing_dgetrf_mic [options] [-h|--help]
M N CPU GFlop/s (sec) MAGMA GFlop/s (sec) ||PA-LU||/(||A||*N)
=========================================================================
1088 1088 --- ( --- ) 13.93 ( 0.06) ---
2112 2112 --- ( --- ) 77.85 ( 0.08) ---
3136 3136 --- ( --- ) 183.21 ( 0.11) ---
4160 4160 --- ( --- ) 227.52 ( 0.21) ---
5184 5184 --- ( --- ) 258.61 ( 0.36) ---
6208 6208 --- ( --- ) 333.12 ( 0.48) ---
7232 7232 --- ( --- ) 416.52 ( 0.61) ---
8256 8256 --- ( --- ) 446.97 ( 0.84) ---
9280 9280 --- ( --- ) 461.15 ( 1.16) ---
10304 10304 --- ( --- ) 500.70 ( 1.46) ---
```
!!! Note "Note"
Please note: MAGMA contains several benchmarks and examples that can be found in:
**$MAGMAROOT/testing/**
!!! Note "Note"
MAGMA relies on the performance of all CPU cores as well as on the performance of the accelerator. Therefore on Anselm number of CPU OpenMP threads has to be set to 16:
**export OMP_NUM_THREADS=16**
See more details at [MAGMA home page](http://icl.cs.utk.edu/magma/).
## References
[1] MAGMA MIC: Linear Algebra Library for Intel Xeon Phi Coprocessors, Jack Dongarra et. al, <http://icl.utk.edu/projectsfiles/magma/pubs/24-MAGMA_MIC_03.pdf>
docs.it4i/anselm-cluster-documentation/software/numerical-libraries/petsc.md 0 → 100644
View file @ 14676d74
# PETSc
PETSc is a suite of building blocks for the scalable solution of scientific and engineering applications modeled by partial differential equations. It supports MPI, shared memory, and GPU through CUDA or OpenCL, as well as hybrid MPI-shared memory or MPI-GPU parallelism.
## Introduction
PETSc (Portable, Extensible Toolkit for Scientific Computation) is a suite of building blocks (data structures and routines) for the scalable solution of scientific and engineering applications modelled by partial differential equations. It allows thinking in terms of high-level objects (matrices) instead of low-level objects (raw arrays). Written in C language but can also be called from FORTRAN, C++, Python and Java codes. It supports MPI, shared memory, and GPUs through CUDA or OpenCL, as well as hybrid MPI-shared memory or MPI-GPU parallelism.
## Resources
- [project webpage](http://www.mcs.anl.gov/petsc/)
- [documentation](http://www.mcs.anl.gov/petsc/documentation/)
- [PETSc Users Manual (PDF)](http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf)
- [index of all manual pages](http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/singleindex.html)
- PRACE Video Tutorial [part1](http://www.youtube.com/watch?v=asVaFg1NDqY), [part2](http://www.youtube.com/watch?v=ubp_cSibb9I), [part3](http://www.youtube.com/watch?v=vJAAAQv-aaw), [part4](http://www.youtube.com/watch?v=BKVlqWNh8jY), [part5](http://www.youtube.com/watch?v=iXkbLEBFjlM)
## Modules
You can start using PETSc on Anselm by loading the PETSc module. Module names obey this pattern:
```bash
# module load petsc/version-compiler-mpi-blas-variant, e.g.
module load petsc/3.4.4-icc-impi-mkl-opt
```
where `variant` is replaced by one of `{dbg, opt, threads-dbg, threads-opt}`. The `opt` variant is compiled without debugging information (no `-g` option) and with aggressive compiler optimizations (`-O3 -xAVX`). This variant is suitable for performance measurements and production runs. In all other cases use the debug (`dbg`) variant, because it contains debugging information, performs validations and self-checks, and provides a clear stack trace and message in case of an error. The other two variants `threads-dbg` and `threads-opt` are `dbg` and `opt`, respectively, built with [OpenMP and pthreads threading support](https://www.mcs.anl.gov/petsc/miscellaneous/petscthreads.html).
## External libraries
PETSc needs at least MPI, BLAS and LAPACK. These dependencies are currently satisfied with Intel MPI and Intel MKL in Anselm `petsc` modules.
PETSc can be linked with a plethora of [external numerical libraries](http://www.mcs.anl.gov/petsc/miscellaneous/external.html), extending PETSc functionality, e.g. direct linear system solvers, preconditioners or partitioners. See below a list of libraries currently included in Anselm `petsc` modules.
All these libraries can be used also alone, without PETSc. Their static or shared program libraries are available in
`$PETSC_DIR/$PETSC_ARCH/lib` and header files in `$PETSC_DIR/$PETSC_ARCH/include`. `PETSC_DIR` and `PETSC_ARCH` are environment variables pointing to a specific PETSc instance based on the petsc module loaded.
### Libraries linked to PETSc on Anselm (as of 11 April 2015)
- dense linear algebra
- [Elemental](http://libelemental.org/)
- sparse linear system solvers
- [Intel MKL Pardiso](https://software.intel.com/en-us/node/470282)
- [MUMPS](http://mumps.enseeiht.fr/)
- [PaStiX](http://pastix.gforge.inria.fr/)
- [SuiteSparse](http://faculty.cse.tamu.edu/davis/suitesparse.html)
- [SuperLU](http://crd.lbl.gov/~xiaoye/SuperLU/#superlu)
- [SuperLU_Dist](http://crd.lbl.gov/~xiaoye/SuperLU/#superlu_dist)
- input/output
- [ExodusII](http://sourceforge.net/projects/exodusii/)
- [HDF5](http://www.hdfgroup.org/HDF5/)
- [NetCDF](http://www.unidata.ucar.edu/software/netcdf/)
- partitioning
- [Chaco](http://www.cs.sandia.gov/CRF/chac.html)
- [METIS](http://glaros.dtc.umn.edu/gkhome/metis/metis/overview)
- [ParMETIS](http://glaros.dtc.umn.edu/gkhome/metis/parmetis/overview)
- [PT-Scotch](http://www.labri.fr/perso/pelegrin/scotch/)
- preconditioners & multigrid
- [Hypre](http://www.nersc.gov/users/software/programming-libraries/math-libraries/petsc/)
- [Trilinos ML](http://trilinos.sandia.gov/packages/ml/)
- [SPAI - Sparse Approximate Inverse](https://bitbucket.org/petsc/pkg-spai)
docs.it4i/anselm-cluster-documentation/software/numerical-libraries/trilinos.md 0 → 100644
View file @ 14676d74
# Trilinos
Packages for large scale scientific and engineering problems. Provides MPI and hybrid parallelization.
### Introduction
Trilinos is a collection of software packages for the numerical solution of large scale scientific and engineering problems. It is based on C++ and features modern object-oriented design. Both serial as well as parallel computations based on MPI and hybrid parallelization are supported within Trilinos packages.
### Installed packages
Current Trilinos installation on ANSELM contains (among others) the following main packages
- **Epetra** - core linear algebra package containing classes for manipulation with serial and distributed vectors, matrices, and graphs. Dense linear solvers are supported via interface to BLAS and LAPACK (Intel MKL on ANSELM). Its extension **EpetraExt** contains e.g. methods for matrix-matrix multiplication.
- **Tpetra** - next-generation linear algebra package. Supports 64-bit indexing and arbitrary data type using C++ templates.
- **Belos** - library of various iterative solvers (CG, block CG, GMRES, block GMRES etc.).
- **Amesos** - interface to direct sparse solvers.
- **Anasazi** - framework for large-scale eigenvalue algorithms.
- **IFPACK** - distributed algebraic preconditioner (includes e.g. incomplete LU factorization)
- **Teuchos** - common tools packages. This package contains classes for memory management, output, performance monitoring, BLAS and LAPACK wrappers etc.
For the full list of Trilinos packages, descriptions of their capabilities, and user manuals see [http://trilinos.sandia.gov.](http://trilinos.sandia.gov)
### Installed version
Currently, Trilinos in version 11.2.3 compiled with Intel Compiler is installed on ANSELM.
### Compiling against Trilinos
First, load the appropriate module:
```bash
$ module load trilinos
```
For the compilation of CMake-aware project, Trilinos provides the FIND_PACKAGE( Trilinos ) capability, which makes it easy to build against Trilinos, including linking against the correct list of libraries. For details, see <http://trilinos.sandia.gov/Finding_Trilinos.txt>
For compiling using simple makefiles, Trilinos provides Makefile.export system, which allows users to include important Trilinos variables directly into their makefiles. This can be done simply by inserting the following line into the makefile:
```bash
include Makefile.export.Trilinos
```
or
```bash
include Makefile.export.<package>
```
if you are interested only in a specific Trilinos package. This will give you access to the variables such as Trilinos_CXX_COMPILER, Trilinos_INCLUDE_DIRS, Trilinos_LIBRARY_DIRS etc. For the detailed description and example makefile see <http://trilinos.sandia.gov/Export_Makefile.txt>.
docs.it4i/anselm-cluster-documentation/software/nvidia-cuda.md 0 → 100644
View file @ 14676d74
# NVIDIA CUDA
\##A guide to NVIDIA CUDA programming and GPU usage
## CUDA Programming on Anselm
The default programming model for GPU accelerators on Anselm is Nvidia CUDA. To set up the environment for CUDA use
```bash
$ module load cuda
```
If the user code is hybrid and uses both CUDA and MPI, the MPI environment has to be set up as well. One way to do this is to use the PrgEnv-gnu module, which sets up correct combination of GNU compiler and MPI library.
```bash
$ module load PrgEnv-gnu
```
CUDA code can be compiled directly on login1 or login2 nodes. User does not have to use compute nodes with GPU accelerator for compilation. To compile a CUDA source code, use nvcc compiler.
```bash
$ nvcc --version
```
CUDA Toolkit comes with large number of examples, that can be helpful to start with. To compile and test these examples user should copy them to its home directory
```bash
$ cd ~
$ mkdir cuda-samples
$ cp -R /apps/nvidia/cuda/6.5.14/samples/* ~/cuda-samples/
```
To compile an examples, change directory to the particular example (here the example used is deviceQuery) and run "make" to start the compilation
```bash
$ cd ~/cuda-samples/1_Utilities/deviceQuery
$ make
```
To run the code user can use PBS interactive session to get access to a node from qnvidia queue (note: use your project name with parameter -A in the qsub command) and execute the binary file
```bash
$ qsub -I -q qnvidia -A OPEN-0-0
$ module load cuda
$ ~/cuda-samples/1_Utilities/deviceQuery/deviceQuery
```
Expected output of the deviceQuery example executed on a node with Tesla K20m is
```bash
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "Tesla K20m"
CUDA Driver Version / Runtime Version 5.0 / 5.0
CUDA Capability Major/Minor version number: 3.5
Total amount of global memory: 4800 MBytes (5032706048 bytes)
(13) Multiprocessors x (192) CUDA Cores/MP: 2496 CUDA Cores
GPU Clock rate: 706 MHz (0.71 GHz)
Memory Clock rate: 2600 Mhz
Memory Bus Width: 320-bit
L2 Cache Size: 1310720 bytes
Max Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536,65536), 3D=(4096,4096,4096)
Max Layered Texture Size (dim) x layers 1D=(16384) x 2048, 2D=(16384,16384) x 2048
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Maximum sizes of each dimension of a block: 1024 x 1024 x 64
Maximum sizes of each dimension of a grid: 2147483647 x 65535 x 65535
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 2 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Enabled
Device supports Unified Addressing (UVA): Yes
Device PCI Bus ID / PCI location ID: 2 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 5.0, CUDA Runtime Version = 5.0, NumDevs = 1, Device0 = Tesla K20m
```
### Code example
In this section we provide a basic CUDA based vector addition code example. You can directly copy and paste the code to test it.
```bash
$ vim test.cu
#define N (2048*2048)
#define THREADS_PER_BLOCK 512
#include <stdio.h>
#include <stdlib.h>
// GPU kernel function to add two vectors
__global__ void add_gpu( int *a, int *b, int *c, int n){
int index = threadIdx.x + blockIdx.x * blockDim.x;
if (index < n)
c[index] = a[index] + b[index];
}
// CPU function to add two vectors
void add_cpu (int *a, int *b, int *c, int n) {
for (int i=0; i < n; i++)
c[i] = a[i] + b[i];
}
// CPU function to generate a vector of random integers
void random_ints (int *a, int n) {
for (int i = 0; i < n; i++)
a[i] = rand() % 10000; // random number between 0 and 9999
}
// CPU function to compare two vectors
int compare_ints( int *a, int *b, int n ){
int pass = 0;
for (int i = 0; i < N; i++){
if (a[i] != b[i]) {
printf("Value mismatch at location %d, values %d and %dn",i, a[i], b[i]);
pass = 1;
}
}
if (pass == 0) printf ("Test passedn"); else printf ("Test Failedn");
return pass;
}
int main( void ) {
int *a, *b, *c; // host copies of a, b, c
int *dev_a, *dev_b, *dev_c; // device copies of a, b, c
int size = N * sizeof( int ); // we need space for N integers
// Allocate GPU/device copies of dev_a, dev_b, dev_c
cudaMalloc( (void**)&dev_a, size );
cudaMalloc( (void**)&dev_b, size );
cudaMalloc( (void**)&dev_c, size );
// Allocate CPU/host copies of a, b, c
a = (int*)malloc( size );
b = (int*)malloc( size );
c = (int*)malloc( size );
// Fill input vectors with random integer numbers
random_ints( a, N );
random_ints( b, N );
// copy inputs to device
cudaMemcpy( dev_a, a, size, cudaMemcpyHostToDevice );
cudaMemcpy( dev_b, b, size, cudaMemcpyHostToDevice );
// launch add_gpu() kernel with blocks and threads
add_gpu<<< N/THREADS_PER_BLOCK, THREADS_PER_BLOCK >>( dev_a, dev_b, dev_c, N );
// copy device result back to host copy of c
cudaMemcpy( c, dev_c, size, cudaMemcpyDeviceToHost );
//Check the results with CPU implementation
int *c_h; c_h = (int*)malloc( size );
add_cpu (a, b, c_h, N);
compare_ints(c, c_h, N);
// Clean CPU memory allocations
free( a ); free( b ); free( c ); free (c_h);
// Clean GPU memory allocations
cudaFree( dev_a );
cudaFree( dev_b );
cudaFree( dev_c );
return 0;
}
```
This code can be compiled using following command
```bash
$ nvcc test.cu -o test_cuda
```
To run the code use interactive PBS session to get access to one of the GPU accelerated nodes
```bash
$ qsub -I -q qnvidia -A OPEN-0-0
$ module load cuda
$ ./test.cuda
```
## CUDA Libraries
### cuBLAS
The NVIDIA CUDA Basic Linear Algebra Subroutines (cuBLAS) library is a GPU-accelerated version of the complete standard BLAS library with 152 standard BLAS routines. Basic description of the library together with basic performance comparison with MKL can be found [here](https://developer.nvidia.com/cublas "Nvidia cuBLAS").
**cuBLAS example: SAXPY**
SAXPY function multiplies the vector x by the scalar alpha and adds it to the vector y overwriting the latest vector with the result. The description of the cuBLAS function can be found in [NVIDIA CUDA documentation](http://docs.nvidia.com/cuda/cublas/index.html#cublas-lt-t-gt-axpy "Nvidia CUDA documentation "). Code can be pasted in the file and compiled without any modification.
```cpp
/* Includes, system */
#include <stdio.h>
#include <stdlib.h>
/* Includes, cuda */
#include <cuda_runtime.h>
#include <cublas_v2.h>
/* Vector size */
#define N (32)
/* Host implementation of a simple version of saxpi */
void saxpy(int n, float alpha, const float *x, float *y)
{
for (int i = 0; i < n; ++i)
y[i] = alpha*x[i] + y[i];
}
/* Main */
int main(int argc, char **argv)
{
float *h_X, *h_Y, *h_Y_ref;
float *d_X = 0;
float *d_Y = 0;
const float alpha = 1.0f;
int i;
cublasHandle_t handle;
/* Initialize CUBLAS */
printf("simpleCUBLAS test running..n");
cublasCreate(&handle);
/* Allocate host memory for the matrices */
h_X = (float *)malloc(N * sizeof(h_X[0]));
h_Y = (float *)malloc(N * sizeof(h_Y[0]));
h_Y_ref = (float *)malloc(N * sizeof(h_Y_ref[0]));
/* Fill the matrices with test data */
for (i = 0; i < N; i++)
{
h_X[i] = rand() / (float)RAND_MAX;
h_Y[i] = rand() / (float)RAND_MAX;
h_Y_ref[i] = h_Y[i];
}
/* Allocate device memory for the matrices */
cudaMalloc((void **)&d_X, N * sizeof(d_X[0]));
cudaMalloc((void **)&d_Y, N * sizeof(d_Y[0]));
/* Initialize the device matrices with the host matrices */
cublasSetVector(N, sizeof(h_X[0]), h_X, 1, d_X, 1);
cublasSetVector(N, sizeof(h_Y[0]), h_Y, 1, d_Y, 1);
/* Performs operation using plain C code */
saxpy(N, alpha, h_X, h_Y_ref);
/* Performs operation using cublas */
cublasSaxpy(handle, N, &alpha, d_X, 1, d_Y, 1);
/* Read the result back */
cublasGetVector(N, sizeof(h_Y[0]), d_Y, 1, h_Y, 1);
/* Check result against reference */
for (i = 0; i < N; ++i)
printf("CPU res = %f t GPU res = %f t diff = %f n", h_Y_ref[i], h_Y[i], h_Y_ref[i] - h_Y[i]);
/* Memory clean up */
free(h_X); free(h_Y); free(h_Y_ref);
cudaFree(d_X); cudaFree(d_Y);
/* Shutdown */
cublasDestroy(handle);
}
```
!!! Note "Note"
Please note: cuBLAS has its own function for data transfers between CPU and GPU memory:
- [cublasSetVector](http://docs.nvidia.com/cuda/cublas/index.html#cublassetvector) - transfers data from CPU to GPU memory
- [cublasGetVector](http://docs.nvidia.com/cuda/cublas/index.html#cublasgetvector) - transfers data from GPU to CPU memory
To compile the code using NVCC compiler a "-lcublas" compiler flag has to be specified:
```bash
$ module load cuda
$ nvcc -lcublas test_cublas.cu -o test_cublas_nvcc
```
To compile the same code with GCC:
```bash
$ module load cuda
$ gcc -std=c99 test_cublas.c -o test_cublas_icc -lcublas -lcudart
```
To compile the same code with Intel compiler:
```bash
$ module load cuda intel
$ icc -std=c99 test_cublas.c -o test_cublas_icc -lcublas -lcudart
```
docs.it4i/anselm-cluster-documentation/software/omics-master/diagnostic-component-team.md 0 → 100644
View file @ 14676d74
# Diagnostic component (TEAM)
## Access
TEAM is available at the [following address](http://omics.it4i.cz/team/)
!!! note
The address is accessible only via VPN.
## Diagnostic component
VCF files are scanned by this diagnostic tool for known diagnostic disease-associated variants. When no diagnostic mutation is found, the file can be sent to the disease-causing gene discovery tool to see whether new disease associated variants can be found.
TEAM (27) is an intuitive and easy-to-use web tool that fills the gap between the predicted mutations and the final diagnostic in targeted enrichment sequencing analysis. The tool searches for known diagnostic mutations, corresponding to a disease panel, among the predicted patient’s variants. Diagnostic variants for the disease are taken from four databases of disease-related variants (HGMD-public, HUMSAVAR , ClinVar and COSMIC) If no primary diagnostic variant is found, then a list of secondary findings that can help to establish a diagnostic is produced. TEAM also provides with an interface for the definition of and customization of panels, by means of which, genes and mutations can be added or discarded to adjust panel definitions.
![Interface of the application. Panels for defining targeted regions of interest can be set up by just drag and drop known disease genes or disease definitions from the lists. Thus, virtual panels can be interactively improved as the knowledge of the disease increases.](../../../img/fig5.png)
** Figure 5. **Interface of the application. Panels for defining targeted regions of interest can be set up by just drag and drop known disease genes or disease definitions from the lists. Thus, virtual panels can be interactively improved as the knowledge of the disease increases.
docs.it4i/anselm-cluster-documentation/software/omics-master/overview.md 0 → 100644
View file @ 14676d74
# Overview
The human NGS data processing solution
## Introduction
The scope of this OMICS MASTER solution is restricted to human genomics research (disease causing gene discovery in whole human genome or exome) or diagnosis (panel sequencing), although it could be extended in the future to other usages.
The pipeline inputs the raw data produced by the sequencing machines and undergoes a processing procedure that consists on a quality control, the mapping and variant calling steps that result in a file containing the set of variants in the sample. From this point, the prioritization component or the diagnostic component can be launched.
![OMICS MASTER solution overview. Data is produced in the external labs and comes to IT4I (represented by the blue dashed line). The data pre-processor converts raw data into a list of variants and annotations for each sequenced patient. These lists files together with primary and secondary (alignment) data files are stored in IT4I sequence DB and uploaded to the discovery (candidate priorization) or diagnostic component where they can be analysed directly by the user that produced
them, depending of the experimental design carried out.](../../../img/fig1.png)
** Figure 1. ** OMICS MASTER solution overview. Data is produced in the external labs and comes to IT4I (represented by the blue dashed line). The data pre-processor converts raw data into a list of variants and annotations for each sequenced patient. These lists files together with primary and secondary (alignment) data files are stored in IT4I sequence DB and uploaded to the discovery (candidate prioritization) or diagnostic component where they can be analyzed directly by the user that produced them, depending of the experimental design carried out.
Typical genomics pipelines are composed by several components that need to be launched manually. The advantage of OMICS MASTER pipeline is that all these components are invoked sequentially in an automated way.
OMICS MASTER pipeline inputs a FASTQ file and outputs an enriched VCF file. This pipeline is able to queue all the jobs to PBS by only launching a process taking all the necessary input files and creates the intermediate and final folders
Let’s see each of the OMICS MASTER solution components:
## Components
### Processing
This component is composed by a set of programs that carry out quality controls, alignment, realignment, variant calling and variant annotation. It turns raw data from the sequencing machine into files containing lists of variants (VCF) that once annotated, can be used by the following components (discovery and diagnosis).
We distinguish three types of sequencing instruments: bench sequencers (MySeq, IonTorrent, and Roche Junior, although this last one is about being discontinued), which produce relatively Genomes in the clinic
low throughput (tens of million reads), and high end sequencers, which produce high throughput (hundreds of million reads) among which we have Illumina HiSeq 2000 (and new models) and SOLiD. All of them but SOLiD produce data in sequence format. SOLiD produces data in a special format called colour space that require of specific software for the mapping process. Once the mapping has been done, the rest of the pipeline is identical. Anyway, SOLiD is a technology which is also about being discontinued by the manufacturer so, this type of data will be scarce in the future.
#### Quality control, preprocessing and statistics for FASTQ
FastQC& FastQC.
These steps are carried out over the original FASTQ file with optimized scripts and includes the following steps: sequence cleansing, estimation of base quality scores, elimination of duplicates and statistics.
Input: ** FASTQ file **.
Output: ** FASTQ file plus an HTML file containing statistics on the data **.
FASTQ format It represents the nucleotide sequence and its corresponding quality scores.
![FASTQ file.](../../../img/fig2.png "fig2.png")
** Figure 2 **.FASTQ file.
#### Mapping
Component: ** Hpg-aligner **.
Sequence reads are mapped over the human reference genome. SOLiD reads are not covered by this solution; they should be mapped with specific software (among the few available options, SHRiMP seems to be the best one). For the rest of NGS machine outputs we use HPG Aligner. HPG-Aligner is an innovative solution, based on a combination of mapping with BWT and local alignment with Smith-Waterman (SW), that drastically increases mapping accuracy (97% versus 62-70% by current mappers, in the most common scenarios). This proposal provides a simple and fast solution that maps almost all the reads, even those containing a high number of mismatches or indels.
Input: ** FASTQ file **.
Output: ** Aligned file in BAM format **.
** Sequence Alignment/Map (SAM) **
It is a human readable tab-delimited format in which each read and its alignment is represented on a single line. The format can represent unmapped reads, reads that are mapped to unique locations, and reads that are mapped to multiple locations.
The SAM format (1) consists of one header section and one alignment section. The lines in the header section start with character ‘@’, and lines in the alignment section do not. All lines are TAB delimited.
In SAM, each alignment line has 11 mandatory fields and a variable number of optional fields. The mandatory fields are briefly described in Table 1. They must be present but their value can be a ‘\*’ or a zero (depending on the field) if the
corresponding information is unavailable.
| ** No. ** | ** Name ** | ** Description ** |
| --------- | ---------- | ----------------------------------------------------- |
| 1 | QNAME | Query NAME of the read or the read pai |
| 2 | FLAG | Bitwise FLAG (pairing,strand,mate strand,etc.) |
| 3 | RNAME | <p>Reference sequence NAME |
| 4 | POS | <p>1-Based leftmost POSition of clipped alignment |
| 5 | MAPQ | <p>MAPping Quality (Phred-scaled) |
| 6 | CIGAR | <p>Extended CIGAR string (operations:MIDNSHP) |
| 7 | MRNM | <p>Mate REference NaMe ('=' if same RNAME) |
| 8 | MPOS | <p>1-Based leftmost Mate POSition |
| 9 | ISIZE | <p>Inferred Insert SIZE |
| 10 | SEQ | <p>Query SEQuence on the same strand as the reference |
| 11 | QUAL | <p>Query QUALity (ASCII-33=Phred base quality) |
** Table 1 **. Mandatory fields in the SAM format.
The standard CIGAR description of pairwise alignment defines three operations: ‘M’ for match/mismatch, ‘I’ for insertion compared with the reference and ‘D’ for deletion. The extended CIGAR proposed in SAM added four more operations: ‘N’ for skipped bases on the reference, ‘S’ for soft clipping, ‘H’ for hard clipping and ‘P’ for padding. These support splicing, clipping, multi-part and padded alignments. Figure 3 shows examples of CIGAR strings for different types of alignments.
![SAM format file. The ‘@SQ’ line in the header section gives the order of reference sequences. Notably, r001 is the name of a read pair. According to FLAG 163 (=1+2+32+128), the read mapped to position 7 is the second read in the pair (128) and regarded as properly paired (1 + 2); its mate is mapped to 37 on the reverse strand (32). Read r002 has three soft-clipped (unaligned) bases. The coordinate shown in SAM is the position of the first aligned base. The CIGAR string for this alignment contains a P (padding) operation which correctly aligns the inserted sequences. Padding operations can be absent when an aligner does not support multiple sequence alignment. The last six bases of read r003 map to position 9, and the first five to position 29 on the reverse strand. The hard clipping operation H indicates that the clipped sequence is not present in the sequence field. The NM tag gives the number of mismatches. Read r004 is aligned across an intron, indicated by the N operation.](../../../img/fig3.png)
** Figure 3 **. SAM format file. The ‘@SQ’ line in the header section gives the order of reference sequences. Notably, r001 is the name of a read pair. According to FLAG 163 (=1+2+32+128), the read mapped to position 7 is the second read in the pair (128) and regarded as properly paired (1 + 2); its mate is mapped to 37 on the reverse strand (32). Read r002 has three soft-clipped (unaligned) bases. The coordinate shown in SAM is the position of the first aligned base. The CIGAR string for this alignment contains a P (padding) operation which correctly aligns the inserted sequences. Padding operations can be absent when an aligner does not support multiple sequence alignment. The last six bases of read r003 map to position 9, and the first five to position 29 on the reverse strand. The hard clipping operation H indicates that the clipped sequence is not present in the sequence field. The NM tag gives the number of mismatches. Read r004 is aligned across an intron, indicated by the N operation.
** Binary Alignment/Map (BAM) **
BAM is the binary representation of SAM and keeps exactly the same information as SAM. BAM uses lossless compression to reduce the size of the data by about 75% and provides an indexing system that allows reads that overlap a region of the genome to be retrieved and rapidly traversed.
#### Quality control, preprocessing and statistics for BAM
** Component **: Hpg-Fastq & FastQC.
Some features
- Quality control
- reads with N errors
- reads with multiple mappings
- strand bias
- paired-end insert
- Filtering: by number of errors, number of hits
- Comparator: stats, intersection, ...
** Input: ** BAM file.
** Output: ** BAM file plus an HTML file containing statistics.
#### Variant Calling
Component: ** GATK **.
Identification of single nucleotide variants and indels on the alignments is performed using the Genome Analysis Toolkit (GATK). GATK (2) is a software package developed at the Broad Institute to analyze high-throughput sequencing data. The toolkit offers a wide variety of tools, with a primary focus on variant discovery and genotyping as well as strong emphasis on data quality assurance.
** Input: ** BAM
** Output: ** VCF
** Variant Call Format (VCF) **
VCF (3) is a standardized format for storing the most prevalent types of sequence variation, including SNPs, indels and larger structural variants, together with rich annotations. The format was developed with the primary intention to represent human genetic variation, but its use is not restricted >to diploid genomes and can be used in different contexts as well. Its flexibility and user extensibility allows representation of a wide variety of genomic variation with respect to a single reference sequence.
A VCF file consists of a header section and a data section. The header contains an arbitrary number of metainformation lines, each starting with characters ‘##’, and a TAB delimited field definition line, starting with a single ‘#’ character. The meta-information header lines provide a standardized description of tags and annotations used in the data section. The use of meta-information allows the information stored within a VCF file to be tailored to the dataset in question. It can be also used to provide information about the means of file creation, date of creation, version of the reference sequence, software used and any other information relevant to the history of the file. The field definition line names eight mandatory columns, corresponding to data columns representing the chromosome (CHROM), a 1-based position of the start of the variant (POS), unique identifiers of the variant (ID), the reference allele (REF), a comma separated list of alternate non-reference alleles (ALT), a phred-scaled quality score (QUAL), site filtering information (FILTER) and a semicolon separated list of additional, user extensible annotation (INFO). In addition, if samples are present in the file, the mandatory header columns are followed by a FORMAT column and an arbitrary number of sample IDs that define the samples included in the VCF file. The FORMAT column is used to define the information contained within each subsequent genotype column, which consists of a colon separated list of fields. For example, the FORMAT field GT:GQ:DP in the fourth data entry of Figure 1a indicates that the subsequent entries contain information regarding the genotype, genotype quality and read depth for each sample. All data lines are TAB delimited and the number of fields in each data line must match the number of fields in the header line. It is strongly recommended that all annotation tags used are declared in the VCF header section.
![a) Example of valid VCF. The header lines ##fileformat and #CHROM are mandatory, the rest is optional but strongly recommended. Each line of the body describes variants present in the sampled population at one genomic position or region. All alternate alleles are listed in the ALT column and referenced from the genotype fields as 1-based indexes to
this list; the reference haplotype is designated as 0. For multiploid data, the separator indicates whether the data are phased (|) or unphased (/). Thus, the two alleles C and G at the positions 2 and 5 in this figure occur on the same chromosome in SAMPLE1. The first data line shows an example of a deletion (present in SAMPLE1) and a replacement of
two bases by another base (SAMPLE2); the second line shows a SNP and an insertion; the third a SNP; the fourth a large structural variant described by the annotation in the INFO column, the coordinate is that of the base before the variant. (b–f ) Alignments and VCF representations of different sequence variants: SNP, insertion, deletion, replacement, and a large deletion. The REF columns shows the reference bases replaced by the haplotype in the ALT column. The coordinate refers to the first reference base. (g) Users are advised to use simplest representation possible and lowest coordinate in cases where the position is ambiguous.](../../../img/fig4.png)
** Figure 4 **. (a) Example of valid VCF. The header lines ##fileformat and #CHROM are mandatory, the rest is optional but strongly recommended. Each line of the body describes variants present in the sampled population at one genomic position or region. All alternate alleles are listed in the ALT column and referenced from the genotype fields as 1-based indexes to this list; the reference haplotype is designated as 0. For multiploid data, the separator indicates whether the data are phased (|) or unphased (/). Thus, the two alleles C and G at the positions 2 and 5 in this figure occur on the same chromosome in SAMPLE1. The first data line shows an example of a deletion (present in SAMPLE1) and a replacement of two bases by another base (SAMPLE2); the second line shows a SNP and an insertion; the third a SNP; the fourth a large structural variant described by the annotation in the INFO column, the coordinate is that of the base before the variant. (b–f ) Alignments and VCF representations of different sequence variants: SNP, insertion, deletion, replacement, and a large deletion. The REF columns shows the reference bases replaced by the haplotype in the ALT column. The coordinate refers to the first reference base. (g) Users are advised to use simplest representation possible and lowest coordinate in cases where the position is ambiguous.
### Annotating
** Component: ** HPG-Variant
The functional consequences of every variant found are then annotated using the HPG-Variant software, which extracts from CellBase, the Knowledge database, all the information relevant on the predicted pathologic effect of the variants.
VARIANT (VARIant Analysis Tool) (4) reports information on the variants found that include consequence type and annotations taken from different databases and repositories (SNPs and variants from dbSNP and 1000 genomes, and disease-related variants from the Genome-Wide Association Study (GWAS) catalog, Online Mendelian Inheritance in Man (OMIM), Catalog of Somatic Mutations in Cancer (COSMIC) mutations, etc. VARIANT also produces a rich variety of annotations that include information on the regulatory (transcription factor or miRNAbinding sites, etc.) or structural roles, or on the selective pressures on the sites affected by the variation. This information allows extending the conventional reports beyond the coding regions and expands the knowledge on the contribution of non-coding or synonymous variants to the phenotype studied.
** Input: ** VCF
** Output: ** The output of this step is the Variant Calling Format (VCF) file, which contains changes with respect to the reference genome with the corresponding QC and functional annotations.
#### CellBase
CellBase(5) is a relational database integrates biological information from different sources and includes:
** Core features: **
We took genome sequences, genes, transcripts, exons, cytobands or cross references (xrefs) identifiers (IDs) from Ensembl (6). Protein information including sequences, xrefs or protein features (natural variants, mutagenesis sites, post-translational modifications, etc.) were imported from UniProt (7).
** Regulatory: **
CellBase imports miRNA from miRBase (8); curated and non-curated miRNA targets from miRecords (9), miRTarBase (10),
TargetScan(11) and microRNA.org (12) and CpG islands and conserved regions from the UCSC database (13).
** Functional annotation **
OBO Foundry (14) develops many biomedical ontologies that are implemented in OBO format. We designed a SQL schema to store these OBO ontologies and 30 ontologies were imported. OBO ontology term annotations were taken from Ensembl (6). InterPro (15) annotations were also imported.
** Variation **
CellBase includes SNPs from dbSNP (16)^; SNP population frequencies from HapMap (17), 1000 genomes project (18) and Ensembl (6); phenotypically annotated SNPs were imported from NHRI GWAS Catalog (19),HGMD (20), Open Access GWAS Database (21), UniProt (7) and OMIM (22); mutations from COSMIC (23) and structural variations from Ensembl (6).
** Systems biology **
We also import systems biology information like interactome information from IntAct (24). Reactome (25) stores pathway and interaction information in BioPAX (26) format. BioPAX data exchange format enables the integration of diverse pathway
resources. We successfully solved the problem of storing data released in BioPAX format into a SQL relational schema, which allowed us importing Reactome in CellBase.
### [Diagnostic component (TEAM)](diagnostic-component-team/)
### [Priorization component (BiERApp)](priorization-component-bierapp/)
## Usage
First of all, we should load ngsPipeline module:
```bash
$ module load ngsPipeline
```
This command will load python/2.7.5 module and all the required modules (hpg-aligner, gatk, etc)
If we launch ngsPipeline with ‘-h’, we will get the usage help:
```bash
$ ngsPipeline -h
Usage: ngsPipeline.py [-h] -i INPUT -o OUTPUT -p PED --project PROJECT --queue
QUEUE [--stages-path STAGES_PATH] [--email EMAIL]
[--prefix PREFIX] [-s START] [-e END] --log
Python pipeline
optional arguments:
-h, --help show this help message and exit
-i INPUT, --input INPUT
-o OUTPUT, --output OUTPUT
Output Data directory
-p PED, --ped PED Ped file with all individuals
--project PROJECT Project Id
--queue QUEUE Queue Id
--stages-path STAGES_PATH
Custom Stages path
--email EMAIL Email
--prefix PREFIX Prefix name for Queue Jobs name
-s START, --start START
Initial stage
-e END, --end END Final stage
--log Log to file
```
Let us see a brief description of the arguments:
```bash
*-h --help*. Show the help.
*-i, --input.* The input data directory. This directory must to have a special structure. We have to create one folder per sample (with the same name). These folders will host the fastq files. These fastq files must have the following pattern “sampleName” + “_” + “1 or 2” + “.fq”. 1 for the first pair (in paired-end sequences), and 2 for the
second one.
*-o , --output.* The output folder. This folder will contain all the intermediate and final folders. When the pipeline will be executed completely, we could remove the intermediate folders and keep only the final one (with the VCF file containing all the variants)
*-p , --ped*. The ped file with the pedigree. This file contains all the sample names. These names must coincide with the names of the input folders. If our input folder contains more samples than the .ped file, the pipeline will use only the samples from the .ped file.
*--email.* Email for PBS notifications.
*--prefix.* Prefix for PBS Job names.
*-s, --start & -e, --end.* Initial and final stage. If we want to launch the pipeline in a specific stage we must use -s. If we want to end the pipeline in a specific stage we must use -e.
*--log*. Using log argument NGSpipeline will prompt all the logs to this file.
*--project*>. Project ID of your supercomputer allocation.
*--queue*. [Queue](../../resource-allocation-and-job-execution/introduction.html) to run the jobs in.
```
Input, output and ped arguments are mandatory. If the output folder does not exist, the pipeline will create it.
## Examples
This is an example usage of NGSpipeline:
We have a folder with the following structure in
```bash
/apps/bio/omics/1.0/sample_data/ >:
/apps/bio/omics/1.0/sample_data
└── data
├── file.ped
├── sample1
│ ├── sample1_1.fq
│ └── sample1_2.fq
└── sample2
├── sample2_1.fq
└── sample2_2.fq
```
The ped file ( file.ped) contains the following info:
```bash
#family_ID sample_ID parental_ID maternal_ID sex phenotype
FAM sample_A 0 0 1 1
FAM sample_B 0 0 2 2
```
Now, lets load the NGSPipeline module and copy the sample data to a [scratch directory](../../storage/storage/):
```bash
$ module load ngsPipeline
$ mkdir -p /scratch/$USER/omics/results
$ cp -r /apps/bio/omics/1.0/sample_data /scratch/$USER/omics/
```
Now, we can launch the pipeline (replace OPEN-0-0 with your Project ID):
```bash
$ ngsPipeline -i /scratch/$USER/omics/sample_data/data -o /scratch/$USER/omics/results -p /scratch/$USER/omics/sample_data/data/file.ped --project OPEN-0-0 --queue qprod
```
This command submits the processing [jobs to the queue](../../resource-allocation-and-job-execution/job-submission-and-execution.html).
If we want to re-launch the pipeline from stage 4 until stage 20 we should use the next command:
```bash
$ ngsPipeline -i /scratch/$USER/omics/sample_data/data -o /scratch/$USER/omics/results -p /scratch/$USER/omics/sample_data/data/file.ped -s 4 -e 20 --project OPEN-0-0 --queue qprod
```
## Details on the pipeline
The pipeline calls the following tools
- [fastqc](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/), quality control tool for high throughput sequence data.
- [gatk](https://www.broadinstitute.org/gatk/), The Genome Analysis Toolkit or GATK is a software package developed at
the Broad Institute to analyze high-throughput sequencing data. The toolkit offers a wide variety of tools, with a primary focus on variant discovery and genotyping as well as strong emphasis on data quality assurance. Its robust architecture, powerful processing engine and high-performance computing features make it capable of taking on projects of any size.
- [hpg-aligner](https://github.com/opencb-hpg/hpg-aligner), HPG Aligner has been designed to align short and long reads with high sensitivity, therefore any number of mismatches or indels are allowed. HPG Aligner implements and combines two well known algorithms: _Burrows-Wheeler Transform_ (BWT) to speed-up mapping high-quality reads, and _Smith-Waterman_> (SW) to increase sensitivity when reads cannot be mapped using BWT.
- [hpg-fastq](http://docs.bioinfo.cipf.es/projects/fastqhpc/wiki), a quality control tool for high throughput sequence data.
- [hpg-variant](http://docs.bioinfo.cipf.es/projects/hpg-variant/wiki), The HPG Variant suite is an ambitious project aimed to provide a complete suite of tools to work with genomic variation data, from VCF tools to variant profiling or genomic statistics. It is being implemented using High Performance Computing technologies to provide the best performance possible.
- [picard](http://picard.sourceforge.net/), Picard comprises Java-based command-line utilities that manipulate SAM files, and a Java API (HTSJDK) for creating new programs that read and write SAM files. Both SAM text format and SAM binary (BAM) format are supported.
- [samtools](http://samtools.sourceforge.net/samtools-c.shtml), SAM Tools provide various utilities for manipulating alignments in the SAM format, including sorting, merging, indexing and generating alignments in a per-position format.
- [snpEff](http://snpeff.sourceforge.net/), Genetic variant annotation and effect prediction toolbox.
This listing show which tools are used in each step of the pipeline
- stage-00: fastqc
- stage-01: hpg_fastq
- stage-02: fastqc
- stage-03: hpg_aligner and samtools
- stage-04: samtools
- stage-05: samtools
- stage-06: fastqc
- stage-07: picard
- stage-08: fastqc
- stage-09: picard
- stage-10: gatk
- stage-11: gatk
- stage-12: gatk
- stage-13: gatk
- stage-14: gatk
- stage-15: gatk
- stage-16: samtools
- stage-17: samtools
- stage-18: fastqc
- stage-19: gatk
- stage-20: gatk
- stage-21: gatk
- stage-22: gatk
- stage-23: gatk
- stage-24: hpg-variant
- stage-25: hpg-variant
- stage-26: snpEff
- stage-27: snpEff
- stage-28: hpg-variant
## Interpretation
The output folder contains all the subfolders with the intermediate data. This folder contains the final VCF with all the variants. This file can be uploaded into [TEAM](diagnostic-component-team.html) by using the VCF file button. It is important to note here that the entire management of the VCF file is local: no patient’s sequence data is sent over the Internet thus avoiding any problem of data privacy or confidentiality.
![TEAM upload panel. Once the file has been uploaded, a panel must be chosen from the Panel list. Then, pressing the Run button the diagnostic process starts.]\((../../../img/fig7.png)
** Figure 7. ** _TEAM upload panel._ _Once the file has been uploaded, a panel must be chosen from the Panel_ list. Then, pressing the Run button the diagnostic process starts.
Once the file has been uploaded, a panel must be chosen from the Panel list. Then, pressing the Run button the diagnostic process starts. TEAM searches first for known diagnostic mutation(s) taken from four databases: HGMD-public (20), [HUMSAVAR](http://www.uniprot.org/docs/humsavar), ClinVar (29) and COSMIC (23).
![The panel manager. The elements used to define a panel are (A) disease terms, (B) diagnostic mutations and (C) genes. Arrows represent actions that can be taken in the panel manager. Panels can be defined by using the known mutations and genes of a particular disease. This can be done by dragging them to the Primary Diagnostic box (action D). This action, in addition to defining the diseases in the Primary Diagnostic box, automatically adds the corresponding genes to the Genes box. The panels can be customized by adding new genes (action F) or removing undesired genes (action G). New disease mutations can be added independently or associated to an already existing disease term (action E). Disease terms can be removed by simply dragging themback (action H).](../../../img/fig7x.png)
** Figure 7. ** The panel manager. The elements used to define a panel are (** A **) disease terms, (** B **) diagnostic mutations and (** C **) genes. Arrows represent actions that can be taken in the panel manager. Panels can be defined by using the known mutations and genes of a particular disease. This can be done by dragging them to the ** Primary Diagnostic ** box (action ** D **). This action, in addition to defining the diseases in the ** Primary Diagnostic ** box, automatically adds the corresponding genes to the ** Genes ** box. The panels can be customized by adding new genes (action ** F **) or removing undesired genes (action **G**). New disease mutations can be added independently or associated to an already existing disease term (action ** E **). Disease terms can be removed by simply dragging them back (action ** H **).
For variant discovering/filtering we should upload the VCF file into BierApp by using the following form:
_![BierApp VCF upload panel. It is recommended to choose a name for the job as well as a description.](../../../img/fig8.png)_
** Figure 8 **. \*BierApp VCF upload panel. It is recommended to choose a name for the job as well as a description \*\*.
Each prioritization (‘job’) has three associated screens that facilitate the filtering steps. The first one, the ‘Summary’ tab, displays a statistic of the data set analyzed, containing the samples analyzed, the number and types of variants found and its distribution according to consequence types. The second screen, in the ‘Variants and effect’ tab, is the actual filtering tool, and the third one, the ‘Genome view’ tab, offers a representation of the selected variants within the genomic context provided by an embedded version of the Genome Maps Tool (30).
![This picture shows all the information associated to the variants. If a variant has an associated phenotype we could see it in the last column. In this case, the variant 7:132481242 CT is associated to the phenotype: large intestine tumor.](../../../img/fig9.png)
** Figure 9 **. This picture shows all the information associated to the variants. If a variant has an associated phenotype we could see it in the last column. In this case, the variant 7:132481242 CT is associated to the phenotype: large intestine tumor.
## References
1. Heng Li, Bob Handsaker, Alec Wysoker, Tim Fennell, Jue Ruan, Nils Homer, Gabor Marth5, Goncalo Abecasis6, Richard Durbin and 1000 Genome Project Data Processing Subgroup: The Sequence Alignment/Map format and SAMtools. Bioinformatics 2009, 25: 2078-2079.
2. McKenna A, Hanna M, Banks E, Sivachenko A, Cibulskis K, Kernytsky A, Garimella K, Altshuler D, Gabriel S, Daly M, DePristo MA: The Genome Analysis Toolkit: a MapReduce framework for analyzing next-generation DNA sequencing data. _Genome Res_ >2010, 20:1297-1303.
3. Petr Danecek, Adam Auton, Goncalo Abecasis, Cornelis A. Albers, Eric Banks, Mark A. DePristo, Robert E. Handsaker, Gerton Lunter, Gabor T. Marth, Stephen T. Sherry, Gilean McVean, Richard Durbin, and 1000 Genomes Project Analysis Group. The variant call format and VCFtools. Bioinformatics 2011, 27: 2156-2158.
4. Medina I, De Maria A, Bleda M, Salavert F, Alonso R, Gonzalez CY, Dopazo J: VARIANT: Command Line, Web service and Web interface for fast and accurate functional characterization of variants found by Next-Generation Sequencing. Nucleic Acids Res 2012, 40:W54-58.
5. Bleda M, Tarraga J, de Maria A, Salavert F, Garcia-Alonso L, Celma M, Martin A, Dopazo J, Medina I: CellBase, a comprehensive collection of RESTful web services for retrieving relevant biological information from heterogeneous sources. Nucleic Acids Res 2012, 40:W609-614.
6. Flicek,P., Amode,M.R., Barrell,D., Beal,K., Brent,S., Carvalho-Silva,D., Clapham,P., Coates,G., Fairley,S., Fitzgerald,S. et al. (2012) Ensembl 2012. Nucleic Acids Res., 40, D84–D90.
7. UniProt Consortium. (2012) Reorganizing the protein space at the Universal Protein Resource (UniProt). Nucleic Acids Res., 40, D71–D75.
8. Kozomara,A. and Griffiths-Jones,S. (2011) miRBase: integrating microRNA annotation and deep-sequencing data. Nucleic Acids Res., 39, D152–D157.
9. Xiao,F., Zuo,Z., Cai,G., Kang,S., Gao,X. and Li,T. (2009) miRecords: an integrated resource for microRNA-target interactions. Nucleic Acids Res., 37, D105–D110.
10. Hsu,S.D., Lin,F.M., Wu,W.Y., Liang,C., Huang,W.C., Chan,W.L., Tsai,W.T., Chen,G.Z., Lee,C.J., Chiu,C.M. et al. (2011) miRTarBase: a database curates experimentally validated microRNA-target interactions. Nucleic Acids Res., 39, D163–D169.
11. Friedman,R.C., Farh,K.K., Burge,C.B. and Bartel,D.P. (2009) Most mammalian mRNAs are conserved targets of microRNAs. Genome Res., 19, 92–105. 12. Betel,D., Wilson,M., Gabow,A., Marks,D.S. and Sander,C. (2008) The microRNA.org resource: targets and expression. Nucleic Acids Res., 36, D149–D153.
12. Dreszer,T.R., Karolchik,D., Zweig,A.S., Hinrichs,A.S., Raney,B.J., Kuhn,R.M., Meyer,L.R., Wong,M., Sloan,C.A., Rosenbloom,K.R. et al. (2012) The UCSC genome browser database: extensions and updates 2011. Nucleic Acids Res.,40, D918–D923.
13. Smith,B., Ashburner,M., Rosse,C., Bard,J., Bug,W., Ceusters,W., Goldberg,L.J., Eilbeck,K., Ireland,A., Mungall,C.J. et al. (2007) The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration. Nat. Biotechnol., 25, 1251–1255.
14. Hunter,S., Jones,P., Mitchell,A., Apweiler,R., Attwood,T.K.,Bateman,A., Bernard,T., Binns,D., Bork,P., Burge,S. et al. (2012) InterPro in 2011: new developments in the family and domain prediction database. Nucleic Acids Res.,40, D306–D312.
15. Sherry,S.T., Ward,M.H., Kholodov,M., Baker,J., Phan,L., Smigielski,E.M. and Sirotkin,K. (2001) dbSNP: the NCBI database of genetic variation. Nucleic Acids Res., 29, 308–311.
16. Altshuler,D.M., Gibbs,R.A., Peltonen,L., Dermitzakis,E., Schaffner,S.F., Yu,F., Bonnen,P.E., de Bakker,P.I., Deloukas,P., Gabriel,S.B. et al. (2010) Integrating common and rare genetic variation in diverse human populations. Nature, 467, 52–58.
17. 1000 Genomes Project Consortium. (2010) A map of human genome variation from population-scale sequencing. Nature, 467, 1061–1073.
18. Hindorff,L.A., Sethupathy,P., Junkins,H.A., Ramos,E.M., Mehta,J.P., Collins,F.S. and Manolio,T.A. (2009) Potential etiologic and functional implications of genome-wide association loci for human diseases and traits. Proc. Natl Acad. Sci. USA, 106, 9362–9367.
19. Stenson,P.D., Ball,E.V., Mort,M., Phillips,A.D., Shiel,J.A., Thomas,N.S., Abeysinghe,S., Krawczak,M. and Cooper,D.N. (2003) Human gene mutation database (HGMD): 2003 update. Hum. Mutat., 21, 577–581.
20. Johnson,A.D. and O’Donnell,C.J. (2009) An open access database of genome-wide association results. BMC Med. Genet, 10, 6.
21. McKusick,V. (1998) A Catalog of Human Genes and Genetic Disorders, 12th edn. John Hopkins University Press,Baltimore, MD.
22. Forbes,S.A., Bindal,N., Bamford,S., Cole,C., Kok,C.Y., Beare,D., Jia,M., Shepherd,R., Leung,K., Menzies,A. et al. (2011) COSMIC: mining complete cancer genomes in the catalogue of somatic mutations in cancer. Nucleic Acids Res., 39, D945–D950.
23. Kerrien,S., Aranda,B., Breuza,L., Bridge,A., Broackes-Carter,F., Chen,C., Duesbury,M., Dumousseau,M., Feuermann,M., Hinz,U. et al. (2012) The Intact molecular interaction database in 2012. Nucleic Acids Res., 40, D841–D846.
24. Croft,D., O’Kelly,G., Wu,G., Haw,R., Gillespie,M., Matthews,L., Caudy,M., Garapati,P., Gopinath,G., Jassal,B. et al. (2011) Reactome: a database of reactions, pathways and biological processes. Nucleic Acids Res., 39, D691–D697.
25. Demir,E., Cary,M.P., Paley,S., Fukuda,K., Lemer,C., Vastrik,I.,Wu,G., D’Eustachio,P., Schaefer,C., Luciano,J. et al. (2010) The BioPAX community standard for pathway data sharing. Nature Biotechnol., 28, 935–942.
26. Alemán Z, García-García F, Medina I, Dopazo J (2014): A web tool for the design and management of panels of genes for targeted enrichment and massive sequencing for clinical applications. Nucleic Acids Res 42: W83-7.
27. [Alemán A](http://www.ncbi.nlm.nih.gov/pubmed?term=Alem%C3%A1n%20A%5BAuthor%5D&cauthor=true&cauthor_uid=24803668)>, [Garcia-Garcia F](http://www.ncbi.nlm.nih.gov/pubmed?term=Garcia-Garcia%20F%5BAuthor%5D&cauthor=true&cauthor_uid=24803668)>, [Salavert F](http://www.ncbi.nlm.nih.gov/pubmed?term=Salavert%20F%5BAuthor%5D&cauthor=true&cauthor_uid=24803668)>, [Medina I](http://www.ncbi.nlm.nih.gov/pubmed?term=Medina%20I%5BAuthor%5D&cauthor=true&cauthor_uid=24803668)>, [Dopazo J](http://www.ncbi.nlm.nih.gov/pubmed?term=Dopazo%20J%5BAuthor%5D&cauthor=true&cauthor_uid=24803668)> (2014). A web-based interactive framework to assist in the prioritization of disease candidate genes in whole-exome sequencing studies. [Nucleic Acids Res.](http://www.ncbi.nlm.nih.gov/pubmed/?term=BiERapp "Nucleic acids research.")>42 :W88-93.
28. Landrum,M.J., Lee,J.M., Riley,G.R., Jang,W., Rubinstein,W.S., Church,D.M. and Maglott,D.R. (2014) ClinVar: public archive of relationships among sequence variation and human phenotype. Nucleic Acids Res., 42, D980–D985.
29. Medina I, Salavert F, Sanchez R, de Maria A, Alonso R, Escobar P, Bleda M, Dopazo J: Genome Maps, a new generation genome browser. Nucleic Acids Res 2013, 41:W41-46.
docs.it4i/anselm-cluster-documentation/software/omics-master/priorization-component-bierapp.md 0 → 100644
View file @ 14676d74
# Prioritization component (BiERapp)
## Access
BiERapp is available at the [following address](http://omics.it4i.cz/bierapp/)
!!! note
The address is accessible only via VPN.
## BiERapp
** This tool is aimed to discover new disease genes or variants by studying affected families or cases and controls. It carries out a filtering process to sequentially remove: (i) variants which are not no compatible with the disease because are not expected to have impact on the protein function; (ii) variants that exist at frequencies incompatible with the disease; (iii) variants that do not segregate with the disease. The result is a reduced set of disease gene candidates that should be further validated experimentally. **
BiERapp (28) efficiently helps in the identification of causative variants in family and sporadic genetic diseases. The program reads lists of predicted variants (nucleotide substitutions and indels) in affected individuals or tumor samples and controls. In family studies, different modes of inheritance can easily be defined to filter out variants that do not segregate with the disease along the family. Moreover, BiERapp integrates additional information such as allelic frequencies in the general population and the most popular damaging scores to further narrow down the number of putative variants in successive filtering steps. BiERapp provides an interactive and user-friendly interface that implements the filtering strategy used in the context of a large-scale genomic project carried out by the Spanish Network for Research, in Rare Diseases (CIBERER) and the Medical Genome Project. in which more than 800 exomes have been analyzed.
![Web interface to the prioritization tool. This figure shows the interface of the web tool for candidate gene prioritization with the filters available. The tool includes a genomic viewer (Genome Maps 30) that enables the representation of the variants in the corresponding genomic coordinates.](../../../img/fig6.png)
** Figure 6 **. Web interface to the prioritization tool. This figure shows the interface of the web tool for candidate gene
prioritization with the filters available. The tool includes a genomic viewer (Genome Maps 30) that enables the representation of the variants in the corresponding genomic coordinates.
docs.it4i/anselm-cluster-documentation/software/openfoam.md 0 → 100644
View file @ 14676d74
# OpenFOAM
\##A free, open source CFD software package
## Introduction
OpenFOAM is a free, open source CFD software package developed by [**OpenCFD Ltd**](http://www.openfoam.com/about) at [**ESI Group**](http://www.esi-group.com/) and distributed by the [**OpenFOAM Foundation **](http://www.openfoam.org/). It has a large user base across most areas of engineering and science, from both commercial and academic organisations.
Homepage: <http://www.openfoam.com/>
\###Installed version
Currently, several version compiled by GCC/ICC compilers in single/double precision with several version of openmpi are available on Anselm.
For example syntax of available OpenFOAM module is:
&lt; openfoam/2.2.1-icc-openmpi1.6.5-DP >
this means openfoam version 2.2.1 compiled by ICC compiler with openmpi1.6.5 in double precision.
Naming convection of the installed versions is following:
openfoam/&lt;>VERSION>>-&lt;>COMPILER<span>>-&lt;</span><span>openmpiVERSION</span><span>>-&lt;</span><span>PRECISION</span><span>></span>
- &lt;VERSION>> - version of openfoam
- &lt;COMPILER> - version of used compiler
- &lt;openmpiVERSION> - version of used openmpi/impi
- &lt;PRECISION> - DP/SP – double/single precision
\###Available OpenFOAM modules
To check available modules use
```bash
$ module avail
```
In /opt/modules/modulefiles/engineering you can see installed engineering softwares:
```bash
------------------------------------ /opt/modules/modulefiles/engineering -------------------------------------------------------------
ansys/14.5.x matlab/R2013a-COM openfoam/2.2.1-icc-impi4.1.1.036-DP
comsol/43b-COM matlab/R2013a-EDU openfoam/2.2.1-icc-openmpi1.6.5-DP
comsol/43b-EDU openfoam/2.2.1-gcc481-openmpi1.6.5-DP paraview/4.0.1-gcc481-bullxmpi1.2.4.1-osmesa10.0
lsdyna/7.x.x openfoam/2.2.1-gcc481-openmpi1.6.5-SP
```
For information how to use modules please [look here](../environment-and-modules/ "Environment and Modules ").
## Getting Started
To create OpenFOAM environment on ANSELM give the commands:
```bash
$ module load openfoam/2.2.1-icc-openmpi1.6.5-DP
$ source $FOAM_BASHRC
```
!!! Note "Note"
Please load correct module with your requirements “compiler - GCC/ICC, precision - DP/SP”.
Create a project directory within the $HOME/OpenFOAM directory named >&lt;USER>-&lt;OFversion> and create a directory named run within it, e.g. by typing:
```bash
$ mkdir -p $FOAM_RUN
```
Project directory is now available by typing:
```bash
$ cd /home/<USER>/OpenFOAM/<USER>-<OFversion>/run
```
&lt;OFversion> - for example &lt;2.2.1>
or
```bash
$ cd $FOAM_RUN
```
Copy the tutorial examples directory in the OpenFOAM distribution to the run directory:
```bash
$ cp -r $FOAM_TUTORIALS $FOAM_RUN
```
Now you can run the first case for example incompressible laminar flow in a cavity.
## Running Serial Applications
Create a Bash script >test.sh
```bash
#!/bin/bash
module load openfoam/2.2.1-icc-openmpi1.6.5-DP
source $FOAM_BASHRC
# source to run functions
. $WM_PROJECT_DIR/bin/tools/RunFunctions
cd $FOAM_RUN/tutorials/incompressible/icoFoam/cavity
runApplication blockMesh
runApplication icoFoam
```
Job submission
```bash
$ qsub -A OPEN-0-0 -q qprod -l select=1:ncpus=16,walltime=03:00:00 test.sh
```
For information about job submission please [look here](../resource-allocation-and-job-execution/job-submission-and-execution/ "Job submission").
## Running applications in parallel
Run the second case for example external incompressible turbulent flow - case - motorBike.
First we must run serial application bockMesh and decomposePar for preparation of parallel computation.
!!! Note "Note"
Create a Bash scrip test.sh:
```bash
#!/bin/bash
module load openfoam/2.2.1-icc-openmpi1.6.5-DP
source $FOAM_BASHRC
# source to run functions
. $WM_PROJECT_DIR/bin/tools/RunFunctions
cd $FOAM_RUN/tutorials/incompressible/simpleFoam/motorBike
runApplication blockMesh
runApplication decomposePar
```
Job submission
```bash
$ qsub -A OPEN-0-0 -q qprod -l select=1:ncpus=16,walltime=03:00:00 test.sh
```
This job create simple block mesh and domain decomposition. Check your decomposition, and submit parallel computation:
!!! Note "Note"
Create a PBS script testParallel.pbs:
```bash
#!/bin/bash
#PBS -N motorBike
#PBS -l select=2:ncpus=16
#PBS -l walltime=01:00:00
#PBS -q qprod
#PBS -A OPEN-0-0
module load openfoam/2.2.1-icc-openmpi1.6.5-DP
source $FOAM_BASHRC
cd $FOAM_RUN/tutorials/incompressible/simpleFoam/motorBike
nproc = 32
mpirun -hostfile ${PBS_NODEFILE} -np $nproc snappyHexMesh -overwrite -parallel | tee snappyHexMesh.log
mpirun -hostfile ${PBS_NODEFILE} -np $nproc potentialFoam -noFunctionObject-writep -parallel | tee potentialFoam.log
mpirun -hostfile ${PBS_NODEFILE} -np $nproc simpleFoam -parallel | tee simpleFoam.log
```
nproc – number of subdomains
Job submission
```bash
$ qsub testParallel.pbs
```
## Compile your own solver
Initialize OpenFOAM environment before compiling your solver
```bash
$ module load openfoam/2.2.1-icc-openmpi1.6.5-DP
$ source $FOAM_BASHRC
$ cd $FOAM_RUN/
```
Create directory applications/solvers in user directory
```bash
$ mkdir -p applications/solvers
$ cd applications/solvers
```
Copy icoFoam solver’s source files
```bash
$ cp -r $FOAM_SOLVERS/incompressible/icoFoam/ My_icoFoam
$ cd My_icoFoam
```
Rename icoFoam.C to My_icoFOAM.C
```bash
$ mv icoFoam.C My_icoFoam.C
```
Edit >_files_ file in _Make_ directory:
```bash
icoFoam.C
EXE = $(FOAM_APPBIN)/icoFoam
```
and change to:
```bash
My_icoFoam.C
EXE = $(FOAM_USER_APPBIN)/My_icoFoam
```
In directory My_icoFoam give the compilation command:
```bash
$ wmake
```
* * *
**Have a fun with OpenFOAM :)**
docs.it4i/anselm-cluster-documentation/software/operating-system.md 0 → 100644
View file @ 14676d74
# Operating System
The operating system on Anselm is Linux - **bullx Linux Server release 6.x**
bullx Linux is based on Red Hat Enterprise Linux. bullx Linux is a Linux distribution provided by Bull and dedicated to HPC applications.
docs.it4i/anselm-cluster-documentation/software/paraview.md 0 → 100644
View file @ 14676d74
# ParaView
\##An open-source, multi-platform data analysis and visualization application
## Introduction
**ParaView** is an open-source, multi-platform data analysis and visualization application. ParaView users can quickly build visualizations to analyze their data using qualitative and quantitative techniques. The data exploration can be done interactively in 3D or programmatically using ParaView's batch processing capabilities.
ParaView was developed to analyze extremely large datasets using distributed memory computing resources. It can be run on supercomputers to analyze datasets of exascale size as well as on laptops for smaller data.
Homepage : <http://www.paraview.org/>
## Installed version
Currently, version 4.0.1 compiled with GCC 4.8.1 against Bull MPI library and OSMesa 10.0 is installed on Anselm.
## Usage
On Anselm, ParaView is to be used in client-server mode. A parallel ParaView server is launched on compute nodes by the user, and client is launched on your desktop PC to control and view the visualization. Download ParaView client application for your OS here: <http://paraview.org/paraview/resources/software.php>. Important : **your version must match the version number installed on Anselm** ! (currently v4.0.1)
### Launching server
To launch the server, you must first allocate compute nodes, for example
```bash
$ qsub -I -q qprod -A OPEN-0-0 -l select=2
```
to launch an interactive session on 2 nodes. Refer to [Resource Allocation and Job Execution](../resource-allocation-and-job-execution/introduction/) for details.
After the interactive session is opened, load the ParaView module :
```bash
$ module add paraview
```
Now launch the parallel server, with number of nodes times 16 processes:
```bash
$ mpirun -np 32 pvserver --use-offscreen-rendering
Waiting for client...
Connection URL: cs://cn77:11111
Accepting connection(s): cn77:11111
```
Note the that the server is listening on compute node cn77 in this case, we shall use this information later.
### Client connection
Because a direct connection is not allowed to compute nodes on Anselm, you must establish a SSH tunnel to connect to the server. Choose a port number on your PC to be forwarded to ParaView server, for example 12345. If your PC is running Linux, use this command to establish a SSH tunnel:
```bash
ssh -TN -L 12345:cn77:11111 username@anselm.it4i.cz
```
replace username with your login and cn77 with the name of compute node your ParaView server is running on (see previous step). If you use PuTTY on Windows, load Anselm connection configuration, t>hen go to Connection-> SSH>->Tunnels to set up the port forwarding. Click Remote radio button. Insert 12345 to Source port textbox. Insert cn77:11111. Click Add button, then Open.
Now launch ParaView client installed on your desktop PC. Select File->Connect..., click Add Server. Fill in the following :
Name : Anselm tunnel
Server Type : Client/Server
Host : localhost
Port : 12345
Click Configure, Save, the configuration is now saved for later use. Now click Connect to connect to the ParaView server. In your terminal where you have interactive session with ParaView server launched, you should see:
```bash
Client connected.
```
You can now use Parallel ParaView.
### Close server
Remember to close the interactive session after you finish working with ParaView server, as it will remain launched even after your client is disconnected and will continue to consume resources.
## GPU support
Currently, GPU acceleration is not supported in the server and ParaView will not take advantage of accelerated nodes on Anselm. Support for GPU acceleration might be added in the future.
docs.it4i/anselm-cluster-documentation/storage.md 0 → 100644
View file @ 14676d74
# Storage
There are two main shared file systems on Anselm cluster, the [HOME](#home) and [SCRATCH](#scratch). All login and compute nodes may access same data on shared file systems. Compute nodes are also equipped with local (non-shared) scratch, ramdisk and tmp file systems.
## Archiving
Please don't use shared filesystems as a backup for large amount of data or long-term archiving mean. The academic staff and students of research institutions in the Czech Republic can use [CESNET storage service](#cesnet-data-storage), which is available via SSHFS.
## Shared Filesystems
Anselm computer provides two main shared filesystems, the [HOME filesystem](#home) and the [SCRATCH filesystem](#scratch). Both HOME and SCRATCH filesystems are realized as a parallel Lustre filesystem. Both shared file systems are accessible via the Infiniband network. Extended ACLs are provided on both Lustre filesystems for the purpose of sharing data with other users using fine-grained control.
### Understanding the Lustre Filesystems
(source <http://www.nas.nasa.gov>)
A user file on the Lustre filesystem can be divided into multiple chunks (stripes) and stored across a subset of the object storage targets (OSTs) (disks). The stripes are distributed among the OSTs in a round-robin fashion to ensure load balancing.
When a client (a compute node from your job) needs to create or access a file, the client queries the metadata server ( MDS) and the metadata target ( MDT) for the layout and location of the [file's stripes](http://www.nas.nasa.gov/hecc/support/kb/Lustre_Basics_224.html#striping). Once the file is opened and the client obtains the striping information, the MDS is no longer involved in the file I/O process. The client interacts directly with the object storage servers (OSSes) and OSTs to perform I/O operations such as locking, disk allocation, storage, and retrieval.
If multiple clients try to read and write the same part of a file at the same time, the Lustre distributed lock manager enforces coherency so that all clients see consistent results.
There is default stripe configuration for Anselm Lustre filesystems. However, users can set the following stripe parameters for their own directories or files to get optimum I/O performance:
1. stripe_size: the size of the chunk in bytes; specify with k, m, or g to use units of KB, MB, or GB, respectively; the size must be an even multiple of 65,536 bytes; default is 1MB for all Anselm Lustre filesystems
2. stripe_count the number of OSTs to stripe across; default is 1 for Anselm Lustre filesystems one can specify -1 to use all OSTs in the filesystem.
3. stripe_offset The index of the OST where the first stripe is to be placed; default is -1 which results in random selection; using a non-default value is NOT recommended.
!!! Note "Note"
Setting stripe size and stripe count correctly for your needs may significantly impact the I/O performance you experience.
Use the lfs getstripe for getting the stripe parameters. Use the lfs setstripe command for setting the stripe parameters to get optimal I/O performance The correct stripe setting depends on your needs and file access patterns.
```bash
$ lfs getstripe dir|filename
$ lfs setstripe -s stripe_size -c stripe_count -o stripe_offset dir|filename
```
Example:
```bash
$ lfs getstripe /scratch/username/
/scratch/username/
stripe_count: 1 stripe_size: 1048576 stripe_offset: -1
$ lfs setstripe -c -1 /scratch/username/
$ lfs getstripe /scratch/username/
/scratch/username/
stripe_count: 10 stripe_size: 1048576 stripe_offset: -1
```
In this example, we view current stripe setting of the /scratch/username/ directory. The stripe count is changed to all OSTs, and verified. All files written to this directory will be striped over 10 OSTs
Use lfs check OSTs to see the number and status of active OSTs for each filesystem on Anselm. Learn more by reading the man page
```bash
$ lfs check osts
$ man lfs
```
### Hints on Lustre Stripping
!!! Note "Note"
Increase the stripe_count for parallel I/O to the same file.
When multiple processes are writing blocks of data to the same file in parallel, the I/O performance for large files will improve when the stripe_count is set to a larger value. The stripe count sets the number of OSTs the file will be written to. By default, the stripe count is set to 1. While this default setting provides for efficient access of metadata (for example to support the ls -l command), large files should use stripe counts of greater than 1. This will increase the aggregate I/O bandwidth by using multiple OSTs in parallel instead of just one. A rule of thumb is to use a stripe count approximately equal to the number of gigabytes in the file.
Another good practice is to make the stripe count be an integral factor of the number of processes performing the write in parallel, so that you achieve load balance among the OSTs. For example, set the stripe count to 16 instead of 15 when you have 64 processes performing the writes.
!!! Note "Note"
Using a large stripe size can improve performance when accessing very large files
Large stripe size allows each client to have exclusive access to its own part of a file. However, it can be counterproductive in some cases if it does not match your I/O pattern. The choice of stripe size has no effect on a single-stripe file.
Read more on <http://doc.lustre.org/lustre_manual.xhtml#managingstripingfreespace>
### Lustre on Anselm
The architecture of Lustre on Anselm is composed of two metadata servers (MDS) and four data/object storage servers (OSS). Two object storage servers are used for file system HOME and another two object storage servers are used for file system SCRATCH.
Configuration of the storages
- HOME Lustre object storage
- One disk array NetApp E5400
- 22 OSTs
- 227 2TB NL-SAS 7.2krpm disks
- 22 groups of 10 disks in RAID6 (8+2)
- 7 hot-spare disks
- SCRATCH Lustre object storage
- Two disk arrays NetApp E5400
- 10 OSTs
- 106 2TB NL-SAS 7.2krpm disks
- 10 groups of 10 disks in RAID6 (8+2)
- 6 hot-spare disks
- Lustre metadata storage
- One disk array NetApp E2600
- 12 300GB SAS 15krpm disks
- 2 groups of 5 disks in RAID5
- 2 hot-spare disks
\###HOME
The HOME filesystem is mounted in directory /home. Users home directories /home/username reside on this filesystem. Accessible capacity is 320TB, shared among all users. Individual users are restricted by filesystem usage quotas, set to 250GB per user. If 250GB should prove as insufficient for particular user, please contact [support](https://support.it4i.cz/rt), the quota may be lifted upon request.
!!! Note "Note"
The HOME filesystem is intended for preparation, evaluation, processing and storage of data generated by active Projects.
The HOME filesystem should not be used to archive data of past Projects or other unrelated data.
The files on HOME filesystem will not be deleted until end of the [users lifecycle](../get-started-with-it4innovations/obtaining-login-credentials/obtaining-login-credentials/).
The filesystem is backed up, such that it can be restored in case of catasthropic failure resulting in significant data loss. This backup however is not intended to restore old versions of user data or to restore (accidentaly) deleted files.
The HOME filesystem is realized as Lustre parallel filesystem and is available on all login and computational nodes.
Default stripe size is 1MB, stripe count is 1. There are 22 OSTs dedicated for the HOME filesystem.
!!! Note "Note"
Setting stripe size and stripe count correctly for your needs may significantly impact the I/O performance you experience.
| HOME filesystem | |
| -------------------- | ------ |
| Mountpoint | /home |
| Capacity | 320 TB |
| Throughput | 2 GB/s |
| User quota | 250 GB |
| Default stripe size | 1 MB |
| Default stripe count | 1 |
| Number of OSTs | 22 |
\###SCRATCH
The SCRATCH filesystem is mounted in directory /scratch. Users may freely create subdirectories and files on the filesystem. Accessible capacity is 146TB, shared among all users. Individual users are restricted by filesystem usage quotas, set to 100TB per user. The purpose of this quota is to prevent runaway programs from filling the entire filesystem and deny service to other users. If 100TB should prove as insufficient for particular user, please contact [support](https://support.it4i.cz/rt), the quota may be lifted upon request.
!!! Note "Note"
The Scratch filesystem is intended for temporary scratch data generated during the calculation as well as for high performance access to input and output files. All I/O intensive jobs must use the SCRATCH filesystem as their working directory.
>Users are advised to save the necessary data from the SCRATCH filesystem to HOME filesystem after the calculations and clean up the scratch files.
Files on the SCRATCH filesystem that are **not accessed for more than 90 days** will be automatically **deleted**.
The SCRATCH filesystem is realized as Lustre parallel filesystem and is available from all login and computational nodes. Default stripe size is 1MB, stripe count is 1. There are 10 OSTs dedicated for the SCRATCH filesystem.
!!! Note "Note"
Setting stripe size and stripe count correctly for your needs may significantly impact the I/O performance you experience.
| SCRATCH filesystem | |
| -------------------- | -------- |
| Mountpoint | /scratch |
| Capacity | 146TB |
| Throughput | 6GB/s |
| User quota | 100TB |
| Default stripe size | 1MB |
| Default stripe count | 1 |
| Number of OSTs | 10 |
### Disk usage and quota commands
User quotas on the file systems can be checked and reviewed using following command:
```bash
$ lfs quota dir
```
Example for Lustre HOME directory:
```bash
$ lfs quota /home
Disk quotas for user user001 (uid 1234):
Filesystem kbytes quota limit grace files quota limit grace
/home 300096 0 250000000 - 2102 0 500000 -
Disk quotas for group user001 (gid 1234):
Filesystem kbytes quota limit grace files quota limit grace
/home 300096 0 0 - 2102 0 0 -
```
In this example, we view current quota size limit of 250GB and 300MB currently used by user001.
Example for Lustre SCRATCH directory:
```bash
$ lfs quota /scratch
Disk quotas for user user001 (uid 1234):
Filesystem kbytes quota limit grace files quota limit grace
/scratch 8 0 100000000000 - 3 0 0 -
Disk quotas for group user001 (gid 1234):
Filesystem kbytes quota limit grace files quota limit grace
/scratch 8 0 0 - 3 0 0 -
```
In this example, we view current quota size limit of 100TB and 8KB currently used by user001.
To have a better understanding of where the space is exactly used, you can use following command to find out.
```bash
$ du -hs dir
```
Example for your HOME directory:
```bash
$ cd /home
$ du -hs * .[a-zA-z0-9]* | grep -E "[0-9]*G|[0-9]*M" | sort -hr
258M cuda-samples
15M .cache
13M .mozilla
5,5M .eclipse
2,7M .idb_13.0_linux_intel64_app
```
This will list all directories which are having MegaBytes or GigaBytes of consumed space in your actual (in this example HOME) directory. List is sorted in descending order from largest to smallest files/directories.
To have a better understanding of previous commands, you can read manpages.
```bash
$ man lfs
```
```bash
$ man du
```
### Extended ACLs
Extended ACLs provide another security mechanism beside the standard POSIX ACLs which are defined by three entries (for owner/group/others). Extended ACLs have more than the three basic entries. In addition, they also contain a mask entry and may contain any number of named user and named group entries.
ACLs on a Lustre file system work exactly like ACLs on any Linux file system. They are manipulated with the standard tools in the standard manner. Below, we create a directory and allow a specific user access.
```bash
[vop999@login1.anselm ~]$ umask 027
[vop999@login1.anselm ~]$ mkdir test
[vop999@login1.anselm ~]$ ls -ld test
drwxr-x--- 2 vop999 vop999 4096 Nov 5 14:17 test
[vop999@login1.anselm ~]$ getfacl test
# file: test
# owner: vop999
# group: vop999
user::rwx
group::r-x
other::---
[vop999@login1.anselm ~]$ setfacl -m user:johnsm:rwx test
[vop999@login1.anselm ~]$ ls -ld test
drwxrwx---+ 2 vop999 vop999 4096 Nov 5 14:17 test
[vop999@login1.anselm ~]$ getfacl test
# file: test
# owner: vop999
# group: vop999
user::rwx
user:johnsm:rwx
group::r-x
mask::rwx
other::---
```
Default ACL mechanism can be used to replace setuid/setgid permissions on directories. Setting a default ACL on a directory (-d flag to setfacl) will cause the ACL permissions to be inherited by any newly created file or subdirectory within the directory. Refer to this page for more information on Linux ACL:
[http://www.vanemery.com/Linux/ACL/POSIX_ACL_on_Linux.html ](http://www.vanemery.com/Linux/ACL/POSIX_ACL_on_Linux.html)
## Local Filesystems
### Local Scratch
!!! Note "Note"
Every computational node is equipped with 330GB local scratch disk.
Use local scratch in case you need to access large amount of small files during your calculation.
The local scratch disk is mounted as /lscratch and is accessible to user at /lscratch/$PBS_JOBID directory.
The local scratch filesystem is intended for temporary scratch data generated during the calculation as well as for high performance access to input and output files. All I/O intensive jobs that access large number of small files within the calculation must use the local scratch filesystem as their working directory. This is required for performance reasons, as frequent access to number of small files may overload the metadata servers (MDS) of the Lustre filesystem.
!!! Note "Note"
The local scratch directory /lscratch/$PBS_JOBID will be deleted immediately after the calculation end. Users should take care to save the output data from within the jobscript.
| local SCRATCH filesystem | |
| ------------------------ | -------------------- |
| Mountpoint | /lscratch |
| Accesspoint | /lscratch/$PBS_JOBID |
| Capacity | 330GB |
| Throughput | 100MB/s |
| User quota | none |
### RAM disk
Every computational node is equipped with filesystem realized in memory, so called RAM disk.
!!! Note "Note"
Use RAM disk in case you need really fast access to your data of limited size during your calculation. Be very careful, use of RAM disk filesystem is at the expense of operational memory.
The local RAM disk is mounted as /ramdisk and is accessible to user at /ramdisk/$PBS_JOBID directory.
The local RAM disk filesystem is intended for temporary scratch data generated during the calculation as well as for high performance access to input and output files. Size of RAM disk filesystem is limited. Be very careful, use of RAM disk filesystem is at the expense of operational memory. It is not recommended to allocate large amount of memory and use large amount of data in RAM disk filesystem at the same time.
!!! Note "Note"
The local RAM disk directory /ramdisk/$PBS_JOBID will be deleted immediately after the calculation end. Users should take care to save the output data from within the jobscript.
| RAM disk | |
| ----------- | ------------------------------------------------------------------------------------------------------- |
| Mountpoint | /ramdisk |
| Accesspoint | /ramdisk/$PBS_JOBID |
| Capacity | 60GB at compute nodes without accelerator, 90GB at compute nodes with accelerator, 500GB at fat nodes |
| Throughput | over 1.5 GB/s write, over 5 GB/s read, single thread, over 10 GB/s write, over 50 GB/s read, 16 threads |
| User quota | none |
### tmp
Each node is equipped with local /tmp directory of few GB capacity. The /tmp directory should be used to work with small temporary files. Old files in /tmp directory are automatically purged.
## Summary
| Mountpoint | Usage | Protocol | Net Capacity | Throughput | Limitations | Access | Services | |
| ---------- | ------------------------- | -------- | -------------- | ---------- | ----------- | ----------------------- | --------------------------- | ------ |
| /home | home directory | Lustre | 320 TiB | 2 GB/s | Quota 250GB | Compute and login nodes | backed up | |
| /scratch | cluster shared jobs' data | Lustre | 146 TiB | 6 GB/s | Quota 100TB | Compute and login nodes | files older 90 days removed | |
| /lscratch | node local jobs' data | local | 330 GB | 100 MB/s | none | Compute nodes | purged after job ends | |
| /ramdisk | node local jobs' data | local | 60, 90, 500 GB | 5-50 GB/s | none | Compute nodes | purged after job ends | |
| /tmp | local temporary files | local | 9.5 GB | 100 MB/s | none | Compute and login nodes | auto | purged |
## CESNET Data Storage
Do not use shared filesystems at IT4Innovations as a backup for large amount of data or long-term archiving purposes.
!!! Note "Note"
The IT4Innovations does not provide storage capacity for data archiving. Academic staff and students of research institutions in the Czech Republic can use [CESNET Storage service](https://du.cesnet.cz/).
The CESNET Storage service can be used for research purposes, mainly by academic staff and students of research institutions in the Czech Republic.
User of data storage CESNET (DU) association can become organizations or an individual person who is either in the current employment relationship (employees) or the current study relationship (students) to a legal entity (organization) that meets the “Principles for access to CESNET Large infrastructure (Access Policy)”.
User may only use data storage CESNET for data transfer and storage which are associated with activities in science, research, development, the spread of education, culture and prosperity. In detail see “Acceptable Use Policy CESNET Large Infrastructure (Acceptable Use Policy, AUP)”.
The service is documented [here](https://du.cesnet.cz/en/start). For special requirements please contact directly CESNET Storage Department via e-mail [du-support(at)cesnet.cz](mailto:du-support@cesnet.cz).
The procedure to obtain the CESNET access is quick and trouble-free.
(source [https://du.cesnet.cz/](https://du.cesnet.cz/wiki/doku.php/en/start "CESNET Data Storage"))
## CESNET storage access
### Understanding CESNET storage
!!! Note "Note"
It is very important to understand the CESNET storage before uploading data. Please read <https://du.cesnet.cz/en/navody/home-migrace-plzen/start> first.
Once registered for CESNET Storage, you may [access the storage](https://du.cesnet.cz/en/navody/faq/start) in number of ways. We recommend the SSHFS and RSYNC methods.
### SSHFS Access
!!! Note "Note"
SSHFS: The storage will be mounted like a local hard drive
The SSHFS provides a very convenient way to access the CESNET Storage. The storage will be mounted onto a local directory, exposing the vast CESNET Storage as if it was a local removable hard drive. Files can be than copied in and out in a usual fashion.
First, create the mount point
```bash
$ mkdir cesnet
```
Mount the storage. Note that you can choose among the ssh.du1.cesnet.cz (Plzen), ssh.du2.cesnet.cz (Jihlava), ssh.du3.cesnet.cz (Brno) Mount tier1_home **(only 5120M !)**:
```bash
$ sshfs username@ssh.du1.cesnet.cz:. cesnet/
```
For easy future access from Anselm, install your public key
```bash
$ cp .ssh/id_rsa.pub cesnet/.ssh/authorized_keys
```
Mount tier1_cache_tape for the Storage VO:
```bash
$ sshfs username@ssh.du1.cesnet.cz:/cache_tape/VO_storage/home/username cesnet/
```
View the archive, copy the files and directories in and out
```bash
$ ls cesnet/
$ cp -a mydir cesnet/.
$ cp cesnet/myfile .
```
Once done, please remember to unmount the storage
```bash
$ fusermount -u cesnet
```
### Rsync access
!!! Note "Note"
Rsync provides delta transfer for best performance, can resume interrupted transfers
Rsync is a fast and extraordinarily versatile file copying tool. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated.
More about Rsync at <https://du.cesnet.cz/en/navody/rsync/start#pro_bezne_uzivatele>
Transfer large files to/from CESNET storage, assuming membership in the Storage VO
```bash
$ rsync --progress datafile username@ssh.du1.cesnet.cz:VO_storage-cache_tape/.
$ rsync --progress username@ssh.du1.cesnet.cz:VO_storage-cache_tape/datafile .
```
Transfer large directories to/from CESNET storage, assuming membership in the Storage VO
```bash
$ rsync --progress -av datafolder username@ssh.du1.cesnet.cz:VO_storage-cache_tape/.
$ rsync --progress -av username@ssh.du1.cesnet.cz:VO_storage-cache_tape/datafolder .
```
Transfer rates of about 28 MB/s can be expected.
docs.it4i/get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/graphical-user-interface.md 0 → 100644
View file @ 14676d74
# Graphical User Interface
## X Window System
The X Window system is a principal way to get GUI access to the clusters.
Read more about configuring [**X Window System**](x-window-system/).
## VNC
The **Virtual Network Computing** (**VNC**) is a graphical [desktop sharing](http://en.wikipedia.org/wiki/Desktop_sharing "Desktop sharing") system that uses the [Remote Frame Buffer protocol (RFB)](http://en.wikipedia.org/wiki/RFB_protocol "RFB protocol") to remotely control another [computer](http://en.wikipedia.org/wiki/Computer "Computer").
Read more about configuring **[VNC](vnc/)**.
docs.it4i/get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/vnc.md 0 → 100644
View file @ 14676d74
# VNC
The **Virtual Network Computing** (**VNC**) is a graphical [desktop sharing](http://en.wikipedia.org/wiki/Desktop_sharing "Desktop sharing") system that uses the [Remote Frame Buffer protocol (RFB)](http://en.wikipedia.org/wiki/RFB_protocol "RFB protocol") to remotely control another [computer](http://en.wikipedia.org/wiki/Computer "Computer"). It transmits the [keyboard](http://en.wikipedia.org/wiki/Computer_keyboard "Computer keyboard") and [mouse](http://en.wikipedia.org/wiki/Computer_mouse") events from one computer to another, relaying the graphical [screen](http://en.wikipedia.org/wiki/Computer_screen "Computer screen") updates back in the other direction, over a [network](http://en.wikipedia.org/wiki/Computer_network "Computer network").
The recommended clients are [TightVNC](http://www.tightvnc.com) or [TigerVNC](http://sourceforge.net/apps/mediawiki/tigervnc/index.php?title=Main_Page) (free, open source, available for almost any platform).
## Create VNC password
!!! Note "Note"
Local VNC password should be set before the first login. Do use a strong password.
```bash
[username@login2 ~]$ vncpasswd
Password:
Verify:
```
## Start vncserver
!!! Note "Note"
To access VNC a local vncserver must be started first and also a tunnel using SSH port forwarding must be established.
[See below](vnc.md#linux-example-of-creating-a-tunnel) for the details on SSH tunnels. In this example we use port 61.
You can find ports which are already occupied. Here you can see that ports " /usr/bin/Xvnc :79" and " /usr/bin/Xvnc :60" are occupied.
```bash
[username@login2 ~]$ ps aux | grep Xvnc
username 5971 0.0 0.0 201072 92564 ? SN Sep22 4:19 /usr/bin/Xvnc :79 -desktop login2:79 (username) -auth /home/gre196/.Xauthority -geometry 1024x768 -rfbwait 30000 -rfbauth /home/username/.vnc/passwd -rfbport 5979 -fp catalogue:/etc/X11/fontpath.d -pn
username 10296 0.0 0.0 131772 21076 pts/29 SN 13:01 0:01 /usr/bin/Xvnc :60 -desktop login2:61 (username) -auth /home/username/.Xauthority -geometry 1600x900 -depth 16 -rfbwait 30000 -rfbauth /home/jir13/.vnc/passwd -rfbport 5960 -fp catalogue:/etc/X11/fontpath.d -pn
.....
```
Choose free port e.g. 61 and start your VNC server:
```bash
[username@login2 ~]$ vncserver :61 -geometry 1600x900 -depth 16
New 'login2:1 (username)' desktop is login2:1
Starting applications specified in /home/username/.vnc/xstartup
Log file is /home/username/.vnc/login2:1.log
```
Check if VNC server is started on the port (in this example 61):
```bash
[username@login2 .vnc]$ vncserver -list
TigerVNC server sessions:
X DISPLAY # PROCESS ID
:61 18437
```
Another command:
```bash
[username@login2 .vnc]$ ps aux | grep Xvnc
username 10296 0.0 0.0 131772 21076 pts/29 SN 13:01 0:01 /usr/bin/Xvnc :61 -desktop login2:61 (username) -auth /home/jir13/.Xauthority -geometry 1600x900 -depth 16 -rfbwait 30000 -rfbauth /home/username/.vnc/passwd -rfbport 5961 -fp catalogue:/etc/X11/fontpath.d -pn
```
To access the VNC server you have to create a tunnel between the login node using TCP **port 5961** and your machine using a free TCP port (for simplicity the very same, in this case).
!!! Note "Note"
The tunnel must point to the same login node where you launched the VNC server, eg. login2. If you use just cluster-name.it4i.cz, the tunnel might point to a different node due to DNS round robin.
## Linux/Mac OS example of creating a tunnel
At your machine, create the tunnel:
```bash
local $ ssh -TN -f username@login2.cluster-name.it4i.cz -L 5961:localhost:5961
```
Issue the following command to check the tunnel is established (please note the PID 2022 in the last column, you'll need it for closing the tunnel):
```bash
local $ netstat -natp | grep 5961
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 127.0.0.1:5961 0.0.0.0:* LISTEN 2022/ssh
tcp6 0 0 ::1:5961 :::* LISTEN 2022/ssh
```
Or on Mac OS use this command:
```bash
local-mac $ lsof -n -i4TCP:5961 | grep LISTEN
ssh 75890 sta545 7u IPv4 0xfb062b5c15a56a3b 0t0 TCP 127.0.0.1:5961 (LISTEN)
```
Connect with the VNC client:
```bash
local $ vncviewer 127.0.0.1:5961
```
In this example, we connect to VNC server on port 5961, via the ssh tunnel. The connection is encrypted and secured. The VNC server listening on port 5961 provides screen of 1600x900 pixels.
You have to destroy the SSH tunnel which is still running at the background after you finish the work. Use the following command (PID 2022 in this case, see the netstat command above):
```bash
kill 2022
```
## Windows example of creating a tunnel
Use PuTTY to log in on cluster.
Start vncserver using command vncserver described above.
Search for the localhost and port number (in this case 127.0.0.1:5961).
```bahs
[username@login2 .vnc]$ netstat -tanp | grep Xvnc
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 127.0.0.1:5961 0.0.0.0:* LISTEN 24031/Xvnc
```
On the PuTTY Configuration screen go to Connection->SSH->Tunnels to set up the tunnel.
Fill the Source port and Destination fields. **Do not forget to click the Add button**.
![](../../../img/putty-tunnel.png)
Run the VNC client of your choice, select VNC server 127.0.0.1, port 5961 and connect using VNC password.
## Example of starting TigerVNC viewer
![](../../../img/vncviewer.png)
In this example, we connect to VNC server on port 5961, via the ssh tunnel, using TigerVNC viewer. The connection is encrypted and secured. The VNC server listening on port 5961 provides screen of 1600x900 pixels.
## Example of starting TightVNC Viewer
Use your VNC password to log using TightVNC Viewer and start a Gnome Session on the login node.
![](../../../img/TightVNC_login.png)
## Gnome session
You should see after the successful login.
![](../../../img/gnome_screen.png)
## Disable your Gnome session screensaver
Open Screensaver preferences dialog:
![](../../../img/gdmscreensaver.png)
Uncheck both options below the slider:
![](../../../img/gdmdisablescreensaver.png)
## Kill screensaver if locked screen
If the screen gets locked you have to kill the screensaver. Do not to forget to disable the screensaver then.
```bash
[username@login2 .vnc]$ ps aux | grep screen
username 1503 0.0 0.0 103244 892 pts/4 S+ 14:37 0:00 grep screen
username 24316 0.0 0.0 270564 3528 ? Ss 14:12 0:00 gnome-screensaver
[username@login2 .vnc]$ kill 24316
```
## Kill vncserver after finished work
You should kill your VNC server using command:
```bash
[username@login2 .vnc]$ vncserver -kill :61
Killing Xvnc process ID 7074
Xvnc process ID 7074 already killed
```
Or this way:
```bash
[username@login2 .vnc]$ pkill vnc
```
## GUI applications on compute nodes over VNC
The very same methods as described above, may be used to run the GUI applications on compute nodes. However, for maximum performance, proceed following these steps:
Open a Terminal (Applications -> System Tools -> Terminal). Run all the next commands in the terminal.
![](../../../img/gnome-terminal.png)
Allow incoming X11 graphics from the compute nodes at the login node:
```bash
$ xhost +
```
Get an interactive session on a compute node (for more detailed info [look here](../../../anselm-cluster-documentation/resource-allocation-and-job-execution/job-submission-and-execution/)). Use the **-v DISPLAY** option to propagate the DISPLAY on the compute node. In this example, we want a complete node (24 cores in this example) from the production queue:
```bash
$ qsub -I -v DISPLAY=$(uname -n):$(echo $DISPLAY | cut -d ':' -f 2) -A PROJECT_ID -q qprod -l select=1:ncpus=24
```
Test that the DISPLAY redirection into your VNC session works, by running a X11 application (e. g. XTerm) on the assigned compute node:
```bash
$ xterm
```
Example described above:
![](../../../img/gnome-compute-nodes-over-vnc.png)
docs.it4i/get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-system.md 0 → 100644
View file @ 14676d74
# X Window System
The X Window system is a principal way to get GUI access to the clusters. The **X Window System** (commonly known as **X11**, based on its current major version being 11, or shortened to simply **X**, and sometimes informally **X-Windows**) is a computer software system and network [protocol](http://en.wikipedia.org/wiki/Protocol_%28computing%29 "Protocol (computing)") that provides a basis for [graphical user interfaces](http://en.wikipedia.org/wiki/Graphical_user_interface "Graphical user interface") (GUIs) and rich input device capability for [networked computers](http://en.wikipedia.org/wiki/Computer_network "Computer network").
!!! tip
The X display forwarding must be activated and the X server running on client side
## X display
In order to display graphical user interface GUI of various software tools, you need to enable the X display forwarding. On Linux and Mac, log in using the -X option tho ssh client:
```bash
local $ ssh -X username@cluster-name.it4i.cz
```
## X Display Forwarding on Windows
On Windows use the PuTTY client to enable X11 forwarding. In PuTTY menu, go to Connection-SSH-X11, mark the Enable X11 forwarding checkbox before logging in. Then log in as usual.
To verify the forwarding, type
```bash
$ echo $DISPLAY
```
if you receive something like
```bash
localhost:10.0
```
then the X11 forwarding is enabled.
## X Server
In order to display graphical user interface GUI of various software tools, you need running X server on your desktop computer. For Linux users, no action is required as the X server is the default GUI environment on most Linux distributions. Mac and Windows users need to install and run the X server on their workstations.
## X Server on OS X
Mac OS users need to install [XQuartz server](https://www.xquartz.org).
## X Server on Windows
There are variety of X servers available for Windows environment. The commercial Xwin32 is very stable and rich featured. The Cygwin environment provides fully featured open-source XWin X server. For simplicity, we recommend open-source X server by the [Xming project](http://sourceforge.net/projects/xming/). For stability and full features we recommend the
[XWin](http://x.cygwin.com/) X server by Cygwin
| How to use Xwin | How to use Xming |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| [Install Cygwin](http://x.cygwin.com/) Find and execute XWin.exeto start the X server on Windows desktop computer.[If no able to forward X11 using PuTTY to CygwinX](#if-no-able-to-forward-x11-using-putty-to-cygwinx) | <p>Use Xlaunch to configure the Xming.<p>Run Xmingto start the X server on Windows desktop computer. |
Read more on [http://www.math.umn.edu/systems_guide/putty_xwin32.html](http://www.math.umn.edu/systems_guide/putty_xwin32.shtml)
## Running GUI Enabled Applications
!!! Note "Note"
Make sure that X forwarding is activated and the X server is running.
Then launch the application as usual. Use the & to run the application in background.
```bash
$ module load intel (idb and gvim not installed yet)
$ gvim &
```
```bash
$ xterm
```
In this example, we activate the intel programing environment tools, then start the graphical gvim editor.
## GUI Applications on Compute Nodes
Allocate the compute nodes using -X option on the qsub command
```bash
$ qsub -q qexp -l select=2:ncpus=24 -X -I
```
In this example, we allocate 2 nodes via qexp queue, interactively. We request X11 forwarding with the -X option. It will be possible to run the GUI enabled applications directly on the first compute node.
**Better performance** is obtained by logging on the allocated compute node via ssh, using the -X option.
```bash
$ ssh -X r24u35n680
```
In this example, we log in on the r24u35n680 compute node, with the X11 forwarding enabled.
## The Gnome GUI Environment
The Gnome 2.28 GUI environment is available on the clusters. We recommend to use separate X server window for displaying the Gnome environment.
## Gnome on Linux and OS X
To run the remote Gnome session in a window on Linux/OS X computer, you need to install Xephyr. Ubuntu package is
xserver-xephyr, on OS X it is part of [XQuartz](http://xquartz.macosforge.org/landing/). First, launch Xephyr on local machine:
```bash
local $ Xephyr -ac -screen 1024x768 -br -reset -terminate :1 &
```
This will open a new X window with size 1024 x 768 at DISPLAY :1. Next, ssh to the cluster with DISPLAY environment variable set and launch gnome-session
```bash
local $ DISPLAY=:1.0 ssh -XC yourname@cluster-name.it4i.cz -i ~/.ssh/path_to_your_key
... cluster-name MOTD...
yourname@login1.cluster-namen.it4i.cz $ gnome-session &
```
On older systems where Xephyr is not available, you may also try Xnest instead of Xephyr. Another option is to launch a new X server in a separate console, via:
```bash
xinit /usr/bin/ssh -XT -i .ssh/path_to_your_key yourname@cluster-namen.it4i.cz gnome-session -- :1 vt12
```
However this method does not seem to work with recent Linux distributions and you will need to manually source
/etc/profile to properly set environment variables for PBS.
## Gnome on Windows
Use Xlaunch to start the Xming server or run the XWin.exe. Select the "One window" mode.
Log in to the cluster, using PuTTY. On the cluster, run the gnome-session command.
```bash
$ gnome-session &
```
In this way, we run remote gnome session on the cluster, displaying it in the local X server
Use System-Log Out to close the gnome-session
### If no able to forward X11 using PuTTY to CygwinX
```bash
[usename@login1.anselm ~]$ gnome-session &
[1] 23691
[usename@login1.anselm ~]$ PuTTY X11 proxy: unable to connect to forwarded X server: Network error: Connection refused
PuTTY X11 proxy: unable to connect to forwarded X server: Network error: Connection refused
(gnome-session:23691): WARNING **: Cannot open display:**
```
1. Locate and modify Cygwin shortcut that uses [startxwin](http://x.cygwin.com/docs/man1/startxwin.1.html)
locate
C:cygwin64binXWin.exe
change it
to
C:_cygwin64binXWin.exe -listen tcp_
![XWin-listen-tcp.png](../../../img/XWinlistentcp.png "XWin-listen-tcp.png")
1. Check Putty settings:
Enable X11 forwarding
![](../../../img/cygwinX11forwarding.png)
docs.it4i/get-started-with-it4innovations/accessing-the-clusters/introduction.md 0 → 100644
View file @ 14676d74
# Accessing the Clusters
The IT4Innovations clusters are accessed by SSH protocol via login nodes.
!!! Note "Note"
Read more on [Accessing the Salomon Cluster](../../salomon/shell-and-data-access.md) or [Accessing the Anselm Cluster](../../anselm-cluster-documentation/shell-and-data-access.md) pages.
## PuTTY
On **Windows**, use [PuTTY ssh client](shell-access-and-data-transfer/putty/).
## SSH keys
Read more about [SSH keys management](shell-access-and-data-transfer/ssh-keys/).
## Graphical User Interface
Read more about [X Window System](./graphical-user-interface/x-window-system/).
Read more about [Virtual Network Computing (VNC)](./graphical-user-interface/vnc/).
## Accessing IT4Innovations internal resources via VPN
Read more about [VPN Access](vpn-access/).