From 0724ba63ff2a1bb2859ada52d2430e63a159dc61 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Luk=C3=A1=C5=A1=20Krup=C4=8D=C3=ADk?= <lukas.krupcik@vsb.cz>
Date: Fri, 26 Aug 2016 21:19:34 +0200
Subject: [PATCH] salomon
---
.../shell-and-data-access.md | 110 ++
.../software/ansys/ansys-cfx.md | 4 +-
.../software/ansys/ansys-fluent.md | 5 +-
.../software/ansys/ansys-ls-dyna.md | 7 +-
.../software/ansys/ansys-mechanical-apdl.md | 5 +-
.../software/chemistry/molpro.md | 4 +-
.../software/chemistry/nwchem.md | 5 +-
.../job-priority.md | 79 +-
.../job-submission-and-execution.md | 430 +++----
.../resources-allocation-policy.md | 181 +--
docs.it4i/salomon/software/ansys/ansys-cfx.md | 116 +-
.../salomon/software/ansys/ansys-fluent.md | 212 ++--
.../salomon/software/ansys/ansys-ls-dyna.md | 110 +-
.../software/ansys/ansys-mechanical-apdl.md | 133 +-
...ys-products-mechanical-fluent-cfx-mapdl.md | 31 -
docs.it4i/salomon/software/ansys/ansys.md | 32 +-
docs.it4i/salomon/software/ansys/licensing.md | 34 +-
.../ansys/setting-license-preferences.md | 24 +-
docs.it4i/salomon/software/ansys/workbench.md | 33 +-
.../salomon/software/chemistry/molpro.md | 86 +-
.../salomon/software/chemistry/nwchem.md | 51 +-
.../salomon/software/chemistry/phono3py.md | 126 +-
docs.it4i/salomon/software/compilers.md | 96 +-
.../software/comsol/comsol-multiphysics.md | 156 +--
.../licensing-and-available-versions.md | 31 +-
docs.it4i/salomon/software/debuggers.md | 90 --
.../software/debuggers/Introduction.md | 72 ++
.../salomon/software/debuggers/aislinn.md | 120 +-
.../salomon/software/debuggers/allinea-ddt.md | 209 +--
.../salomon/software/debuggers/debuggers.md | 61 +
.../debuggers/intel-vtune-amplifier.md | 111 +-
.../salomon/software/debuggers/summary.md | 87 --
.../salomon/software/debuggers/total-view.md | 90 +-
.../salomon/software/debuggers/valgrind.md | 168 +--
.../salomon/software/debuggers/vampir.md | 20 +-
.../software/intel-suite/intel-advisor.md | 46 +-
.../software/intel-suite/intel-compilers.md | 66 +-
.../software/intel-suite/intel-debugger.md | 63 +-
.../software/intel-suite/intel-inspector.md | 51 +-
...intel-integrated-performance-primitives.md | 46 +-
.../salomon/software/intel-suite/intel-mkl.md | 172 +--
.../intel-parallel-studio-introduction.md | 88 +-
.../salomon/software/intel-suite/intel-tbb.md | 41 +-
.../intel-trace-analyzer-and-collector.md | 36 +-
docs.it4i/salomon/software/intel-xeon-phi.md | 1127 +++++++----------
docs.it4i/salomon/software/java.md | 27 +-
docs.it4i/salomon/software/mpi-1/mpi.md | 185 ---
.../{mpi-1 => mpi}/Running_OpenMPI.md | 137 +-
docs.it4i/salomon/software/mpi/mpi.md | 142 +++
.../{mpi-1 => mpi}/mpi4py-mpi-for-python.md | 51 +-
.../numerical-languages/introduction.md | 36 +-
.../software/numerical-languages/matlab.md | 219 ++--
.../software/numerical-languages/octave.md | 64 +-
.../salomon/software/numerical-languages/r.md | 207 ++-
.../salomon/software/operating-system.md | 11 +-
.../salomon/storage/cesnet-data-storage.md | 103 +-
docs.it4i/salomon/storage/storage.md | 487 +++----
57 files changed, 2388 insertions(+), 4146 deletions(-)
create mode 100644 docs.it4i/anselm-cluster-documentation/accessing-the-cluster/shell-and-data-access.md
delete mode 100644 docs.it4i/salomon/software/ansys/ansys-products-mechanical-fluent-cfx-mapdl.md
delete mode 100644 docs.it4i/salomon/software/debuggers.md
create mode 100644 docs.it4i/salomon/software/debuggers/Introduction.md
create mode 100644 docs.it4i/salomon/software/debuggers/debuggers.md
delete mode 100644 docs.it4i/salomon/software/debuggers/summary.md
delete mode 100644 docs.it4i/salomon/software/mpi-1/mpi.md
rename docs.it4i/salomon/software/{mpi-1 => mpi}/Running_OpenMPI.md (63%)
create mode 100644 docs.it4i/salomon/software/mpi/mpi.md
rename docs.it4i/salomon/software/{mpi-1 => mpi}/mpi4py-mpi-for-python.md (62%)
diff --git a/docs.it4i/anselm-cluster-documentation/accessing-the-cluster/shell-and-data-access.md b/docs.it4i/anselm-cluster-documentation/accessing-the-cluster/shell-and-data-access.md
new file mode 100644
index 000000000..18d077a50
--- /dev/null
+++ b/docs.it4i/anselm-cluster-documentation/accessing-the-cluster/shell-and-data-access.md
@@ -0,0 +1,110 @@
+Shell access and data transfer
+==============================
+
+Interactive Login
+-----------------
+The Anselm cluster is accessed by SSH protocol via login nodes login1 and login2 at address anselm.it4i.cz. The login nodes may be addressed specifically, by prepending the login node name to the address.
+
+|Login address|Port|Protocol|Login node|
+|---|---|
+|anselm.it4i.cz|22|ssh|round-robin DNS record for login1 and login2|
+|login1.anselm.it4i.cz|22|ssh|login1|
+|login2.anselm.it4i.cz|22|ssh|login2|
+
+The authentication is by the [private key](../../../get-started-with-it4innovations/accessing-the-clusters/shell-access-and-data-transfer/ssh-keys.html)
+
+>Please verify SSH fingerprints during the first logon. They are identical on all login nodes:
+29:b3:f4:64:b0:73:f5:6f:a7:85:0f:e0:0d:be:76:bf (DSA)
+d4:6f:5c:18:f4:3f:70:ef:bc:fc:cc:2b:fd:13:36:b7Â (RSA)
+
+Private key authentication:
+
+On **Linux** or **Mac**, use
+
+```bash
+local $ ssh -i /path/to/id_rsa username@anselm.it4i.cz
+```
+
+If you see warning message "UNPROTECTED PRIVATE KEY FILE!", use this command to set lower permissions to private key file.
+
+```bash
+local $ chmod 600 /path/to/id_rsa
+```
+
+On **Windows**, use [PuTTY ssh client](../../../get-started-with-it4innovations/accessing-the-clusters/shell-access-and-data-transfer/putty/putty.html).
+
+After logging in, you will see the command prompt:
+
+```bash
+ _
+ /\ | |
+ / \ _ __ ___ ___| |_ __ ___
+ / /\ \ | '_ \/ __|/ _ \ | '_ ` _ \
+ / ____ \| | | \__ \ __/ | | | | | |
+ /_/ \_\_| |_|___/\___|_|_| |_| |_|
+
+
+ http://www.it4i.cz/?lang=en
+
+Last login: Tue Jul 9 15:57:38 2013 from your-host.example.com
+[username@login2.anselm ~]$
+```
+
+>The environment is **not** shared between login nodes, except for [shared filesystems](../storage-1.html#section-1).
+
+Data Transfer
+-------------
+Data in and out of the system may be transferred by the [scp](http://en.wikipedia.org/wiki/Secure_copy) and sftp protocols. (Not available yet.) In case large volumes of data are transferred, use dedicated data mover node dm1.anselm.it4i.cz for increased performance.
+
+|Address|Port|Protocol|
+|---|---|
+|anselm.it4i.cz|22|scp, sftp|
+|login1.anselm.it4i.cz|22|scp, sftp|
+|login2.anselm.it4i.cz|22|scp, sftp|
+|dm1.anselm.it4i.cz|22|scp, sftp|
+
+The authentication is by the [private key](../../../get-started-with-it4innovations/accessing-the-clusters/shell-access-and-data-transfer/ssh-keys.html)
+
+>Data transfer rates up to **160MB/s** can be achieved with scp or sftp.
+1TB may be transferred in 1:50h.
+
+To achieve 160MB/s transfer rates, the end user must be connected by 10G line all the way to IT4Innovations and use computer with fast processor for the transfer. Using Gigabit ethernet connection, up to 110MB/s may be expected. Fast cipher (aes128-ctr) should be used.
+
+>If you experience degraded data transfer performance, consult your local network provider.
+
+On linux or Mac, use scp or sftp client to transfer the data to Anselm:
+
+```bash
+local $ scp -i /path/to/id_rsa my-local-file username@anselm.it4i.cz:directory/file
+```
+
+```bash
+local $ scp -i /path/to/id_rsa -r my-local-dir username@anselm.it4i.cz:directory
+```
+
+or
+
+```bash
+local $ sftp -o IdentityFile=/path/to/id_rsa username@anselm.it4i.cz
+```
+
+Very convenient way to transfer files in and out of the Anselm computer is via the fuse filesystem [sshfs](http://linux.die.net/man/1/sshfs)
+
+```bash
+local $ sshfs -o IdentityFile=/path/to/id_rsa username@anselm.it4i.cz:. mountpoint
+```
+
+Using sshfs, the users Anselm home directory will be mounted on your local computer, just like an external disk.
+
+Learn more on ssh, scp and sshfs by reading the manpages
+
+```bash
+$ man ssh
+$ man scp
+$ man sshfs
+```
+
+On Windows, use [WinSCP client](http://winscp.net/eng/download.php) to transfer the data. The [win-sshfs client](http://code.google.com/p/win-sshfs/) provides a way to mount the Anselm filesystems directly as an external disc.
+
+More information about the shared file systems is available [here](../../storage.html).
+
diff --git a/docs.it4i/anselm-cluster-documentation/software/ansys/ansys-cfx.md b/docs.it4i/anselm-cluster-documentation/software/ansys/ansys-cfx.md
index 681bcea98..f5cf8a8bf 100644
--- a/docs.it4i/anselm-cluster-documentation/software/ansys/ansys-cfx.md
+++ b/docs.it4i/anselm-cluster-documentation/software/ansys/ansys-cfx.md
@@ -49,9 +49,9 @@ echo Machines: $hl
/ansys_inc/v145/CFX/bin/cfx5solve -def input.def -size 4 -size-ni 4x -part-large -start-method "Platform MPI Distributed Parallel" -par-dist $hl -P aa_r
```
-Header of the pbs file (above) is common and description can be find on [this site](../../resource-allocation-and-job-execution/job-submission-and-execution.html). SVS FEM recommends to utilize sources by keywords: nodes, ppn. These keywords allows to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources.
+Header of the pbs file (above) is common and description can be find on [this site](../../resource-allocation-and-job-execution/job-submission-and-execution.md). SVS FEM recommends to utilize sources by keywords: nodes, ppn. These keywords allows to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources.
Working directory has to be created before sending pbs job into the queue. Input file should be in working directory or full path to input file has to be specified. >Input file has to be defined by common CFX def file which is attached to the cfx solver via parameter -def
**License**Â should be selected by parameter -P (Big letter **P**). Licensed products are the following: aa_r (ANSYSÂ **Academic**Â Research), ane3fl (ANSYS Multiphysics)-**Commercial**.
-[More about licensing here](licensing.html)
\ No newline at end of file
+[More about licensing here](licensing.md)
\ No newline at end of file
diff --git a/docs.it4i/anselm-cluster-documentation/software/ansys/ansys-fluent.md b/docs.it4i/anselm-cluster-documentation/software/ansys/ansys-fluent.md
index 63c40c1ec..458ac3295 100644
--- a/docs.it4i/anselm-cluster-documentation/software/ansys/ansys-fluent.md
+++ b/docs.it4i/anselm-cluster-documentation/software/ansys/ansys-fluent.md
@@ -39,7 +39,7 @@ NCORES=`wc -l $PBS_NODEFILE |awk '{print $1}'`
/ansys_inc/v145/fluent/bin/fluent 3d -t$NCORES -cnf=$PBS_NODEFILE -g -i fluent.jou
```
-Header of the pbs file (above) is common and description can be find on [this site](../../resource-allocation-and-job-execution/job-submission-and-execution.html). [SVS FEM](http://www.svsfem.cz) recommends to utilize sources by keywords: nodes, ppn. These keywords allows to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources.
+Header of the pbs file (above) is common and description can be find on [this site](../../resource-allocation-and-job-execution/job-submission-and-execution.md). [SVS FEM](http://www.svsfem.cz) recommends to utilize sources by keywords: nodes, ppn. These keywords allows to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources.
Working directory has to be created before sending pbs job into the queue. Input file should be in working directory or full path to input file has to be specified. Input file has to be defined by common Fluent journal file which is attached to the Fluent solver via parameter -i fluent.jou
@@ -84,8 +84,7 @@ input is the name of the input file.
case is the name of the .cas file that the input file will utilize.
-fluent_args are extra ANSYS FLUENT arguments. As shown in the previous example, you can specify the interconnect by using the -p interconnect command. The available interconnects include ethernet (the default), myrinet, infiniband, vendor,
-altix, and crayx. The MPI is selected automatically, based on the specified interconnect.
+fluent_args are extra ANSYS FLUENT arguments. As shown in the previous example, you can specify the interconnect by using the -p interconnect command. The available interconnects include ethernet (the default), myrinet, infiniband, vendor, altix, and crayx. The MPI is selected automatically, based on the specified interconnect.
outfile is the name of the file to which the standard output will be sent.
diff --git a/docs.it4i/anselm-cluster-documentation/software/ansys/ansys-ls-dyna.md b/docs.it4i/anselm-cluster-documentation/software/ansys/ansys-ls-dyna.md
index 4949412e1..c2b188646 100644
--- a/docs.it4i/anselm-cluster-documentation/software/ansys/ansys-ls-dyna.md
+++ b/docs.it4i/anselm-cluster-documentation/software/ansys/ansys-ls-dyna.md
@@ -51,9 +51,6 @@ echo Machines: $hl
/ansys_inc/v145/ansys/bin/ansys145 -dis -lsdynampp i=input.k -machines $hl
```
-Header of the pbs file (above) is common and description can be find on [this site](../../resource-allocation-and-job-execution/job-submission-and-execution.html). [SVS FEM](http://www.svsfem.cz) recommends to utilize sources by keywords: nodes, ppn. These keywords allows to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources.
-
-Working directory has to be created before sending pbs job into the queue. Input file should be in working directory or full path to input file has to be specified. Input file has to be defined by common LS-DYNA .**k** file which is attached to the ansys solver via parameter i=
-
-Â
+Header of the pbs file (above) is common and description can be find on [this site](../../resource-allocation-and-job-execution/job-submission-and-execution.md). [SVS FEM](http://www.svsfem.cz) recommends to utilize sources by keywords: nodes, ppn. These keywords allows to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources.
+Working directory has to be created before sending pbs job into the queue. Input file should be in working directory or full path to input file has to be specified. Input file has to be defined by common LS-DYNA .**k** file which is attached to the ansys solver via parameter i=
\ No newline at end of file
diff --git a/docs.it4i/anselm-cluster-documentation/software/ansys/ansys-mechanical-apdl.md b/docs.it4i/anselm-cluster-documentation/software/ansys/ansys-mechanical-apdl.md
index a84a1bf99..56d5f78aa 100644
--- a/docs.it4i/anselm-cluster-documentation/software/ansys/ansys-mechanical-apdl.md
+++ b/docs.it4i/anselm-cluster-documentation/software/ansys/ansys-mechanical-apdl.md
@@ -50,10 +50,9 @@ echo Machines: $hl
/ansys_inc/v145/ansys/bin/ansys145 -b -dis -p aa_r -i input.dat -o file.out -machines $hl -dir $WORK_DIR
```
-Header of the pbs file (above) is common and description can be find on [this site](../../resource-allocation-and-job-execution/job-submission-and-execution.html). [SVS FEM](http://www.svsfem.cz) recommends to utilize sources by keywords: nodes, ppn. These keywords allows to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources.
+Header of the pbs file (above) is common and description can be find on [this site](../../resource-allocation-and-job-execution/job-submission-and-execution.md). [SVS FEM](http://www.svsfem.cz) recommends to utilize sources by keywords: nodes, ppn. These keywords allows to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources.
Working directory has to be created before sending pbs job into the queue. Input file should be in working directory or full path to input file has to be specified. Input file has to be defined by common APDL file which is attached to the ansys solver via parameter -i
**License** should be selected by parameter -p. Licensed products are the following: aa_r (ANSYS **Academic** Research), ane3fl (ANSYS Multiphysics)-**Commercial**, aa_r_dy (ANSYS **Academic** AUTODYN)
-[More about licensing here](licensing.html)
-
+[More about licensing here](licensing.md)
\ No newline at end of file
diff --git a/docs.it4i/anselm-cluster-documentation/software/chemistry/molpro.md b/docs.it4i/anselm-cluster-documentation/software/chemistry/molpro.md
index 55f052e49..f993cf58b 100644
--- a/docs.it4i/anselm-cluster-documentation/software/chemistry/molpro.md
+++ b/docs.it4i/anselm-cluster-documentation/software/chemistry/molpro.md
@@ -35,7 +35,7 @@ Molpro is compiled for parallel execution using MPI and OpenMP. By default, Molp
>The OpenMP parallelization in Molpro is limited and has been observed to produce limited scaling. We therefore recommend to use MPI parallelization only. This can be achieved by passing option mpiprocs=16:ompthreads=1 to PBS.
-You are advised to use the -d option to point to a directory in [SCRATCH filesystem](../../storage.html). Molpro can produce a large amount of temporary data during its run, and it is important that these are placed in the fast scratch filesystem.
+You are advised to use the -d option to point to a directory in [SCRATCH filesystem](../../storage.md). Molpro can produce a large amount of temporary data during its run, and it is important that these are placed in the fast scratch filesystem.
### Example jobscript
@@ -60,4 +60,4 @@ You are advised to use the -d option to point to a directory in [SCRATCH filesys
# delete scratch directory
rm -rf /scratch/$USER/$PBS_JOBID
-```
+```
\ No newline at end of file
diff --git a/docs.it4i/anselm-cluster-documentation/software/chemistry/nwchem.md b/docs.it4i/anselm-cluster-documentation/software/chemistry/nwchem.md
index 9869c37e9..c079a1683 100644
--- a/docs.it4i/anselm-cluster-documentation/software/chemistry/nwchem.md
+++ b/docs.it4i/anselm-cluster-documentation/software/chemistry/nwchem.md
@@ -1,7 +1,7 @@
NWChem
======
-##High-Performance Computational Chemistry
+**High-Performance Computational Chemistry**
Introduction
-------------------------
@@ -43,5 +43,4 @@ Options
Please refer to [the documentation](http://www.nwchem-sw.org/index.php/Release62:Top-level)Â and in the input file set the following directives :
- MEMORY : controls the amount of memory NWChem will use
-- SCRATCH_DIR : set this to a directory in [SCRATCH filesystem](../../storage.html#scratch) (or run the calculation completely in a scratch directory). For certain calculations, it might be advisable to reduce I/O by forcing "direct" mode, eg. "scf direct"
-
+- SCRATCH_DIR : set this to a directory in [SCRATCH filesystem](../../storage.md#scratch) (or run the calculation completely in a scratch directory). For certain calculations, it might be advisable to reduce I/O by forcing "direct" mode, eg. "scf direct"
\ No newline at end of file
diff --git a/docs.it4i/salomon/resource-allocation-and-job-execution/job-priority.md b/docs.it4i/salomon/resource-allocation-and-job-execution/job-priority.md
index 43919cced..43c44f123 100644
--- a/docs.it4i/salomon/resource-allocation-and-job-execution/job-priority.md
+++ b/docs.it4i/salomon/resource-allocation-and-job-execution/job-priority.md
@@ -1,14 +1,11 @@
-Job scheduling
+Job scheduling
==============
Job execution priority
----------------------
+Scheduler gives each job an execution priority and then uses this job execution priority to select which job(s) to run.
-Scheduler gives each job an execution priority and then uses this job
-execution priority to select which job(s) to run.
-
-Job execution priority is determined by these job properties (in order
-of importance):
+Job execution priority is determined by these job properties (in order of importance):
1. queue priority
2. fairshare priority
@@ -16,24 +13,15 @@ of importance):
### Queue priority
-Queue priority is priority of queue where job is queued before
-execution.
+Queue priority is priority of queue where job is queued before execution.
-Queue priority has the biggest impact on job execution priority.
-Execution priority of jobs in higher priority queues is always greater
-than execution priority of jobs in lower priority queues. Other
-properties of job used for determining job execution priority (fairshare
-priority, eligible time) cannot compete with queue priority.
+Queue priority has the biggest impact on job execution priority. Execution priority of jobs in higher priority queues is always greater than execution priority of jobs in lower priority queues. Other properties of job used for determining job execution priority (fairshare priority, eligible time) cannot compete with queue priority.
-Queue priorities can be seen at
-<https://extranet.it4i.cz/rsweb/salomon/queues>
+Queue priorities can be seen at <https://extranet.it4i.cz/rsweb/salomon/queues>
### Fairshare priority
-Fairshare priority is priority calculated on recent usage of resources.
-Fairshare priority is calculated per project, all members of project
-share same fairshare priority. Projects with higher recent usage have
-lower fairshare priority than projects with lower or none recent usage.
+Fairshare priority is priority calculated on recent usage of resources. Fairshare priority is calculated per project, all members of project share same fairshare priority. Projects with higher recent usage have lower fairshare priority than projects with lower or none recent usage.
Fairshare priority is used for ranking jobs with equal queue priority.
@@ -41,33 +29,21 @@ Fairshare priority is calculated as
-where MAX_FAIRSHARE has value 1E6,
-usage~Project~ is cumulated usage by all members of selected project,
-usage~Total~ is total usage by all users, by all projects.
+where MAX_FAIRSHARE has value 1E6, usage~Project~ is cumulated usage by all members of selected project, usage~Total~ is total usage by all users, by all projects.
-Usage counts allocated corehours (ncpus*walltime). Usage is decayed, or
-cut in half periodically, at the interval 168 hours (one week).
-Jobs queued in queue qexp are not calculated to project's usage.
+Usage counts allocated corehours (ncpus*walltime). Usage is decayed, or cut in half periodically, at the interval 168 hours (one week). Jobs queued in queue qexp are not calculated to project's usage.
-Calculated usage and fairshare priority can be seen at
-<https://extranet.it4i.cz/rsweb/salomon/projects>.
+>Calculated usage and fairshare priority can be seen at <https://extranet.it4i.cz/rsweb/salomon/projects>.
-Calculated fairshare priority can be also seen as
-Resource_List.fairshare attribute of a job.
+Calculated fairshare priority can be also seen as Resource_List.fairshare attribute of a job.
###Eligible time
-Eligible time is amount (in seconds) of eligible time job accrued while
-waiting to run. Jobs with higher eligible time gains higher
-priority.
+Eligible time is amount (in seconds) of eligible time job accrued while waiting to run. Jobs with higher eligible time gains higher priority.
-Eligible time has the least impact on execution priority. Eligible time
-is used for sorting jobs with equal queue priority and fairshare
-priority. It is very, very difficult for >eligible time to
-compete with fairshare priority.
+Eligible time has the least impact on execution priority. Eligible time is used for sorting jobs with equal queue priority and fairshare priority. It is very, very difficult for eligible time to compete with fairshare priority.
-Eligible time can be seen as eligible_time attribute of
-job.
+Eligible time can be seen as eligible_time attribute of job.
### Formula
@@ -79,31 +55,16 @@ Job execution priority (job sort formula) is calculated as:
The scheduler uses job backfilling.
-Backfilling means fitting smaller jobs around the higher-priority jobs
-that the scheduler is going to run next, in such a way that the
-higher-priority jobs are not delayed. Backfilling allows us to keep
-resources from becoming idle when the top job (job with the highest
-execution priority) cannot run.
+Backfilling means fitting smaller jobs around the higher-priority jobs that the scheduler is going to run next, in such a way that the higher-priority jobs are not delayed. Backfilling allows us to keep resources from becoming idle when the top job (job with the highest execution priority) cannot run.
-The scheduler makes a list of jobs to run in order of execution
-priority. Scheduler looks for smaller jobs that can fit into the usage
-gaps
-around the highest-priority jobs in the list. The scheduler looks in the
-prioritized list of jobs and chooses the highest-priority smaller jobs
-that fit. Filler jobs are run only if they will not delay the start time
-of top jobs.
+The scheduler makes a list of jobs to run in order of execution priority. Scheduler looks for smaller jobs that can fit into the usage gaps around the highest-priority jobs in the list. The scheduler looks in the prioritized list of jobs and chooses the highest-priority smaller jobs that fit. Filler jobs are run only if they will not delay the start time of top jobs.
-It means, that jobs with lower execution priority can be run before jobs
-with higher execution priority.
+It means, that jobs with lower execution priority can be run before jobs with higher execution priority.
-It is **very beneficial to specify the walltime** when submitting jobs.
+>It is **very beneficial to specify the walltime** when submitting jobs.
-Specifying more accurate walltime enables better schedulling, better
-execution times and better resource usage. Jobs with suitable (small)
-walltime could be backfilled - and overtake job(s) with higher priority.
+Specifying more accurate walltime enables better schedulling, better execution times and better resource usage. Jobs with suitable (small) walltime could be backfilled - and overtake job(s) with higher priority.
### Job placement
-Job [placement can be controlled by flags during
-submission](job-submission-and-execution.html#job_placement).
-
+Job [placement can be controlled by flags during submission](job-submission-and-execution.html#job_placement).
\ No newline at end of file
diff --git a/docs.it4i/salomon/resource-allocation-and-job-execution/job-submission-and-execution.md b/docs.it4i/salomon/resource-allocation-and-job-execution/job-submission-and-execution.md
index e0499050b..da291a92a 100644
--- a/docs.it4i/salomon/resource-allocation-and-job-execution/job-submission-and-execution.md
+++ b/docs.it4i/salomon/resource-allocation-and-job-execution/job-submission-and-execution.md
@@ -1,166 +1,106 @@
-Job submission and execution
+Job submission and execution
============================
-
-
Job Submission
--------------
-
When allocating computational resources for the job, please specify
1. suitable queue for your job (default is qprod)
2. number of computational nodes required
3. number of cores per node required
-4. maximum wall time allocated to your calculation, note that jobs
- exceeding maximum wall time will be killed
+4. maximum wall time allocated to your calculation, note that jobs exceeding maximum wall time will be killed
5. Project ID
6. Jobscript or interactive switch
-Use the **qsub** command to submit your job to a queue for allocation of
-the computational resources.
+>Use the **qsub** command to submit your job to a queue for allocation of the computational resources.
Submit the job using the qsub command:
-`
+```bash
$ qsub -A Project_ID -q queue -l select=x:ncpus=y,walltime=[[hh:]mm:]ss[.ms] jobscript
-`
+```
-The qsub submits the job into the queue, in another words the qsub
-command creates a request to the PBS Job manager for allocation of
-specified resources. The resources will be allocated when available,
-subject to above described policies and constraints. **After the
-resources are allocated the jobscript or interactive shell is executed
-on first of the allocated nodes.**
+The qsub submits the job into the queue, in another words the qsub command creates a request to the PBS Job manager for allocation of specified resources. The resources will be allocated when available, subject to above described policies and constraints. **After the resources are allocated the jobscript or interactive shell is executed on first of the allocated nodes.**
-PBS statement nodes (qsub -l nodes=nodespec) is not supported on Salomon
-cluster.**
+>PBS statement nodes (qsub -l nodes=nodespec) is not supported on Salomon cluster.
### Job Submission Examples
-`
+```bash
$ qsub -A OPEN-0-0 -q qprod -l select=64:ncpus=24,walltime=03:00:00 ./myjob
-`
-
-In this example, we allocate 64 nodes, 24 cores per node, for 3 hours.
-We allocate these resources via the qprod queue, consumed resources will
-be accounted to the Project identified by Project ID OPEN-0-0. Jobscript
-myjob will be executed on the first node in the allocation.
+```
-Â
+In this example, we allocate 64 nodes, 24 cores per node, for 3 hours. We allocate these resources via the qprod queue, consumed resources will be accounted to the Project identified by Project ID OPEN-0-0. Jobscript myjob will be executed on the first node in the allocation.
-`
+```bash
$ qsub -q qexp -l select=4:ncpus=24 -I
-`
+```
-In this example, we allocate 4 nodes, 24 cores per node, for 1 hour. We
-allocate these resources via the qexp queue. The resources will be
-available interactively
+In this example, we allocate 4 nodes, 24 cores per node, for 1 hour. We allocate these resources via the qexp queue. The resources will be available interactively
-Â
-
-`
+```bash
$ qsub -A OPEN-0-0 -q qlong -l select=10:ncpus=24 ./myjob
-`
-
-In this example, we allocate 10 nodes, 24 cores per node, for 72 hours.
-We allocate these resources via the qlong queue. Jobscript myjob will be
-executed on the first node in the allocation.
+```
-Â
+In this example, we allocate 10 nodes, 24 cores per node, for 72 hours. We allocate these resources via the qlong queue. Jobscript myjob will be executed on the first node in the allocation.
-`
+```bash
$ qsub -A OPEN-0-0 -q qfree -l select=10:ncpus=24 ./myjob
-`
+```
-In this example, we allocate 10Â nodes, 24 cores per node, for 12 hours.
-We allocate these resources via the qfree queue. It is not required that
-the project OPEN-0-0 has any available resources left. Consumed
-resources are still accounted for. Jobscript myjob will be executed on
-the first node in the allocation.
+In this example, we allocate 10Â nodes, 24 cores per node, for 12 hours. We allocate these resources via the qfree queue. It is not required that the project OPEN-0-0 has any available resources left. Consumed resources are still accounted for. Jobscript myjob will be executed on the first node in the allocation.
### Intel Xeon Phi co-processors
-To allocate a node with Xeon Phi co-processor, user needs to specify
-that in select statement. Currently only allocation of whole nodes with
-both Phi cards as the smallest chunk is supported. Standard PBSPro
-approach through attributes "accelerator", "naccelerators" and
-"accelerator_model" is used. The "accelerator_model" can be omitted,
-since on Salomon only one type of accelerator type/model is available.
-
-The absence of specialized queue for accessing the nodes with cards
-means, that the Phi cards can be utilized in any queue, including qexp
-for testing/experiments, qlong for longer jobs, qfree after the project
-resources have been spent, etc. The Phi cards are thus also available to
-PRACE users. There's no need to ask for permission to utilize the Phi
-cards in project proposals.
-
-`
+To allocate a node with Xeon Phi co-processor, user needs to specify that in select statement. Currently only allocation of whole nodes with both Phi cards as the smallest chunk is supported. Standard PBSPro approach through attributes "accelerator", "naccelerators" and "accelerator_model" is used. The "accelerator_model" can be omitted, since on Salomon only one type of accelerator type/model is available.
+
+The absence of specialized queue for accessing the nodes with cards means, that the Phi cards can be utilized in any queue, including qexp for testing/experiments, qlong for longer jobs, qfree after the project resources have been spent, etc. The Phi cards are thus also available to PRACE users. There's no need to ask for permission to utilize the Phi cards in project proposals.
+
+```bash
$ qsub -A OPEN-0-0 -I -q qprod -l select=1:ncpus=24:accelerator=True:naccelerators=2:accelerator_model=phi7120 ./myjob
-`
+```
-In this example, we allocate 1 node, with 24 cores, with 2 Xeon Phi
-7120p cards, running batch job ./myjob. The default time for qprod is
-used, e. g. 24 hours.
+In this example, we allocate 1 node, with 24 cores, with 2 Xeon Phi 7120p cards, running batch job ./myjob. The default time for qprod is used, e. g. 24 hours.
-`
+```bash
$ qsub -A OPEN-0-0 -I -q qlong -l select=4:ncpus=24:accelerator=True:naccelerators=2 -l walltime=56:00:00 -I
-`
+```
-In this example, we allocate 4 nodes, with 24 cores per node (totalling
-96 cores), with 2 Xeon Phi 7120p cards per node (totalling 8 Phi cards),
-running interactive job for 56 hours. The accelerator model name was
-omitted.
+In this example, we allocate 4 nodes, with 24 cores per node (totalling 96 cores), with 2 Xeon Phi 7120p cards per node (totalling 8 Phi cards), running interactive job for 56 hours. The accelerator model name was omitted.
### UV2000 SMP
-14 NUMA nodes available on UV2000
+>14 NUMA nodes available on UV2000
Per NUMA node allocation.
Jobs are isolated by cpusets.
-The UV2000 (node uv1) offers 3328GB of RAM and 112 cores, distributed in
-14 NUMA nodes. A NUMA node packs 8 cores and approx. 236GB RAM. In the
-PBSÂ the UV2000 provides 14 chunks, a chunk per NUMA node (seeÂ
-[Resource allocation
-policy](resources-allocation-policy.html)). The jobs on
-UV2000 are isolated from each other by cpusets, so that a job by one
-user may not utilize CPU or memory allocated to a job by other user.
-Always, full chunks are allocated, a job may only use resources of the
-NUMA nodes allocated to itself.
-
-`
+The UV2000 (node uv1) offers 3328GB of RAM and 112 cores, distributed in 14 NUMA nodes. A NUMA node packs 8 cores and approx. 236GB RAM. In the PBS the UV2000 provides 14 chunks, a chunk per NUMA node (see [Resource allocation policy](resources-allocation-policy.html)). The jobs on UV2000 are isolated from each other by cpusets, so that a job by one user may not utilize CPU or memory allocated to a job by other user. Always, full chunks are allocated, a job may only use resources of the NUMA nodes allocated to itself.
+
+```bash
 $ qsub -A OPEN-0-0 -q qfat -l select=14 ./myjob
-`
+```
-In this example, we allocate all 14 NUMA nodes (corresponds to 14
-chunks), 112 cores of the SGI UV2000 node for 72 hours. Jobscript myjob
-will be executed on the node uv1.
+In this example, we allocate all 14 NUMA nodes (corresponds to 14 chunks), 112 cores of the SGI UV2000 node for 72 hours. Jobscript myjob will be executed on the node uv1.
-`
+```bash
$ qsub -A OPEN-0-0 -q qfat -l select=1:mem=2000GB ./myjob
-`
+```
-In this example, we allocate 2000GB of memory on the UV2000 for 72
-hours. By requesting 2000GB of memory, 10 chunks are allocated.
-Jobscript myjob will be executed on the node uv1.
+In this example, we allocate 2000GB of memory on the UV2000 for 72 hours. By requesting 2000GB of memory, 10 chunks are allocated. Jobscript myjob will be executed on the node uv1.
### Useful tricks
-All qsub options may be [saved directly into the
-jobscript](job-submission-and-execution.html#PBSsaved). In
-such a case, no options to qsub are needed.
+All qsub options may be [saved directly into the jobscript](job-submission-and-execution.html#PBSsaved). In such a case, no options to qsub are needed.
-`
+```bash
$ qsub ./myjob
-`
-
-Â
+```
-By default, the PBS batch system sends an e-mail only when the job is
-aborted. Disabling mail events completely can be done like this:
+By default, the PBS batch system sends an e-mail only when the job is aborted. Disabling mail events completely can be done like this:
-`
+```bash
$ qsub -m n
-`
+```
Advanced job placement
--------------------------
@@ -169,69 +109,57 @@ Advanced job placement
Specific nodes may be allocated via the PBS
-`
+```bash
qsub -A OPEN-0-0 -q qprod -l select=1:ncpus=24:host=r24u35n680+1:ncpus=24:host=r24u36n681 -I
-`
+```
Or using short names
-`
+```bash
qsub -A OPEN-0-0 -q qprod -l select=1:ncpus=24:host=cns680+1:ncpus=24:host=cns681 -I
-`
+```
-In this example, we allocate nodes r24u35n680 and r24u36n681, all 24
-cores per node, for 24 hours. Consumed resources will be accounted to
-the Project identified by Project ID OPEN-0-0. The resources will be
-available interactively.
+In this example, we allocate nodes r24u35n680 and r24u36n681, all 24 cores per node, for 24 hours. Consumed resources will be accounted to the Project identified by Project ID OPEN-0-0. The resources will be available interactively.
### Placement by |Hypercube|dimension|
Nodes may be selected via the PBS resource attribute ehc_[1-7]d .
- |Hypercube|dimension|
- --------------- |---|---|---------------------------------
- |1D|ehc_1d|
- |2D|ehc_2d|
- |3D|ehc_3d|
- |4D|ehc_4d|
- |5D|ehc_5d|
- |6D|ehc_6d|
- |7D|ehc_7d|
-
-Â
-
-`
+|Hypercube|dimension|
+|---|---|
+|1D|ehc_1d|
+|2D|ehc_2d|
+|3D|ehc_3d|
+|4D|ehc_4d|
+|5D|ehc_5d|
+|6D|ehc_6d|
+|7D|ehc_7d|
+
+```bash
$ qsub -A OPEN-0-0 -q qprod -l select=4:ncpus=24 -l place=group=ehc_1d -I
-`
+```
-In this example, we allocate 4 nodes, 24 cores, selecting only the nodes
-with [hypercube
-dimension](../network-1/7d-enhanced-hypercube.html) 1.
+In this example, we allocate 4 nodes, 24 cores, selecting only the nodes with [hypercube dimension](../network-1/7d-enhanced-hypercube.html) 1.
### Placement by IB switch
-Groups of computational nodes are connected to chassis integrated
-Infiniband switches. These switches form the leaf switch layer of the
-[Infiniband network](../network-1.html) . Nodes sharing
-the leaf switch can communicate most efficiently. Sharing the same
-switch prevents hops in the network and provides for unbiased, most
-efficient network communication.
+Groups of computational nodes are connected to chassis integrated Infiniband switches. These switches form the leaf switch layer of the [Infiniband network](../network-1.html) . Nodes sharing the leaf switch can communicate most efficiently. Sharing the same switch prevents hops in the network and provides for unbiased, most efficient network communication.
There are at most 9 nodes sharing the same Infiniband switch.
Infiniband switch list:
-`
+```bash
$ qmgr -c "print node @a" | grep switch
set node r4i1n11 resources_available.switch = r4i1s0sw1
set node r2i0n0 resources_available.switch = r2i0s0sw1
set node r2i0n1 resources_available.switch = r2i0s0sw1
...
-`
+```
List of all nodes per Infiniband switch:
-`
+```bash
$ qmgr -c "print node @a" | grep r36sw3
set node r36u31n964 resources_available.switch = r36sw3
set node r36u32n965 resources_available.switch = r36sw3
@@ -242,47 +170,41 @@ set node r36u36n969 resources_available.switch = r36sw3
set node r37u32n970 resources_available.switch = r36sw3
set node r37u33n971 resources_available.switch = r36sw3
set node r37u34n972 resources_available.switch = r36sw3
-`
+```
-Nodes sharing the same switch may be selected via the PBS resource
-attribute switch.
+Nodes sharing the same switch may be selected via the PBS resource attribute switch.
-We recommend allocating compute nodes of a single switch when best
-possible computational network performance is required to run the job
-efficiently:
+We recommend allocating compute nodes of a single switch when best possible computational network performance is required to run the job efficiently:
-`
+```bash
$ qsub -A OPEN-0-0 -q qprod -l select=9:ncpus=24:switch=r4i1s0sw1 ./myjob
-`
+```
-In this example, we request all the 9 nodes sharing the r4i1s0sw1 switch
-for 24 hours.
+In this example, we request all the 9 nodes sharing the r4i1s0sw1 switch for 24 hours.
-`
+```bash
$ qsub -A OPEN-0-0 -q qprod -l select=9:ncpus=24 -l place=group=switch ./myjob
-`
+```
-In this example, we request 9 nodes placed on the same switch using node
-grouping placement for 24 hours.
+In this example, we request 9 nodes placed on the same switch using node grouping placement for 24 hours.
HTML commented section #1 (turbo boost is to be implemented)
Job Management
--------------
-Check status of your jobs using the **qstat** and **check-pbs-jobs**
-commands
+>Check status of your jobs using the **qstat** and **check-pbs-jobs** commands
-`
+```bash
$ qstat -a
$ qstat -a -u username
$ qstat -an -u username
$ qstat -f 12345.isrv5
-`
+```
Example:
-`
+```bash
$ qstat -a
srv11:
@@ -292,44 +214,34 @@ Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
16287.isrv5 user1 qlong job1 6183 4 64 -- 144:0 R 38:25
16468.isrv5 user1 qlong job2 8060 4 64 -- 144:0 R 17:44
16547.isrv5 user2 qprod job3x 13516 2 32 -- 48:00 R 00:58
-`
-
-In this example user1 and user2 are running jobs named job1, job2 and
-job3x. The jobs job1 and job2 are using 4 nodes, 16 cores per node each.
-The job1 already runs for 38 hours and 25 minutes, job2 for 17 hours 44
-minutes. The job1 already consumed 64*38.41 = 2458.6 core hours. The
-job3x already consumed 0.96*32 = 30.93 core hours. These consumed core
-hours will be accounted on the respective project accounts, regardless
-of whether the allocated cores were actually used for computations.
-
-Check status of your jobs using check-pbs-jobs command. Check presence
-of user's PBS jobs' processes on execution hosts. Display load,
-processes. Display job standard and error output. Continuously display
-(tail -f) job standard or error output.
-
-`
+```
+
+In this example user1 and user2 are running jobs named job1, job2 and job3x. The jobs job1 and job2 are using 4 nodes, 16 cores per node each. The job1 already runs for 38 hours and 25 minutes, job2 for 17 hours 44 minutes. The job1 already consumed 64*38.41 = 2458.6 core hours. The job3x already consumed 0.96*32 = 30.93 core hours. These consumed core hours will be accounted on the respective project accounts, regardless of whether the allocated cores were actually used for computations.
+
+Check status of your jobs using check-pbs-jobs command. Check presence of user's PBS jobs' processes on execution hosts. Display load, processes. Display job standard and error output. Continuously display (tail -f) job standard or error output.
+
+```bash
$ check-pbs-jobs --check-all
$ check-pbs-jobs --print-load --print-processes
$ check-pbs-jobs --print-job-out --print-job-err
$ check-pbs-jobs --jobid JOBID --check-all --print-all
$ check-pbs-jobs --jobid JOBID --tailf-job-out
-`
+```
Examples:
-`
+```bash
$ check-pbs-jobs --check-all
JOB 35141.dm2, session_id 71995, user user2, nodes r3i6n2,r3i6n3
Check session id: OK
Check processes
r3i6n2: OK
r3i6n3: No process
-`
+```
-In this example we see that job 35141.dm2 currently runs no process on
-allocated node r3i6n2, which may indicate an execution error.
+In this example we see that job 35141.dm2 currently runs no process on allocated node r3i6n2, which may indicate an execution error.
-`
+```bash
$ check-pbs-jobs --print-load --print-processes
JOB 35141.dm2, session_id 71995, user user2, nodes r3i6n2,r3i6n3
Print load
@@ -341,13 +253,11 @@ r3i6n2: 0.0 -bash
r3i6n2: 0.0 /bin/bash /var/spool/PBS/mom_priv/jobs/35141.dm2.SC
r3i6n2: 99.7 run-task
...
-`
+```
-In this example we see that job 35141.dm2 currently runs process
-run-task on node r3i6n2, using one thread only, while node r3i6n3 is
-empty, which may indicate an execution error.
+In this example we see that job 35141.dm2 currently runs process run-task on node r3i6n2, using one thread only, while node r3i6n3 is empty, which may indicate an execution error.
-`
+```bash
$ check-pbs-jobs --jobid 35141.dm2 --print-job-out
JOB 35141.dm2, session_id 71995, user user2, nodes r3i6n2,r3i6n3
Print job standard output:
@@ -357,49 +267,42 @@ Script name  : script
Run loop 1
Run loop 2
Run loop 3
-`
+```
-In this example, we see actual output (some iteration loops) of the job
-35141.dm2
+In this example, we see actual output (some iteration loops) of the job 35141.dm2
-Manage your queued or running jobs, using the **qhold**, **qrls**,
-qdel,** **qsig** or **qalter** commands
+>Manage your queued or running jobs, using the **qhold**, **qrls**, **qdel,** **qsig** or **qalter** commands
You may release your allocation at any time, using qdel command
-`
+```bash
$ qdel 12345.isrv5
-`
+```
You may kill a running job by force, using qsig command
-`
+```bash
$ qsig -s 9 12345.isrv5
-`
+```
Learn more by reading the pbs man page
-`
+```bash
$ man pbs_professional
-`
+```
Job Execution
-------------
### Jobscript
-Prepare the jobscript to run batch jobs in the PBS queue system
+>Prepare the jobscript to run batch jobs in the PBS queue system
-The Jobscript is a user made script, controlling sequence of commands
-for executing the calculation. It is often written in bash, other
-scripts may be used as well. The jobscript is supplied to PBS **qsub**
-command as an argument and executed by the PBS Professional workload
-manager.
+The Jobscript is a user made script, controlling sequence of commands for executing the calculation. It is often written in bash, other scripts may be used as well. The jobscript is supplied to PBS **qsub** command as an argument and executed by the PBS Professional workload manager.
-The jobscript or interactive shell is executed on first of the allocated
-nodes.
+>The jobscript or interactive shell is executed on first of the allocated nodes.
-`
+```bash
$ qsub -q qexp -l select=4:ncpus=24 -N Name0 ./myjob
$ qstat -n -u username
@@ -409,39 +312,30 @@ Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
--------------- -------- -- |---|---| ------ --- --- ------ ----- - -----
15209.isrv5 username qexp Name0 5530 4 96 -- 01:00 R 00:00
r21u01n577/0*24+r21u02n578/0*24+r21u03n579/0*24+r21u04n580/0*24
-`
+```
- In this example, the nodes r21u01n577, r21u02n578, r21u03n579,
-r21u04n580 were allocated for 1 hour via the qexp queue. The jobscript
-myjob will be executed on the node r21u01n577, while the
-nodes r21u02n578, r21u03n579, r21u04n580 are available for use as well.
+In this example, the nodes r21u01n577, r21u02n578, r21u03n579, r21u04n580 were allocated for 1 hour via the qexp queue. The jobscript myjob will be executed on the node r21u01n577, while the nodes r21u02n578, r21u03n579, r21u04n580 are available for use as well.
-The jobscript or interactive shell is by default executed in home
-directory
+>The jobscript or interactive shell is by default executed in home directory
-`
+```bash
$ qsub -q qexp -l select=4:ncpus=24 -I
qsub: waiting for job 15210.isrv5 to start
qsub: job 15210.isrv5 ready
$ pwd
/home/username
-`
+```
-In this example, 4 nodes were allocated interactively for 1 hour via the
-qexp queue. The interactive shell is executed in the home directory.
+In this example, 4 nodes were allocated interactively for 1 hour via the qexp queue. The interactive shell is executed in the home directory.
-All nodes within the allocation may be accessed via ssh. Unallocated
-nodes are not accessible to user.
+>All nodes within the allocation may be accessed via ssh. Unallocated nodes are not accessible to user.
-The allocated nodes are accessible via ssh from login nodes. The nodes
-may access each other via ssh as well.
+The allocated nodes are accessible via ssh from login nodes. The nodes may access each other via ssh as well.
-Calculations on allocated nodes may be executed remotely via the MPI,
-ssh, pdsh or clush. You may find out which nodes belong to the
-allocation by reading the $PBS_NODEFILE file
+Calculations on allocated nodes may be executed remotely via the MPI, ssh, pdsh or clush. You may find out which nodes belong to the allocation by reading the $PBS_NODEFILE file
-`
+```bash
qsub -q qexp -l select=2:ncpus=24 -I
qsub: waiting for job 15210.isrv5 to start
qsub: job 15210.isrv5 ready
@@ -455,28 +349,22 @@ r4i6n13.ib0.smc.salomon.it4i.cz
r4i7n0.ib0.smc.salomon.it4i.cz
r4i7n2.ib0.smc.salomon.it4i.cz
-
$ pdsh -w r2i5n6,r4i6n13,r4i7n[0,2] hostname
r4i6n13: r4i6n13
r2i5n6: r2i5n6
r4i7n2: r4i7n2
r4i7n0: r4i7n0
-`
+```
-In this example, the hostname program is executed via pdsh from the
-interactive shell. The execution runs on all four allocated nodes. The
-same result would be achieved if the pdsh is called from any of the
-allocated nodes or from the login nodes.
+In this example, the hostname program is executed via pdsh from the interactive shell. The execution runs on all four allocated nodes. The same result would be achieved if the pdsh is called from any of the allocated nodes or from the login nodes.
### Example Jobscript for MPI Calculation
-Production jobs must use the /scratch directory for I/O
+>Production jobs must use the /scratch directory for I/O
-The recommended way to run production jobs is to change to /scratch
-directory early in the jobscript, copy all inputs to /scratch, execute
-the calculations and copy outputs to home directory.
+The recommended way to run production jobs is to change to /scratch directory early in the jobscript, copy all inputs to /scratch, execute the calculations and copy outputs to home directory.
-`
+```bash
#!/bin/bash
# change to scratch directory, exit on failure
@@ -484,7 +372,7 @@ SCRDIR=/scratch/work/user/$USER/myjob
mkdir -p $SCRDIR
cd $SCRDIR || exit
-# copy input file to scratch
+# copy input file to scratch
cp $PBS_O_WORKDIR/input .
cp $PBS_O_WORKDIR/mympiprog.x .
@@ -499,34 +387,19 @@ cp output $PBS_O_WORKDIR/.
#exit
exit
-`
-
-In this example, some directory on the /home holds the input file input
-and executable mympiprog.x . We create a directory myjob on the /scratch
-filesystem, copy input and executable files from the /home directory
-where the qsub was invoked ($PBS_O_WORKDIR) to /scratch, execute the
-MPI programm mympiprog.x and copy the output file back to the /home
-directory. The mympiprog.x is executed as one process per node, on all
-allocated nodes.
-
-Consider preloading inputs and executables onto [shared
-scratch](../storage.html) before the calculation starts.
-
-In some cases, it may be impractical to copy the inputs to scratch and
-outputs to home. This is especially true when very large input and
-output files are expected, or when the files should be reused by a
-subsequent calculation. In such a case, it is users responsibility to
-preload the input files on shared /scratch before the job submission and
-retrieve the outputs manually, after all calculations are finished.
-
-Store the qsub options within the jobscript.
-Use **mpiprocs** and **ompthreads** qsub options to control the MPI job
-execution.
-
-Example jobscript for an MPI job with preloaded inputs and executables,
-options for qsub are stored within the script :
-
-`
+```
+
+In this example, some directory on the /home holds the input file input and executable mympiprog.x . We create a directory myjob on the /scratch filesystem, copy input and executable files from the /home directory where the qsub was invoked ($PBS_O_WORKDIR) to /scratch, execute the MPI programm mympiprog.x and copy the output file back to the /home directory. The mympiprog.x is executed as one process per node, on all allocated nodes.
+
+>Consider preloading inputs and executables onto [shared scratch](../storage.html) before the calculation starts.
+
+In some cases, it may be impractical to copy the inputs to scratch and outputs to home. This is especially true when very large input and output files are expected, or when the files should be reused by a subsequent calculation. In such a case, it is users responsibility to preload the input files on shared /scratch before the job submission and retrieve the outputs manually, after all calculations are finished.
+
+>Store the qsub options within the jobscript. Use **mpiprocs** and **ompthreads** qsub options to control the MPI job execution.
+
+Example jobscript for an MPI job with preloaded inputs and executables, options for qsub are stored within the script :
+
+```bash
#!/bin/bash
#PBS -q qprod
#PBS -N MYJOB
@@ -545,34 +418,25 @@ mpiexec ./mympiprog.x
#exit
exit
-`
+```
-In this example, input and executable files are assumed preloaded
-manually in /scratch/$USER/myjob directory. Note the **mpiprocs** and
-ompthreads** qsub options, controlling behavior of the MPI execution.
-The mympiprog.x is executed as one process per node, on all 100
-allocated nodes. If mympiprog.x implements OpenMP threads, it will run
-24 threads per node.
+In this example, input and executable files are assumed preloaded manually in /scratch/$USER/myjob directory. Note the **mpiprocs** and **ompthreads** qsub options, controlling behavior of the MPI execution. The mympiprog.x is executed as one process per node, on all 100 allocated nodes. If mympiprog.x implements OpenMP threads, it will run 24 threads per node.
HTML commented section #2 (examples need to be reworked)
### Example Jobscript for Single Node Calculation
-Local scratch directory is often useful for single node jobs. Local
-scratch will be deleted immediately after the job ends.
-Be very careful, use of RAM disk filesystem is at the expense of
-operational memory.
+>Local scratch directory is often useful for single node jobs. Local scratch will be deleted immediately after the job ends. Be very careful, use of RAM disk filesystem is at the expense of operational memory.
-Example jobscript for single node calculation, using [local
-scratch](../storage.html) on the node:
+Example jobscript for single node calculation, using [local scratch](../storage.html) on the node:
-`
+```bash
#!/bin/bash
# change to local scratch directory
cd /lscratch/$PBS_JOBID || exit
-# copy input file to scratch
+# copy input file to scratch
cp $PBS_O_WORKDIR/input .
cp $PBS_O_WORKDIR/myprog.x .
@@ -584,12 +448,6 @@ cp output $PBS_O_WORKDIR/.
#exit
exit
-`
-
-In this example, some directory on the home holds the input file input
-and executable myprog.x . We copy input and executable files from the
-home directory where the qsub was invoked ($PBS_O_WORKDIR) to local
-scratch /lscratch/$PBS_JOBID, execute the myprog.x and copy the output
-file back to the /home directory. The myprog.x runs on one node only and
-may use threads.
+```
+In this example, some directory on the home holds the input file input and executable myprog.x . We copy input and executable files from the home directory where the qsub was invoked ($PBS_O_WORKDIR) to local scratch /lscratch/$PBS_JOBID, execute the myprog.x and copy the output file back to the /home directory. The myprog.x runs on one node only and may use threads.
\ No newline at end of file
diff --git a/docs.it4i/salomon/resource-allocation-and-job-execution/resources-allocation-policy.md b/docs.it4i/salomon/resource-allocation-and-job-execution/resources-allocation-policy.md
index 3395d6f73..924af1483 100644
--- a/docs.it4i/salomon/resource-allocation-and-job-execution/resources-allocation-policy.md
+++ b/docs.it4i/salomon/resource-allocation-and-job-execution/resources-allocation-policy.md
@@ -1,147 +1,55 @@
-Resources Allocation Policy
+Resources Allocation Policy
===========================
-
-
-Resources Allocation Policy
+Resources Allocation Policy
---------------------------
+The resources are allocated to the job in a fairshare fashion, subject to constraints set by the queue and resources available to the Project. The Fairshare at Anselm ensures that individual users may consume approximately equal amount of resources per week. Detailed information in the [Job scheduling](job-priority.md) section. The resources are accessible via several queues for queueing the jobs. The queues provide prioritized and exclusive access to the computational resources. Following table provides the queue partitioning overview:
-The resources are allocated to the job in a fairshare fashion, subject
-to constraints set by the queue and resources available to the Project.
-The Fairshare at Anselm ensures that individual users may consume
-approximately equal amount of resources per week. Detailed information
-in the [Job scheduling](job-priority.html) section. The
-resources are accessible via several queues for queueing the jobs. The
-queues provide prioritized and exclusive access to the computational
-resources. Following table provides the queue partitioning overview:
-
-Â
-
- |queue |active project |project resources |nodes<th align="left">min ncpus*<th align="left">priority<th align="left">authorization<th align="left">walltime |
+ |queue |active project |project resources |nodes|min ncpus*|priority|authorization|walltime |
| --- | --- |
- |<strong>qexp</strong>\ |no |none required |32 nodes, max 8 per user |24 |>150 |no |1 / 1h |
- |<strong>qprod</strong>\ |yes |> 0 |>1006 nodes, max 86 per job\ |24 |0 |no |24 / 48h |
- |<strong>qlong</strong>Long queue |yes |> 0 |256 nodes, max 40 per job, only non-accelerated nodes allowed |24 |0 |no |72 / 144h |
- |<strong>qmpp</strong>Massive parallel queue |yes |> 0 |1006 nodes |24 |0 |yes |2 / 4h |
- |<strong>qfat</strong>UV2000 queue |yes |> 0\ |1 (uv1) |8 |0 |yes |24 / 48h |
- |<strong>qfree</strong>\ |yes |none required |752 nodes, max 86 per job |24 |-1024 |no |12 / 12h |
- |<strong><strong>qviz</strong></strong>Visualization queue |yes |none required |2 (with NVIDIA Quadro K5000) |4 |150 |no |1 / 2h |
-
-Â
-
-The qfree queue is not free of charge**. [Normal
-accounting](resources-allocation-policy.html#resources-accounting-policy)
-applies. However, it allows for utilization of free resources, once a
-Project exhausted all its allocated computational resources. This does
-not apply for Directors Discreation's projects (DD projects) by default.
-Usage of qfree after exhaustion of DD projects computational resources
-is allowed after request for this queue.
-
-Â
-
-- **qexp**, the \: This queue is dedicated for testing and
- running very small jobs. It is not required to specify a project to
- enter the qexp. >*>There are 2 nodes always reserved for
- this queue (w/o accelerator), maximum 8 nodes are available via the
- qexp for a particular user. *The nodes may be
- allocated on per core basis. No special authorization is required to
- use it. The maximum runtime in qexp is 1 hour.
-- **qprod**, the \***: This queue is intended for
- normal production runs. It is required that active project with
- nonzero remaining resources is specified to enter the qprod. All
- nodes may be accessed via the qprod queue, however only 86 per job.
- ** Full nodes, 24 cores per node are allocated. The queue runs with
- medium priority and no special authorization is required to use it.
- The maximum runtime in qprod is 48 hours.
-- **qlong**, the Long queue***: This queue is intended for long
- production runs. It is required that active project with nonzero
- remaining resources is specified to enter the qlong. Only 336 nodes
- without acceleration may be accessed via the qlong queue. Full
- nodes, 24 cores per node are allocated. The queue runs with medium
- priority and no special authorization is required to use it.>
- *The maximum runtime in qlong is 144 hours (three times of the
- standard qprod time - 3 * 48 h)*
-- >***qmpp**, the massively parallel queue. This queue is
- intended for massively parallel runs. It is required that active
- project with nonzero remaining resources is specified to enter
- the qmpp. All nodes may be accessed via the qmpp queue. ** Full
- nodes, 24 cores per node are allocated. The queue runs with medium
- priority and no special authorization is required to use it. The
- maximum runtime in qmpp is 4 hours. An PI> *needs explicitly*
- ask [support](https://support.it4i.cz/rt/)
- for authorization to enter the queue for all users associated to
- her/his Project.
-
-- >***qfat**, the UV2000 queue. This queue is dedicated
- to access the fat SGI UV2000 SMP machine. The machine (uv1) has 112
- Intel IvyBridge cores at 3.3GHz and 3.25TB RAM. An PI> *needs
- explicitly* ask
- [support](https://support.it4i.cz/rt/) for
- authorization to enter the queue for all users associated to her/his
- Project.***
-- **qfree**, the \***: The queue qfree is intended
- for utilization of free resources, after a Project exhausted all its
- allocated computational resources (Does not apply to DD projects
- by default. DD projects have to request for persmission on qfree
- after exhaustion of computational resources.). It is required that
- active project is specified to enter the queue, however no remaining
- resources are required. Consumed resources will be accounted to
- the Project. Only 178 nodes without accelerator may be accessed from
- this queue. Full nodes, 24 cores per node are allocated. The queue
- runs with very low priority and no special authorization is required
- to use it. The maximum runtime in qfree is 12 hours.
-- **qviz**, the Visualization queue***: Intended for
- pre-/post-processing using OpenGL accelerated graphics. Currently
- when accessing the node, each user gets 4 cores of a CPU allocated,
- thus approximately 73 GB of RAM and 1/7 of the GPU capacity
- (default "chunk"). *If more GPU power or RAM is required, it is
- recommended to allocate more chunks (with 4 cores each) up to one
- whole node per user, so that all 28 cores, 512 GB RAM and whole GPU
- is exclusive. This is currently also the maximum allowed allocation
- per one user. One hour of work is allocated by default, the user may
- ask for 2 hours maximum.*
-
-Â
-
-To access node with Xeon Phi co-processor user needs to specify that in
-[job submission select
-statement](job-submission-and-execution.html).
+ |**qexe** Express queue|no |none required |32 nodes, max 8 per user |24 |>150 |no |1 / 1h |
+ |**qprod** Production queue|yes |> 0 |>1006 nodes, max 86 per job |24 |0 |no |24 / 48h |
+ |**qlong** Long queue |yes |> 0 |256 nodes, max 40 per job, only non-accelerated nodes allowed |24 |0 |no |72 / 144h |
+ |**qmpp** Massive parallel queue |yes |> 0 |1006 nodes |24 |0 |yes |2 / 4h |
+ |**qfat** UV2000 queue |yes |> 0 |1 (uv1) |8 |0 |yes |24 / 48h |
+ |**qfree** Free resource queue|yes |none required |752 nodes, max 86 per job |24 |-1024 |no |12 / 12h |
+ |**qviz** Visualization queue |yes |none required |2 (with NVIDIA Quadro K5000) |4 |150 |no |1 / 2h |
+
+>**The qfree queue is not free of charge**. [Normal accounting](resources-allocation-policy.html#resources-accounting-policy) applies. However, it allows for utilization of free resources, once a Project exhausted all its allocated computational resources. This does not apply for Directors Discreation's projects (DD projects) by default. Usage of qfree after exhaustion of DD projects computational resources is allowed after request for this queue.
+
+- **qexp**, the Express queue: This queue is dedicated for testing and running very small jobs. It is not required to specify a project to enter the qexp. There are 2 nodes always reserved for this queue (w/o accelerator), maximum 8 nodes are available via the qexp for a particular user. The nodes may be allocated on per core basis. No special authorization is required to use it. The maximum runtime in qexp is 1 hour.
+- **qprod**, the Production queue: This queue is intended for normal production runs. It is required that active project with nonzero remaining resources is specified to enter the qprod. All nodes may be accessed via the qprod queue, however only 86 per job. Full nodes, 24 cores per node are allocated. The queue runs with medium priority and no special authorization is required to use it. The maximum runtime in qprod is 48 hours.
+- **qlong**, the Long queue: This queue is intended for long production runs. It is required that active project with nonzero remaining resources is specified to enter the qlong. Only 336 nodes without acceleration may be accessed via the qlong queue. Full nodes, 24 cores per node are allocated. The queue runs with medium priority and no special authorization is required to use it. The maximum runtime in qlong is 144 hours (three times of the standard qprod time - 3 * 48 h)
+- **qmpp**, the massively parallel queue. This queue is intended for massively parallel runs. It is required that active project with nonzero remaining resources is specified to enter the qmpp. All nodes may be accessed via the qmpp queue. Full nodes, 24 cores per node are allocated. The queue runs with medium priority and no special authorization is required to use it. The maximum runtime in qmpp is 4 hours. An PI needs explicitly ask support for authorization to enter the queue for all users associated to her/his Project.
+- **qfat**, the UV2000 queue. This queue is dedicated to access the fat SGI UV2000 SMP machine. The machine (uv1) has 112 Intel IvyBridge cores at 3.3GHz and 3.25TB RAM. An PI needs explicitly ask support for authorization to enter the queue for all users associated to her/his Project.
+- **qfree**, the Free resource queue: The queue qfree is intended for utilization of free resources, after a Project exhausted all its allocated computational resources (Does not apply to DD projects by default. DD projects have to request for persmission on qfree after exhaustion of computational resources.). It is required that active project is specified to enter the queue, however no remaining resources are required. Consumed resources will be accounted to the Project. Only 178 nodes without accelerator may be accessed from this queue. Full nodes, 24 cores per node are allocated. The queue runs with very low priority and no special authorization is required to use it. The maximum runtime in qfree is 12 hours.
+- **qviz**, the Visualization queue: Intended for pre-/post-processing using OpenGL accelerated graphics. Currently when accessing the node, each user gets 4 cores of a CPU allocated, thus approximately 73 GB of RAM and 1/7 of the GPU capacity (default "chunk"). If more GPU power or RAM is required, it is recommended to allocate more chunks (with 4 cores each) up to one whole node per user, so that all 28 cores, 512 GB RAM and whole GPU is exclusive. This is currently also the maximum allowed allocation per one user. One hour of work is allocated by default, the user may ask for 2 hours maximum.
+
+>To access node with Xeon Phi co-processor user needs to specify that in [job submission select statement](job-submission-and-execution.md).
### Notes
-The job wall clock time defaults to **half the maximum time**, see table
-above. Longer wall time limits can be [set manually, see
-examples](job-submission-and-execution.html).
+The job wall clock time defaults to **half the maximum time**, see table above. Longer wall time limits can be [set manually, see examples](job-submission-and-execution.md).
-Jobs that exceed the reserved wall clock time (Req'd Time) get killed
-automatically. Wall clock time limit can be changed for queuing jobs
-(state Q) using the qalter command, however can not be changed for a
-running job (state R).
+Jobs that exceed the reserved wall clock time (Req'd Time) get killed automatically. Wall clock time limit can be changed for queuing jobs (state Q) using the qalter command, however can not be changed for a running job (state R).
-Salomon users may check current queue configuration at
-<https://extranet.it4i.cz/rsweb/salomon/queues>.
+Salomon users may check current queue configuration at <https://extranet.it4i.cz/rsweb/salomon/queues>.
### Queue status
-Check the status of jobs, queues and compute nodes at
-[https://extranet.it4i.cz/rsweb/salomon/](https://extranet.it4i.cz/rsweb/salomon)
-
-Â
+>Check the status of jobs, queues and compute nodes at [https://extranet.it4i.cz/rsweb/salomon/](https://extranet.it4i.cz/rsweb/salomon)
-Â
-
Display the queue status on Salomon:
-`
+```bash
$ qstat -q
-`
+```
-The PBS allocation overview may be obtained also using the rspbs
-command.
+The PBS allocation overview may be obtained also using the rspbs command.
-`
+```bash
$ rspbs
Usage: rspbs [options]
@@ -196,41 +104,26 @@ Options:
                       Only for given node state (affects only --get-node*
                       --get-qlist-* --get-ibswitch-* actions)
 --incl-finished      Include finished jobs
-`
+```
Resources Accounting Policy
-------------------------------
### The Core-Hour
-The resources that are currently subject to accounting are the
-core-hours. The core-hours are accounted on the wall clock basis. The
-accounting runs whenever the computational cores are allocated or
-blocked via the PBS Pro workload manager (the qsub command), regardless
-of whether the cores are actually used for any calculation. 1 core-hour
-is defined as 1 processor core allocated for 1 hour of wall clock time.
-Allocating a full node (24 cores) for 1 hour accounts to 24 core-hours.
-See example in the [Job submission and
-execution](job-submission-and-execution.html) section.
+The resources that are currently subject to accounting are the core-hours. The core-hours are accounted on the wall clock basis. The accounting runs whenever the computational cores are allocated or blocked via the PBS Pro workload manager (the qsub command), regardless of whether the cores are actually used for any calculation. 1 core-hour is defined as 1 processor core allocated for 1 hour of wall clock time. Allocating a full node (24 cores) for 1 hour accounts to 24 core-hours. See example in the [Job submission and execution](job-submission-and-execution.md) section.
### Check consumed resources
-The **it4ifree** command is a part of it4i.portal.clients package,
-located here:
-<https://pypi.python.org/pypi/it4i.portal.clients>
+The **it4ifree** command is a part of it4i.portal.clients package, located here: <https://pypi.python.org/pypi/it4i.portal.clients>
-User may check at any time, how many core-hours have been consumed by
-himself/herself and his/her projects. The command is available on
-clusters' login nodes.
+User may check at any time, how many core-hours have been consumed by himself/herself and his/her projects. The command is available on clusters' login nodes.
-`
+```bash
$ it4ifree
Password:
    PID  Total Used ...by me Free
  -------- ------- ------ -------- -------
  OPEN-0-0 1500000 400644  225265 1099356
  DD-13-1   10000 2606 2606 7394
-`
-
-Â
-
+```
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/ansys/ansys-cfx.md b/docs.it4i/salomon/software/ansys/ansys-cfx.md
index 39777fa74..f5cf8a8bf 100644
--- a/docs.it4i/salomon/software/ansys/ansys-cfx.md
+++ b/docs.it4i/salomon/software/ansys/ansys-cfx.md
@@ -1,87 +1,57 @@
-ANSYS CFX
+ANSYS CFX
=========
-[ANSYS
-CFX](http://www.ansys.com/Products/Simulation+Technology/Fluid+Dynamics/Fluid+Dynamics+Products/ANSYS+CFX)
-software is a high-performance, general purpose fluid dynamics program
-that has been applied to solve wide-ranging fluid flow problems for over
-20 years. At the heart of ANSYS CFX is its advanced solver technology,
-the key to achieving reliable and accurate solutions quickly and
-robustly. The modern, highly parallelized solver is the foundation for
-an abundant choice of physical models to capture virtually any type of
-phenomena related to fluid flow. The solver and its many physical models
-are wrapped in a modern, intuitive, and flexible GUI and user
-environment, with extensive capabilities for customization and
-automation using session files, scripting and a powerful expression
-language.
+[ANSYS CFX](http://www.ansys.com/Products/Simulation+Technology/Fluid+Dynamics/Fluid+Dynamics+Products/ANSYS+CFX)
+software is a high-performance, general purpose fluid dynamics program that has been applied to solve wide-ranging fluid flow problems for over 20 years. At the heart of ANSYS CFX is its advanced solver technology, the key to achieving reliable and accurate solutions quickly and robustly. The modern, highly parallelized solver is the foundation for an abundant choice of physical models to capture virtually any type of phenomena related to fluid flow. The solver and its many physical models are wrapped in a modern, intuitive, and flexible GUI and user environment, with extensive capabilities for customization and automation using session files, scripting and a powerful expression language.
-To run ANSYS CFX in batch mode you can utilize/modify the default
-cfx.pbs script and execute it via the qsub command.
+To run ANSYS CFX in batch mode you can utilize/modify the default cfx.pbs script and execute it via the qsub command.
- #!/bin/bash
- #PBS -l nodes=2:ppn=24
- #PBS -q qprod
- #PBS -N $USER-CFX-Project
- #PBS -A OPEN-0-0
+```bash
+#!/bin/bash
+#PBS -l nodes=2:ppn=16
+#PBS -q qprod
+#PBS -N $USER-CFX-Project
+#PBS -A XX-YY-ZZ
- #! Mail to user when job terminate or abort
- #PBS -m ae
+#! Mail to user when job terminate or abort
+#PBS -m ae
- #!change the working directory (default is home directory)
- #cd <working directory> (working directory must exists)
- WORK_DIR="/scratch/work/user/$USER"
- cd $WORK_DIR
+#!change the working directory (default is home directory)
+#cd <working directory> (working directory must exists)
+WORK_DIR="/scratch/$USER/work"
+cd $WORK_DIR
- echo Running on host `hostname`
- echo Time is `date`
- echo Directory is `pwd`
- echo This jobs runs on the following processors:
- echo `cat $PBS_NODEFILE`
+echo Running on host `hostname`
+echo Time is `date`
+echo Directory is `pwd`
+echo This jobs runs on the following processors:
+echo `cat $PBS_NODEFILE`
- module load ANSYS
+module load ansys
- #### Set number of processors per host listing
- procs_per_host=24
- #### Create host list
- hl=""
- for host in `cat $PBS_NODEFILE`
- do
- if [ "$hl" = "" ]
- then hl="$host:$procs_per_host"
- else hl="${hl}:$host:$procs_per_host"
- fi
- done
+#### Set number of processors per host listing
+#### (set to 1 as $PBS_NODEFILE lists each node twice if :ppn=2)
+procs_per_host=1
+#### Create host list
+hl=""
+for host in `cat $PBS_NODEFILE`
+do
+ if [ "$hl" = "" ]
+ then hl="$host:$procs_per_host"
+ else hl="${hl}:$host:$procs_per_host"
+ fi
+done
- echo Machines: $hl
+echo Machines: $hl
- # prevent ANSYS from attempting to use scif0 interface
- export MPI_IC_ORDER="UDAPL"
+#-dev input.def includes the input of CFX analysis in DEF format
+#-P the name of prefered license feature (aa_r=ANSYS Academic Research, ane3fl=Multiphysics(commercial))
+/ansys_inc/v145/CFX/bin/cfx5solve -def input.def -size 4 -size-ni 4x -part-large -start-method "Platform MPI Distributed Parallel" -par-dist $hl -P aa_r
+```
- #-dev input.def includes the input of CFX analysis in DEF format
- #-P the name of prefered license feature (aa_r=ANSYS Academic Research, ane3fl=Multiphysics(commercial))
- cfx5solve -def input.def -size 4 -size-ni 4x -part-large -start-method "Platform MPI Distributed Parallel" -par-dist $hl -P aa_r
+Header of the pbs file (above) is common and description can be find on [this site](../../resource-allocation-and-job-execution/job-submission-and-execution.md). SVS FEM recommends to utilize sources by keywords: nodes, ppn. These keywords allows to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources.
-Header of the pbs file (above) is common and description can be findÂ
-[this
-site](../../resource-allocation-and-job-execution/job-submission-and-execution.html).
-SVS FEM recommends to utilize sources by keywords: nodes, ppn. These
-keywords allows to address directly the number of nodes (computers) and
-cores (ppn) which will be utilized in the job. Also the rest of code
-assumes such structure of allocated resources.
-
-Working directory has to be created before sending pbs job into the
-queue. Input file should be in working directory or full path to input
-file has to be specified. >Input file has to be defined by common
-CFX def file which is attached to the cfx solver via parameter
--def
-
-License**Â should be selected by parameter -P (Big letter **P**).
-Licensed products are the following: aa_r
-(ANSYS **Academic Research), ane3fl (ANSYS
-Multiphysics)-**Commercial.
-[More about licensing here](licensing.html)
-
-Â We have observed that the -P settings does not always work. Please set
-your [license
-preferences](setting-license-preferences.html) instead.
+Working directory has to be created before sending pbs job into the queue. Input file should be in working directory or full path to input file has to be specified. >Input file has to be defined by common CFX def file which is attached to the cfx solver via parameter -def
+**License**Â should be selected by parameter -P (Big letter **P**). Licensed products are the following: aa_r (ANSYSÂ **Academic**Â Research), ane3fl (ANSYS Multiphysics)-**Commercial**.
+[More about licensing here](licensing.md)
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/ansys/ansys-fluent.md b/docs.it4i/salomon/software/ansys/ansys-fluent.md
index bbacf3ea5..8fe15741c 100644
--- a/docs.it4i/salomon/software/ansys/ansys-fluent.md
+++ b/docs.it4i/salomon/software/ansys/ansys-fluent.md
@@ -1,71 +1,51 @@
-ANSYS Fluent
+ANSYS Fluent
============
-[ANSYS
-Fluent](http://www.ansys.com/Products/Simulation+Technology/Fluid+Dynamics/Fluid+Dynamics+Products/ANSYS+Fluent)
-software contains the broad physical modeling capabilities needed to
-model flow, turbulence, heat transfer, and reactions for industrial
-applications ranging from air flow over an aircraft wing to combustion
-in a furnace, from bubble columns to oil platforms, from blood flow to
-semiconductor manufacturing, and from clean room design to wastewater
-treatment plants. Special models that give the software the ability to
-model in-cylinder combustion, aeroacoustics, turbomachinery, and
-multiphase systems have served to broaden its reach.
+[ANSYS Fluent](http://www.ansys.com/Products/Simulation+Technology/Fluid+Dynamics/Fluid+Dynamics+Products/ANSYS+Fluent)
+software contains the broad physical modeling capabilities needed to model flow, turbulence, heat transfer, and reactions for industrial applications ranging from air flow over an aircraft wing to combustion in a furnace, from bubble columns to oil platforms, from blood flow to semiconductor manufacturing, and from clean room design to wastewater treatment plants. Special models that give the software the ability to model in-cylinder combustion, aeroacoustics, turbomachinery, and multiphase systems have served to broaden its reach.
1. Common way to run Fluent over pbs file
------------------------------------------------------
+To run ANSYS Fluent in batch mode you can utilize/modify the default fluent.pbs script and execute it via the qsub command.
-To run ANSYS Fluent in batch mode you can utilize/modify the
-default fluent.pbs script and execute it via the qsub command.
-
- #!/bin/bash
- #PBS -S /bin/bash
- #PBS -l nodes=2:ppn=24
- #PBS -q qprod
- #PBS -N Fluent-Project
- #PBS -A OPEN-0-0
-
- #! Mail to user when job terminate or abort
- #PBS -m ae
-
- #!change the working directory (default is home directory)
- #cd <working directory> (working directory must exists)
- WORK_DIR="/scratch/work/user/$USER"
- cd $WORK_DIR
-
- echo Running on host `hostname`
- echo Time is `date`
- echo Directory is `pwd`
- echo This jobs runs on the following processors:
- echo `cat $PBS_NODEFILE`
-
- #### Load ansys module so that we find the cfx5solve command
- module load ANSYS
-
- # Use following line to specify MPI for message-passing instead
- NCORES=`wc -l $PBS_NODEFILE |awk '{print $1}'`
-
- /apps/cae/ANSYS/16.1/v161/fluent/bin/fluent 3d -t$NCORES -cnf=$PBS_NODEFILE -g -i fluent.jou
-
-Header of the pbs file (above) is common and description can be find on
-[this
-site](../../resource-allocation-and-job-execution/job-submission-and-execution.html).
-[SVS FEM](http://www.svsfem.cz) recommends to utilize
-sources by keywords: nodes, ppn. These keywords allows to address
-directly the number of nodes (computers) and cores (ppn) which will be
-utilized in the job. Also the rest of code assumes such structure of
-allocated resources.
-
-Working directory has to be created before sending pbs job into the
-queue. Input file should be in working directory or full path to input
-file has to be specified. Input file has to be defined by common Fluent
-journal file which is attached to the Fluent solver via parameter -i
-fluent.jou
-
-Journal file with definition of the input geometry and boundary
-conditions and defined process of solution has e.g. the following
-structure:
+```bash
+#!/bin/bash
+#PBS -S /bin/bash
+#PBS -l nodes=2:ppn=16
+#PBS -q qprod
+#PBS -N $USER-Fluent-Project
+#PBS -A XX-YY-ZZ
+#! Mail to user when job terminate or abort
+#PBS -m ae
+
+#!change the working directory (default is home directory)
+#cd <working directory> (working directory must exists)
+WORK_DIR="/scratch/$USER/work"
+cd $WORK_DIR
+
+echo Running on host `hostname`
+echo Time is `date`
+echo Directory is `pwd`
+echo This jobs runs on the following processors:
+echo `cat $PBS_NODEFILE`
+
+#### Load ansys module so that we find the cfx5solve command
+module load ansys
+
+# Use following line to specify MPI for message-passing instead
+NCORES=`wc -l $PBS_NODEFILE |awk '{print $1}'`
+
+/ansys_inc/v145/fluent/bin/fluent 3d -t$NCORES -cnf=$PBS_NODEFILE -g -i fluent.jou
+```
+
+Header of the pbs file (above) is common and description can be find on [this site](../../resource-allocation-and-job-execution/job-submission-and-execution.md). [SVS FEM](http://www.svsfem.cz) recommends to utilize sources by keywords: nodes, ppn. These keywords allows to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources.
+
+Working directory has to be created before sending pbs job into the queue. Input file should be in working directory or full path to input file has to be specified. Input file has to be defined by common Fluent journal file which is attached to the Fluent solver via parameter -i fluent.jou
+
+Journal file with definition of the input geometry and boundary conditions and defined process of solution has e.g. the following structure:
+
+```bash
/file/read-case aircraft_2m.cas.gz
/solve/init
init
@@ -73,83 +53,54 @@ structure:
10
/file/write-case-dat aircraft_2m-solution
/exit yes
+```
-The appropriate dimension of the problem has to be set by
-parameter (2d/3d).Â
+The appropriate dimension of the problem has to be set by parameter (2d/3d).
2. Fast way to run Fluent from command line
--------------------------------------------------------
- fluent solver_version [FLUENT_options] -i journal_file -pbs
+```bash
+fluent solver_version [FLUENT_options] -i journal_file -pbs
+```
-This syntax will start the ANSYS FLUENT job under PBS Professional using
-the qsub command in a batch manner. When
-resources are available, PBS Professional will start the job and return
-a job ID, usually in the form of
-*job_ID.hostname*. This job ID can then be used
-to query, control, or stop the job using standard PBS Professional
-commands, such as qstat or
-qdel. The job will be run out of the current
-working directory, and all output will be written to the file
-fluent.o>Â
-*job_ID*. Â Â Â Â
+This syntax will start the ANSYS FLUENT job under PBS Professional using the qsub command in a batch manner. When resources are available, PBS Professional will start the job and return a job ID, usually in the form of *job_ID.hostname*. This job ID can then be used to query, control, or stop the job using standard PBS Professional commands, such as qstat or qdel. The job will be run out of the current working directory, and all output will be written to the file fluent.o *job_ID*.
3. Running Fluent via user's config file
----------------------------------------
+The sample script uses a configuration file called pbs_fluent.conf  if no command line arguments are present. This configuration file should be present in the directory from which the jobs are submitted (which is also the directory in which the jobs are executed). The following is an example of what the content of pbs_fluent.conf can be:
-The sample script uses a configuration file called
-pbs_fluent.conf  if no command line arguments
-are present. This configuration file should be present in the directory
-from which the jobs are submitted (which is also the directory in which
-the jobs are executed). The following is an example of what the content
-of pbs_fluent.conf can be:
-
-`
- input="example_small.flin"
- case="Small-1.65m.cas"
- fluent_args="3d -pmyrinet"
- outfile="fluent_test.out"
- mpp="true"
-`
+```bash
+input="example_small.flin"
+case="Small-1.65m.cas"
+fluent_args="3d -pmyrinet"
+outfile="fluent_test.out"
+mpp="true"
+```
The following is an explanation of the parameters:
- input is the name of the input
-file.
+input is the name of the input file.
- case is the name of the
-.cas file that the input file will utilize.
+case is the name of the .cas file that the input file will utilize.
- fluent_args are extra ANSYS FLUENT
-arguments. As shown in the previous example, you can specify the
-interconnect by using the -p interconnect
-command. The available interconnects include
-ethernet (the default),
-myrinet, class="monospace">
-infiniband, vendor,
-altix>, and
-crayx. The MPI is selected automatically, based
-on the specified interconnect.
+fluent_args are extra ANSYS FLUENT arguments. As shown in the previous example, you can specify the interconnect by using the -p interconnect command. The available interconnects include ethernet (the default), myrinet, infiniband, vendor, altix, and crayx. The MPI is selected automatically, based on the specified interconnect.
- outfile is the name of the file to which
-the standard output will be sent.
+outfile is the name of the file to which the standard output will be sent.
- mpp="true" will tell the job script to
-execute the job across multiple processors. Â Â Â Â Â Â Â Â
+mpp="true" will tell the job script to execute the job across multiple processors.
-To run ANSYS Fluent in batch mode with user's config file you can
-utilize/modify the following script and execute it via the qsub
-command.
+To run ANSYS Fluent in batch mode with user's config file you can utilize/modify the following script and execute it via the qsub command.
-`
+```bash
#!/bin/sh
-#PBS -l nodes=2:ppn=24
+#PBS -l nodes=2:ppn=4
#PBS -1 qprod
-#PBS -N Fluent-Project
-#PBS -A OPEN-0-0
+#PBS -N $USE-Fluent-Project
+#PBS -A XX-YY-ZZ
cd $PBS_O_WORKDIR
-
+
#We assume that if they didn’t specify arguments then they should use the
#config file if [ "xx${input}${case}${mpp}${fluent_args}zz" = "xxzz" ]; then
if [ -f pbs_fluent.conf ]; then
@@ -159,7 +110,7 @@ command.
printf "and no configuration file found. Exiting n"
fi
fi
-
+
#Augment the ANSYS FLUENT command line arguments case "$mpp" in
true)
@@ -185,22 +136,29 @@ command.
Case: $case
Output: $outfile
Fluent arguments: $fluent_args"
-
+
#run the solver
- /apps/cae/ANSYS/16.1/v161/fluent/bin/fluent $fluent_args > $outfile
-`
+ /ansys_inc/v145/fluent/bin/fluent $fluent_args > $outfile
+```
-It runs the jobs out of the directory from which they are
-submitted (PBS_O_WORKDIR).
+It runs the jobs out of the directory from which they are submitted (PBS_O_WORKDIR).
4. Running Fluent in parralel
-----------------------------
+Fluent could be run in parallel only under Academic Research license. To do so this ANSYS Academic Research license must be placed before ANSYS CFD license in user preferences. To make this change anslic_admin utility should be run
+
+```bash
+/ansys_inc/shared_les/licensing/lic_admin/anslic_admin
+```
+
+ANSLIC_ADMIN Utility will be run
+
+
+
+
-Fluent could be run in parallel only under Academic Research
-license. To do so this ANSYS Academic Research license must be placed
-before ANSYS CFD license in user preferences. To make this change
-[anslic_admin utility should be
-run](setting-license-preferences.html).
+
-Â
+ANSYS Academic Research license should be moved up to the top of the list.
+
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/ansys/ansys-ls-dyna.md b/docs.it4i/salomon/software/ansys/ansys-ls-dyna.md
index 6120f49e8..c2b188646 100644
--- a/docs.it4i/salomon/software/ansys/ansys-ls-dyna.md
+++ b/docs.it4i/salomon/software/ansys/ansys-ls-dyna.md
@@ -1,84 +1,56 @@
-ANSYS LS-DYNA
+ANSYS LS-DYNA
=============
-[ANSYS
-LS-DYNA](http://www.ansys.com/Products/Simulation+Technology/Structural+Mechanics/Explicit+Dynamics/ANSYS+LS-DYNA)
-software provides convenient and easy-to-use access to the
-technology-rich, time-tested explicit solver without the need to contend
-with the complex input requirements of this sophisticated program.
-Introduced in 1996, ANSYS LS-DYNA capabilities have helped customers in
-numerous industries to resolve highly intricate design
-issues. >ANSYS Mechanical users have been able take advantage of
-complex explicit solutions for a long time utilizing the traditional
-ANSYS Parametric Design Language (APDL) environment. >These
-explicit capabilities are available to ANSYS Workbench users as well.
-The Workbench platform is a powerful, comprehensive, easy-to-use
-environment for engineering simulation. CAD import from all sources,
-geometry cleanup, automatic meshing, solution, parametric optimization,
-result visualization and comprehensive report generation are all
-available within a single fully interactive modern graphical user
-environment.
+**[ANSYSLS-DYNA](http://www.ansys.com/Products/Simulation+Technology/Structural+Mechanics/Explicit+Dynamics/ANSYS+LS-DYNA)** software provides convenient and easy-to-use access to the technology-rich, time-tested explicit solver without the need to contend with the complex input requirements of this sophisticated program. Introduced in 1996, ANSYS LS-DYNA capabilities have helped customers in numerous industries to resolve highly intricate design issues. ANSYS Mechanical users have been able take advantage of complex explicit solutions for a long time utilizing the traditional ANSYS Parametric Design Language (APDL) environment. These explicit capabilities are available to ANSYS Workbench users as well. The Workbench platform is a powerful, comprehensive, easy-to-use environment for engineering simulation. CAD import from all sources, geometry cleanup, automatic meshing, solution, parametric optimization, result visualization and comprehensive report generation are all available within a single fully interactive modern graphical user environment.
-To run ANSYS LS-DYNA in batch mode you can utilize/modify the
-default ansysdyna.pbs script and execute it via the qsub command.
+To run ANSYS LS-DYNA in batch mode you can utilize/modify the default ansysdyna.pbs script and execute it via the qsub command.
- #!/bin/bash
- #PBS -l nodes=2:ppn=24
- #PBS -q qprod
- #PBS -N DYNA-Project
- #PBS -A OPEN-0-0
+```bash
+#!/bin/bash
+#PBS -l nodes=2:ppn=16
+#PBS -q qprod
+#PBS -N $USER-DYNA-Project
+#PBS -A XX-YY-ZZ
- #! Mail to user when job terminate or abort
- #PBS -m ae
+#! Mail to user when job terminate or abort
+#PBS -m ae
- #!change the working directory (default is home directory)
- #cd <working directory>
- WORK_DIR="/scratch/work/user/$USER"
- cd $WORK_DIR
+#!change the working directory (default is home directory)
+#cd <working directory>
+WORK_DIR="/scratch/$USER/work"
+cd $WORK_DIR
- echo Running on host `hostname`
- echo Time is `date`
- echo Directory is `pwd`
- echo This jobs runs on the following processors:
- echo `cat $PBS_NODEFILE`
+echo Running on host `hostname`
+echo Time is `date`
+echo Directory is `pwd`
+echo This jobs runs on the following processors:
+echo `cat $PBS_NODEFILE`
- module load ANSYS
+#! Counts the number of processors
+NPROCS=`wc -l < $PBS_NODEFILE`
- #### Set number of processors per node
- procs_per_host=24
- #### Create host list
- hl=""
- for host in `cat $PBS_NODEFILE`
- do
- if [ "$hl" = "" ]
- then hl="$host:$procs_per_host"
- else hl="${hl}:$host:$procs_per_host"
- fi
- done
+echo This job has allocated $NPROCS nodes
- echo Machines: $hl
+module load ansys
- # prevent ANSYS from attempting to use scif0 interface
- export MPI_IC_ORDER="UDAPL"
+#### Set number of processors per host listing
+#### (set to 1 as $PBS_NODEFILE lists each node twice if :ppn=2)
+procs_per_host=1
+#### Create host list
+hl=""
+for host in `cat $PBS_NODEFILE`
+do
+ if [ "$hl" = "" ]
+ then hl="$host:$procs_per_host"
+ else hl="${hl}:$host:$procs_per_host"
+ fi
+done
- lsdyna161 -dis -usessh -machines "$hl" i=input.k
+echo Machines: $hl
-Header of the pbs file (above) is common and description can be
-find > on [this
-site](../../resource-allocation-and-job-execution/job-submission-and-execution.html).
-[SVS FEM](http://www.svsfem.cz) recommends to utilize
-sources by keywords: nodes, ppn. These keywords allows to address
-directly the number of nodes (computers) and cores (ppn) which will be
-utilized in the job. Also the rest of code assumes such structure of
-allocated resources.
+/ansys_inc/v145/ansys/bin/ansys145 -dis -lsdynampp i=input.k -machines $hl
+```
-Working directory has to be created before sending pbs job into the
-queue. Input file should be in working directory or full path to input
-file has to be specified. Input file has to be defined by common LS-DYNA
-.**k** file which is attached to the ansys solver via parameter i=
-
-Without setting environment variable MPI_IC_ORDER="UDAPL", ANSYS will
-fail to run on nodes with Xeon Phi accelerator (it will use the virtual
-interface of Phi cards instead of the real InfiniBand interface and MPI
-will fail.
+Header of the pbs file (above) is common and description can be find on [this site](../../resource-allocation-and-job-execution/job-submission-and-execution.md). [SVS FEM](http://www.svsfem.cz) recommends to utilize sources by keywords: nodes, ppn. These keywords allows to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources.
+Working directory has to be created before sending pbs job into the queue. Input file should be in working directory or full path to input file has to be specified. Input file has to be defined by common LS-DYNA .**k** file which is attached to the ansys solver via parameter i=
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/ansys/ansys-mechanical-apdl.md b/docs.it4i/salomon/software/ansys/ansys-mechanical-apdl.md
index a33a8bf91..56d5f78aa 100644
--- a/docs.it4i/salomon/software/ansys/ansys-mechanical-apdl.md
+++ b/docs.it4i/salomon/software/ansys/ansys-mechanical-apdl.md
@@ -1,79 +1,58 @@
-ANSYS MAPDL
+ANSYS MAPDL
===========
-**[ANSYS
-Multiphysics](http://www.ansys.com/Products/Simulation+Technology/Structural+Mechanics/ANSYS+Multiphysics)**
-software offers a comprehensive product solution for both multiphysics
-and single-physics analysis. The product includes structural, thermal,
-fluid and both high- and low-frequency electromagnetic analysis. The
-product also contains solutions for both direct and sequentially coupled
-physics problems including direct coupled-field elements and the ANSYS
-multi-field solver.
-
-To run ANSYS MAPDL in batch mode you can utilize/modify the
-default mapdl.pbs script and execute it via the qsub command.
-
- #!/bin/bash
- #PBS -l nodes=2:ppn=24
- #PBS -q qprod
- #PBS -N ANSYS-Project
- #PBS -A OPEN-0-0
-
- #! Mail to user when job terminate or abort
- #PBS -m ae
-
- #!change the working directory (default is home directory)
- #cd <working directory> (working directory must exists)
- WORK_DIR="/scratch/work/user/$USER"
- cd $WORK_DIR
-
- echo Running on host `hostname`
- echo Time is `date`
- echo Directory is `pwd`
- echo This jobs runs on the following processors:
- echo `cat $PBS_NODEFILE`
-
- module load ANSYS/16.1
-
- #### Set number of processors per host listing
- procs_per_host=24
- #### Create host list
- hl=""
- for host in `cat $PBS_NODEFILE`
- do
- if [ "$hl" = "" ]
- then hl="$host:$procs_per_host"
- else hl="${hl}:$host:$procs_per_host"
- fi
- done
-
- echo Machines: $hl
-
- # prevent ANSYS from attempting to use scif0 interface
- export MPI_IC_ORDER="UDAPL"
-
- #-i input.dat includes the input of analysis in APDL format
- #-o file.out is output file from ansys where all text outputs will be redirected
- #-p the name of license feature (aa_r=ANSYS Academic Research, ane3fl=Multiphysics(commercial), aa_r_dy=Academic AUTODYN)
- ansys161 -b -dis -usessh -p aa_r -i input.dat -o file.out -machines "$hl" -dir $WORK_DIR
-
-Header of the PBS file (above) is common and description can be find on
-[this
-site](../../resource-allocation-and-job-execution/job-submission-and-execution.html).
-[SVS FEM](http://www.svsfem.cz) recommends to utilize
-sources by keywords: nodes, ppn. These keywords allows to address
-directly the number of nodes (computers) and cores (ppn) which will be
-utilized in the job. Also the rest of code assumes such structure of
-allocated resources.
-
-Working directory has to be created before sending pbs job into the
-queue. Input file should be in working directory or full path to input
-file has to be specified. Input file has to be defined by common APDL
-file which is attached to the ansys solver via parameter -i
-
-License** should be selected by parameter -p. Licensed products are
-the following: aa_r (ANSYS **Academic Research), ane3fl (ANSYS
-Multiphysics)-**Commercial**, aa_r_dy (ANSYS **Academic
-AUTODYN)>
-[More about licensing here](licensing.html)
-
+**[ANSYS Multiphysics](http://www.ansys.com/Products/Simulation+Technology/Structural+Mechanics/ANSYS+Multiphysics)**
+software offers a comprehensive product solution for both multiphysics and single-physics analysis. The product includes structural, thermal, fluid and both high- and low-frequency electromagnetic analysis. The product also contains solutions for both direct and sequentially coupled physics problems including direct coupled-field elements and the ANSYS multi-field solver.
+
+To run ANSYS MAPDL in batch mode you can utilize/modify the default mapdl.pbs script and execute it via the qsub command.
+
+```bash
+#!/bin/bash
+#PBS -l nodes=2:ppn=16
+#PBS -q qprod
+#PBS -N $USER-ANSYS-Project
+#PBS -A XX-YY-ZZ
+
+#! Mail to user when job terminate or abort
+#PBS -m ae
+
+#!change the working directory (default is home directory)
+#cd <working directory> (working directory must exists)
+WORK_DIR="/scratch/$USER/work"
+cd $WORK_DIR
+
+echo Running on host `hostname`
+echo Time is `date`
+echo Directory is `pwd`
+echo This jobs runs on the following processors:
+echo `cat $PBS_NODEFILE`
+
+module load ansys
+
+#### Set number of processors per host listing
+#### (set to 1 as $PBS_NODEFILE lists each node twice if :ppn=2)
+procs_per_host=1
+#### Create host list
+hl=""
+for host in `cat $PBS_NODEFILE`
+do
+ if [ "$hl" = "" ]
+ then hl="$host:$procs_per_host"
+ else hl="${hl}:$host:$procs_per_host"
+ fi
+done
+
+echo Machines: $hl
+
+#-i input.dat includes the input of analysis in APDL format
+#-o file.out is output file from ansys where all text outputs will be redirected
+#-p the name of license feature (aa_r=ANSYS Academic Research, ane3fl=Multiphysics(commercial), aa_r_dy=Academic AUTODYN)
+/ansys_inc/v145/ansys/bin/ansys145 -b -dis -p aa_r -i input.dat -o file.out -machines $hl -dir $WORK_DIR
+```
+
+Header of the pbs file (above) is common and description can be find on [this site](../../resource-allocation-and-job-execution/job-submission-and-execution.md). [SVS FEM](http://www.svsfem.cz) recommends to utilize sources by keywords: nodes, ppn. These keywords allows to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources.
+
+Working directory has to be created before sending pbs job into the queue. Input file should be in working directory or full path to input file has to be specified. Input file has to be defined by common APDL file which is attached to the ansys solver via parameter -i
+
+**License** should be selected by parameter -p. Licensed products are the following: aa_r (ANSYS **Academic** Research), ane3fl (ANSYS Multiphysics)-**Commercial**, aa_r_dy (ANSYS **Academic** AUTODYN)
+[More about licensing here](licensing.md)
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/ansys/ansys-products-mechanical-fluent-cfx-mapdl.md b/docs.it4i/salomon/software/ansys/ansys-products-mechanical-fluent-cfx-mapdl.md
deleted file mode 100644
index 43dffc3a1..000000000
--- a/docs.it4i/salomon/software/ansys/ansys-products-mechanical-fluent-cfx-mapdl.md
+++ /dev/null
@@ -1,31 +0,0 @@
-Overview of ANSYS Products
-==========================
-
-[SVS FEM](http://www.svsfem.cz/)** as **[ANSYS
-Channel partner](http://www.ansys.com/)**Â for Czech
-Republic provided all ANSYS licenses for all clusters and supports of
-all ANSYS Products (Multiphysics, Mechanical, MAPDL, CFX, Fluent,
-Maxwell, LS-DYNA...) to IT staff and ANSYS users. If you are challenging
-to problem of ANSYS functionality contact
-please [hotline@svsfem.cz](mailto:hotline@svsfem.cz?subject=Ostrava%20-%20ANSELM)
-
-The clusters provides as commercial as academic variants. Academic
-variants are distinguished by "**Academic...**" word in the name of
-Â license or by two letter preposition "**aa_**" in the license feature
-name. Change of license is realized on command line respectively
-directly in user's pbs file (see individual products). [More about
-licensing here](licensing.html)
-
-To load the latest version of any ANSYS product (Mechanical, Fluent,
-CFX, MAPDL,...) load the module:
-
- $ module load ANSYS
-
-ANSYS supports interactive regime, but due to assumed solution of
-extremely difficult tasks it is not recommended.
-
-If user needs to work in interactive regime we recommend to configure
-the RSM service on the client machine which allows to forward the
-solution to the clusters directly from the client's Workbench project
-(see ANSYS RSM service).
-
diff --git a/docs.it4i/salomon/software/ansys/ansys.md b/docs.it4i/salomon/software/ansys/ansys.md
index 93f9c151d..53e175f0f 100644
--- a/docs.it4i/salomon/software/ansys/ansys.md
+++ b/docs.it4i/salomon/software/ansys/ansys.md
@@ -1,31 +1,17 @@
-Overview of ANSYS Products
+Overview of ANSYS Products
==========================
-[SVS FEM](http://www.svsfem.cz/)** as **[ANSYS
-Channel partner](http://www.ansys.com/)**Â for Czech
-Republic provided all ANSYS licenses for all clusters and supports of
-all ANSYS Products (Multiphysics, Mechanical, MAPDL, CFX, Fluent,
-Maxwell, LS-DYNA...) to IT staff and ANSYS users. If you are challenging
-to problem of ANSYS functionality contact
-please [hotline@svsfem.cz](mailto:hotline@svsfem.cz?subject=Ostrava%20-%20ANSELM)
+**[SVS FEM](http://www.svsfem.cz/)** as **[ANSYS Channel partner](http://www.ansys.com/)** for Czech Republic provided all ANSYS licenses for ANSELM cluster and supports of all ANSYS Products (Multiphysics, Mechanical, MAPDL, CFX, Fluent, Maxwell, LS-DYNA...) to IT staff and ANSYS users. If you are challenging to problem of ANSYS functionality contact please [hotline@svsfem.cz](mailto:hotline@svsfem.cz?subject=Ostrava%20-%20ANSELM)
-The clusters provides as commercial as academic variants. Academic
-variants are distinguished by "**Academic...**" word in the name of
-Â license or by two letter preposition "**aa_**" in the license feature
-name. Change of license is realized on command line respectively
-directly in user's pbs file (see individual products). [More about
-licensing here](ansys/licensing.html)
+Anselm provides as commercial as academic variants. Academic variants are distinguished by "**Academic...**" word in the name of  license or by two letter preposition "**aa_**" in the license feature name. Change of license is realized on command line respectively directly in user's pbs file (see individual products). [ More about licensing here](ansys/licensing.html)
-To load the latest version of any ANSYS product (Mechanical, Fluent,
-CFX, MAPDL,...) load the module:
+To load the latest version of any ANSYS product (Mechanical, Fluent, CFX, MAPDL,...) load the module:
- $ module load ANSYS
+```bash
+ $ module load ansys
+```
-ANSYS supports interactive regime, but due to assumed solution of
-extremely difficult tasks it is not recommended.
+ANSYS supports interactive regime, but due to assumed solution of extremely difficult tasks it is not recommended.
-If user needs to work in interactive regime we recommend to configure
-the RSM service on the client machine which allows to forward the
-solution to the clusters directly from the client's Workbench project
-(see ANSYS RSM service).
+If user needs to work in interactive regime we recommend to configure the RSM service on the client machine which allows to forward the solution to the Anselm directly from the client's Workbench project (see ANSYS RSM service).
diff --git a/docs.it4i/salomon/software/ansys/licensing.md b/docs.it4i/salomon/software/ansys/licensing.md
index 8ee4bc1c6..eb87b1545 100644
--- a/docs.it4i/salomon/software/ansys/licensing.md
+++ b/docs.it4i/salomon/software/ansys/licensing.md
@@ -1,45 +1,25 @@
-Licensing and Available Versions
+Licensing and Available Versions
================================
ANSYS licence can be used by:
-----------------------------
-
-- all persons in the carrying out of the CE IT4Innovations Project (In
- addition to the primary licensee, which is VSB - Technical
- University of Ostrava, users are CE IT4Innovations third parties -
- CE IT4Innovations project partners, particularly the University of
- Ostrava, the Brno University of Technology - Faculty of Informatics,
- the Silesian University in Opava, Institute of Geonics AS CR.)
-- all
- persons who have a valid
- license
-- students
- of the Technical University
+- all persons in the carrying out of the CE IT4Innovations Project (In addition to the primary licensee, which is VSB - Technical University of Ostrava, users are CE IT4Innovations third parties - CE IT4Innovations project partners, particularly the University of Ostrava, the Brno University of Technology - Faculty of Informatics, the Silesian University in Opava, Institute of Geonics AS CR.)
+- all persons who have a valid license
+- students of the Technical University
ANSYS Academic Research
-----------------------
-
-The licence intended to be used for science and research, publications,
-students’ projects (academic licence).
+The licence intended to be used for science and research, publications, students’ projects (academic licence).
ANSYS COM
---------
-
-The licence intended to be used for science and research, publications,
-students’ projects, commercial research with no commercial use
-restrictions.
+The licence intended to be used for science and research, publications, students’ projects, commercial research with no commercial use restrictions.
Available Versions
------------------
-
- 16.1
- 17.0
License Preferences
-------------------
-
-Please [see this page to set license
-preferences](setting-license-preferences.html).
-
-Â
-
+Please [see this page to set license preferences](setting-license-preferences.md).
diff --git a/docs.it4i/salomon/software/ansys/setting-license-preferences.md b/docs.it4i/salomon/software/ansys/setting-license-preferences.md
index 5c6d63842..2132ca04c 100644
--- a/docs.it4i/salomon/software/ansys/setting-license-preferences.md
+++ b/docs.it4i/salomon/software/ansys/setting-license-preferences.md
@@ -1,19 +1,15 @@
-Setting license preferences
+Setting license preferences
===========================
-Some ANSYS tools allow you to explicitly specify usage of academic or
-commercial licenses in the command line (eg.Â
-ansys161 -p aa_r to select Academic Research
-license). However, we have observed that not all tools obey this option
-and choose commercial license.
+Some ANSYS tools allow you to explicitly specify usage of academic or commercial licenses in the command line (eg. ansys161 -p aa_r to select Academic Research license). However, we have observed that not all tools obey this option and choose commercial license.
-Thus you need to configure preferred license order with ANSLIC_ADMIN.
-Please follow these steps and move Academic Research license to the  top
-or bottom of the list accordingly.
+Thus you need to configure preferred license order with ANSLIC_ADMIN. Please follow these steps and move Academic Research license to the  top or bottom of the list accordingly.
Launch the ANSLIC_ADMIN utility in a graphical environment:
+```bash
$ANSYSLIC_DIR/lic_admin/anslic_admin
+```
ANSLIC_ADMIN Utility will be run
@@ -23,12 +19,6 @@ ANSLIC_ADMIN Utility will be run
-Â
-
-ANSYS Academic Research license should be moved up to the top or down to
-the bottom of the list.
-
-Â
-
-
+ANSYS Academic Research license should be moved up to the top or down to the bottom of the list.
+
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/ansys/workbench.md b/docs.it4i/salomon/software/ansys/workbench.md
index 78ec8caa5..298a03a90 100644
--- a/docs.it4i/salomon/software/ansys/workbench.md
+++ b/docs.it4i/salomon/software/ansys/workbench.md
@@ -1,32 +1,23 @@
-Workbench
+Workbench
=========
Workbench Batch Mode
--------------------
-
-It is possible to run Workbench scripts in batch mode. You need to
-configure solvers of individual components to run in parallel mode. Open
-your project in Workbench. Then, for example, in Mechanical, go to Tools
-- Solve Process Settings ..., click Advanced button as shown on the
-screenshot.
+It is possible to run Workbench scripts in batch mode. You need to configure solvers of individual components to run in parallel mode. Open your project in Workbench. Then, for example, in Mechanical, go to Tools - Solve Process Settings ..., click Advanced button as shown on the screenshot.
-Enable Distribute Solution checkbox and enter number of cores (eg. 48 to
-run on two Salomon nodes). If you want the job to run on more then 1
-node, you must also provide a so called MPI appfile. In the Additional
-Command Line Arguments input field, enter :
+Enable Distribute Solution checkbox and enter number of cores (eg. 48 to run on two Salomon nodes). If you want the job to run on more then 1 node, you must also provide a so called MPI appfile. In the Additional Command Line Arguments input field, enter:
+```bash
-mpifile /path/to/my/job/mpifile.txt
+```
-Where /path/to/my/job is the directory where your project is saved. We
-will create the file mpifile.txt programatically later in the batch
-script. For more information, refer to *ANSYS Mechanical APDL Parallel
-Processing* *Guide*.
+Where /path/to/my/job is the directory where your project is saved. We will create the file mpifile.txt programatically later in the batch script. For more information, refer to *ANSYS Mechanical APDL Parallel Processing* *Guide*.
-Now, save the project and close Workbench. We will use this script to
-launch the job:
+Now, save the project and close Workbench. We will use this script to launch the job:
+```bash
#!/bin/bash
#PBS -l select=2:ncpus=24
#PBS -q qprod
@@ -36,7 +27,7 @@ launch the job:
# Mail to user when job terminate or abort
#PBS -m a
- # change the working directory
+ # change the working directory
WORK_DIR="$PBS_O_WORKDIR"
cd $WORK_DIR
@@ -67,8 +58,6 @@ launch the job:
export MPI_REMSH="/usr/bin/ssh"
runwb2 -R jou6.wbjn -B -F test9.wbpj
+```
-The solver settings are saved in file solvehandlers.xml, which is not
-located in the project directory. Verify your solved settings when
-uploading a project from your local computer.
-
+The solver settings are saved in file solvehandlers.xml, which is not located in the project directory. Verify your solved settings when uploading a project from your local computer.
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/chemistry/molpro.md b/docs.it4i/salomon/software/chemistry/molpro.md
index 44f8d9233..f993cf58b 100644
--- a/docs.it4i/salomon/software/chemistry/molpro.md
+++ b/docs.it4i/salomon/software/chemistry/molpro.md
@@ -1,91 +1,63 @@
-Molpro
+Molpro
======
-Molpro is a complete system of ab initio programs for molecular
-electronic structure calculations.
+Molpro is a complete system of ab initio programs for molecular electronic structure calculations.
About Molpro
------------
-
-Molpro is a software package used for accurate ab-initio quantum
-chemistry calculations. More information can be found at the [official
-webpage](http://www.molpro.net/).
+Molpro is a software package used for accurate ab-initio quantum chemistry calculations. More information can be found at the [official webpage](http://www.molpro.net/).
License
-------
+Molpro software package is available only to users that have a valid license. Please contact support to enable access to Molpro if you have a valid license appropriate for running on our cluster (eg. academic research group licence, parallel execution).
-Molpro software package is available only to users that have a valid
-license. Please contact support to enable access to Molpro if you have a
-valid license appropriate for running on our cluster (eg. >academic
-research group licence, parallel execution).
-
-To run Molpro, you need to have a valid license token present in
-" $HOME/.molpro/token". You can
-download the token from [Molpro
-website](https://www.molpro.net/licensee/?portal=licensee).
+To run Molpro, you need to have a valid license token present in " $HOME/.molpro/token". You can download the token from [Molpro website](https://www.molpro.net/licensee/?portal=licensee).
Installed version
-----------------
+Currently on Anselm is installed version 2010.1, patch level 45, parallel version compiled with Intel compilers and Intel MPI.
-Currently on Salomon is installed version 2010.1, patch level 57,
-parallel version compiled with Intel compilers and Intel MPI.
-
-Compilation parameters are default :
-
- |Parameter|Value|
- ------------------------------------------- |---|---|-------------------
- |max number of atoms|200|
- |max number of valence orbitals|300|
- |max number of basis functions|4095|
- |max number of states per symmmetry|20|
- |max number of state symmetries|16|
- |max number of records|200|
- |max number of primitives|maxbfn x [2]|
+Compilation parameters are default:
-Â
+|Parameter|Value|
+|---|---|
+|max number of atoms|200|
+|max number of valence orbitals|300|
+|max number of basis functions|4095|
+|max number of states per symmmetry|20|
+|max number of state symmetries|16|
+|max number of records|200|
+|max number of primitives|maxbfn x [2]|
Running
--------
+------
+Molpro is compiled for parallel execution using MPI and OpenMP. By default, Molpro reads the number of allocated nodes from PBS and launches a data server on one node. On the remaining allocated nodes, compute processes are launched, one process per node, each with 16 threads. You can modify this behavior by using -n, -t and helper-server options. Please refer to the [Molpro documentation](http://www.molpro.net/info/2010.1/doc/manual/node9.html) for more details.
+
+>The OpenMP parallelization in Molpro is limited and has been observed to produce limited scaling. We therefore recommend to use MPI parallelization only. This can be achieved by passing option mpiprocs=16:ompthreads=1 to PBS.
-Molpro is compiled for parallel execution using MPI and OpenMP. By
-default, Molpro reads the number of allocated nodes from PBS and
-launches a data server on one node. On the remaining allocated nodes,
-compute processes are launched, one process per node, each with 16
-threads. You can modify this behavior by using -n, -t and helper-server
-options. Please refer to the [Molpro
-documentation](http://www.molpro.net/info/2010.1/doc/manual/node9.html)
-for more details.Â
-
-The OpenMP parallelization in Molpro is limited and has been observed to
-produce limited scaling. We therefore recommend to use MPI
-parallelization only. This can be achieved by passing option
-mpiprocs=24:ompthreads=1 to PBS.
-
-You are advised to use the -d option to point to a directory in [SCRATCH
-filesystem](../../storage.html). Molpro can produce a
-large amount of temporary data during its run, and it is important that
-these are placed in the fast scratch filesystem.
+You are advised to use the -d option to point to a directory in [SCRATCH filesystem](../../storage.md). Molpro can produce a large amount of temporary data during its run, and it is important that these are placed in the fast scratch filesystem.
### Example jobscript
+```bash
#PBS -A IT4I-0-0
#PBS -q qprod
- #PBS -l select=1:ncpus=24:mpiprocs=24:ompthreads=1
+ #PBS -l select=1:ncpus=16:mpiprocs=16:ompthreads=1
cd $PBS_O_WORKDIR
# load Molpro module
- module add Molpro/2010.1-patch-57-intel2015b
+ module add molpro
# create a directory in the SCRATCH filesystem
- mkdir -p /scratch/work/user/$USER/$PBS_JOBID
+ mkdir -p /scratch/$USER/$PBS_JOBID
# copy an example input
- cp /apps/all/Molpro/2010.1-patch57/molprop_2010_1_Linux_x86_64_i8/examples/caffeine_opt_diis.com .
+ cp /apps/chem/molpro/2010.1/molprop_2010_1_Linux_x86_64_i8/examples/caffeine_opt_diis.com .
# run Molpro with default options
- molpro -d /scratch/work/user/$USER/$PBS_JOBID caffeine_opt_diis.com
+ molpro -d /scratch/$USER/$PBS_JOBID caffeine_opt_diis.com
# delete scratch directory
- rm -rf /scratch/work/user/$USER/$PBS_JOBIDÂ
-
+ rm -rf /scratch/$USER/$PBS_JOBID
+```
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/chemistry/nwchem.md b/docs.it4i/salomon/software/chemistry/nwchem.md
index b37cb96e6..fd589860d 100644
--- a/docs.it4i/salomon/software/chemistry/nwchem.md
+++ b/docs.it4i/salomon/software/chemistry/nwchem.md
@@ -1,47 +1,35 @@
-NWChem
+NWChem
======
-High-Performance Computational Chemistry
+**High-Performance Computational Chemistry**
Introduction
-------------------------
-
-NWChem aims to provide its users with computational chemistry
-tools that are scalable both in their ability to treat large scientific
-computational chemistry problems efficiently, and in their use of
-available parallel computing resources from high-performance parallel
-supercomputers to conventional workstation clusters.
+NWChem aims to provide its users with computational chemistry tools that are scalable both in their ability to treat large scientific computational chemistry problems efficiently, and in their use of available parallel computing resources from high-performance parallel supercomputers to conventional workstation clusters.
[Homepage](http://www.nwchem-sw.org/index.php/Main_Page)
Installed versions
------------------
-The following versions are currently installed :Â
-
-- >NWChem/6.3.revision2-2013-10-17-Python-2.7.8, current release.
- Compiled with Intel compilers, MKL and Intel MPI
-
- Â
+The following versions are currently installed:
-- >NWChem/6.5.revision26243-intel-2015b-2014-09-10-Python-2.7.8
+- NWChem/6.3.revision2-2013-10-17-Python-2.7.8, current release. Compiled with Intel compilers, MKL and Intel MPI
+- NWChem/6.5.revision26243-intel-2015b-2014-09-10-Python-2.7.8
-For a current list of installed versions, execute :Â
+For a current list of installed versions, execute:
+```bash
module avail NWChem
+```
-The recommend to use version 6.5. Version 6.3 fails on Salomon nodes
-with accelerator, because it attempts to communicate over scif0
-interface. In 6.5 this is avoided by
-setting ARMCI_OPENIB_DEVICE=mlx4_0, this setting is included in the
-module.
+The recommend to use version 6.5. Version 6.3 fails on Salomon nodes with accelerator, because it attempts to communicate over scif0 interface. In 6.5 this is avoided by setting ARMCI_OPENIB_DEVICE=mlx4_0, this setting is included in the module.
Running
-------
+ NWChem is compiled for parallel MPI execution. Normal procedure for MPI jobs applies. Sample jobscript :
-NWChem is compiled for parallel MPI execution. Normal procedure for MPI
-jobs applies. Sample jobscript :
-
+```bash
#PBS -A IT4I-0-0
#PBS -q qprod
#PBS -l select=1:ncpus=24:mpiprocs=24
@@ -49,18 +37,11 @@ jobs applies. Sample jobscript :
cd $PBS_O_WORKDIR
module add NWChem/6.5.revision26243-intel-2015b-2014-09-10-Python-2.7.8
mpirun nwchem h2o.nw
+```
Options
--------------------
+Please refer to [the documentation](http://www.nwchem-sw.org/index.php/Release62:Top-level)Â and in the input file set the following directives :
-Please refer to [the
-documentation](http://www.nwchem-sw.org/index.php/Release62:Top-level)Â and
-in the input file set the following directives :
-
-- >MEMORY : controls the amount of memory NWChem will use
-- >SCRATCH_DIR : set this to a directory in [SCRATCH
- filesystem](../../storage.html)Â (or run the
- calculation completely in a scratch directory). For certain
- calculations, it might be advisable to reduce I/O by forcing
- "direct" mode, eg. "scf direct"
-
+- MEMORY : controls the amount of memory NWChem will use
+- SCRATCH_DIR : set this to a directory in [SCRATCH filesystem](../../storage.md) (or run the calculation completely in a scratch directory). For certain calculations, it might be advisable to reduce I/O by forcing "direct" mode, eg. "scf direct"
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/chemistry/phono3py.md b/docs.it4i/salomon/software/chemistry/phono3py.md
index 14148e5a9..61cf61648 100644
--- a/docs.it4i/salomon/software/chemistry/phono3py.md
+++ b/docs.it4i/salomon/software/chemistry/phono3py.md
@@ -1,35 +1,24 @@
-Phono3py
+Phono3py
========
-
-
-Â Introduction
+Introduction
-------------
+This GPL software calculates phonon-phonon interactions via the third order force constants. It allows to obtain lattice thermal conductivity, phonon lifetime/linewidth, imaginary part of self energy at the lowest order, joint density of states (JDOS) and weighted-JDOS. For details see Phys. Rev. B 91, 094306 (2015) and http://atztogo.github.io/phono3py/index.html
-This GPL software calculates phonon-phonon interactions via the third
-order force constants. It allows to obtain lattice thermal conductivity,
-phonon lifetime/linewidth, imaginary part of self energy at the lowest
-order, joint density of states (JDOS) and weighted-JDOS. For details see
-Phys. Rev. B 91, 094306 (2015) and
-http://atztogo.github.io/phono3py/index.html
-
-Load the phono3py/0.9.14-ictce-7.3.5-Python-2.7.9 module
+>Load the phono3py/0.9.14-ictce-7.3.5-Python-2.7.9 module
-`
+```bash
$ module load phono3py/0.9.14-ictce-7.3.5-Python-2.7.9
-`
+```
Example of calculating thermal conductivity of Si using VASP code.
------------------------------------------------------------------
### Calculating force constants
-One needs to calculate second order and third order force constants
-using the diamond structure of silicon stored in
-[POSCAR](phono3py-input/poscar-si)Â (the same form as in
-VASP) using single displacement calculations within supercell.
+One needs to calculate second order and third order force constants using the diamond structure of silicon stored in [POSCAR](phono3py-input/poscar-si)Â (the same form as in VASP) using single displacement calculations within supercell.
-`
+```bash
$ cat POSCAR
 Si
  1.0
@@ -47,19 +36,18 @@ Direct
  0.1250000000000000 0.6250000000000000 0.6250000000000000
  0.6250000000000000 0.1250000000000000 0.6250000000000000
  0.6250000000000000 0.6250000000000000 0.1250000000000000
-`
+```
### Generating displacement using 2x2x2 supercell for both second and third order force constants
-`
+```bash
$ phono3py -d --dim="2 2 2" -c POSCAR
-`
+```
- 111 displacements is created stored in
-disp_fc3.yaml, and the structure input files with this
-displacements are POSCAR-00XXX, where the XXX=111.
+ 111 displacements is created stored in
+disp_fc3.yaml, and the structure input files with this displacements are POSCAR-00XXX, where the XXX=111.
-`
+```bash
disp_fc3.yaml POSCAR-00008 POSCAR-00017 POSCAR-00026 POSCAR-00035 POSCAR-00044 POSCAR-00053 POSCAR-00062 POSCAR-00071 POSCAR-00080 POSCAR-00089 POSCAR-00098 POSCAR-00107
POSCARÂ Â Â Â Â Â Â Â POSCAR-00009Â POSCAR-00018Â POSCAR-00027Â POSCAR-00036Â POSCAR-00045Â POSCAR-00054Â POSCAR-00063Â POSCAR-00072Â POSCAR-00081Â POSCAR-00090Â POSCAR-00099Â POSCAR-00108
POSCAR-00001Â Â POSCAR-00010Â POSCAR-00019Â POSCAR-00028Â POSCAR-00037Â POSCAR-00046Â POSCAR-00055Â POSCAR-00064Â POSCAR-00073Â POSCAR-00082Â POSCAR-00091Â POSCAR-00100Â POSCAR-00109
@@ -69,21 +57,11 @@ POSCAR-00004Â Â POSCAR-00013Â POSCAR-00022Â POSCAR-00031Â POSCAR-00040Â POS
POSCAR-00005Â Â POSCAR-00014Â POSCAR-00023Â POSCAR-00032Â POSCAR-00041Â POSCAR-00050Â POSCAR-00059Â POSCAR-00068Â POSCAR-00077Â POSCAR-00086Â POSCAR-00095Â POSCAR-00104
POSCAR-00006Â Â POSCAR-00015Â POSCAR-00024Â POSCAR-00033Â POSCAR-00042Â POSCAR-00051Â POSCAR-00060Â POSCAR-00069Â POSCAR-00078Â POSCAR-00087Â POSCAR-00096Â POSCAR-00105
POSCAR-00007Â Â POSCAR-00016Â POSCAR-00025Â POSCAR-00034Â POSCAR-00043Â POSCAR-00052Â POSCAR-00061Â POSCAR-00070Â POSCAR-00079Â POSCAR-00088Â POSCAR-00097Â POSCAR-00106
-`
-
- For each displacement the forces needs to be
-calculated, i.e. in form of the output file of VASP (vasprun.xml). For a
-single VASP calculations one needs
-[KPOINTS](phono3py-input/KPOINTS),
-[POTCAR](phono3py-input/POTCAR),
-[INCAR](phono3py-input/INCAR) in your case directory
-(where you have POSCARS) and those 111 displacements calculations can be
-generated by [prepare.sh](phono3py-input/prepare.sh)
-script. Then each of the single 111 calculations is submitted
-[run.sh](phono3py-input/run.sh) by
-[submit.sh](phono3py-input/submit.sh).
-
-`
+```
+
+For each displacement the forces needs to be calculated, i.e. in form of the output file of VASP (vasprun.xml). For a single VASP calculations one needs [KPOINTS](phono3py-input/KPOINTS), [POTCAR](phono3py-input/POTCAR), [INCAR](phono3py-input/INCAR) in your case directory (where you have POSCARS) and those 111 displacements calculations can be generated by [prepare.sh](phono3py-input/prepare.sh) script. Then each of the single 111 calculations is submitted [run.sh](phono3py-input/run.sh) by [submit.sh](phono3py-input/submit.sh).
+
+```bash
$./prepare.sh
$ls
disp-00001Â disp-00009Â disp-00017Â disp-00025Â disp-00033Â disp-00041Â disp-00049Â disp-00057Â disp-00065Â disp-00073Â disp-00081Â disp-00089Â disp-00097Â disp-00105Â Â Â Â INCAR
@@ -94,50 +72,41 @@ disp-00005Â disp-00013Â disp-00021Â disp-00029Â disp-00037Â disp-00045Â di
disp-00006Â disp-00014Â disp-00022Â disp-00030Â disp-00038Â disp-00046Â disp-00054Â disp-00062Â disp-00070Â disp-00078Â disp-00086Â disp-00094Â disp-00102Â disp-00110Â Â Â Â run.sh
disp-00007Â disp-00015Â disp-00023Â disp-00031Â disp-00039Â disp-00047Â disp-00055Â disp-00063Â disp-00071Â disp-00079Â disp-00087Â disp-00095Â disp-00103Â disp-00111Â Â Â Â submit.sh
disp-00008Â disp-00016Â disp-00024Â disp-00032Â disp-00040Â disp-00048Â disp-00056Â disp-00064Â disp-00072Â disp-00080Â disp-00088Â disp-00096Â disp-00104Â disp_fc3.yaml
-`
+```
- Taylor your run.sh script to fit into your project and
-other needs and submit all 111 calculations using submit.sh
-script
+Taylor your run.sh script to fit into your project and other needs and submit all 111 calculations using submit.sh script
-`
+```bash
$ ./submit.sh
-`
+```
- Collecting results and post-processing with phono3py
+Collecting results and post-processing with phono3py
---------------------------------------------------------------------------
+Once all jobs are finished and vasprun.xml is created in each disp-XXXXX directory the collection is done by
- Once all jobs are finished and vasprun.xml is created in
-each disp-XXXXX directory the collection is done by
-
-`
+```bash
$ phono3py --cf3 disp-{00001..00111}/vasprun.xml
-`
+```
- and
-`disp_fc2.yaml, FORCES_FC2`, `FORCES_FC3`{.docutils
-.literal} and disp_fc3.yaml should appear and put into the hdf
-format by
+and `disp_fc2.yaml, FORCES_FC2`, `FORCES_FC3` and disp_fc3.yaml should appear and put into the hdf format by
-`
+```bash
$ phono3py --dim="2 2 2" -c POSCAR
-`
+```
-resulting in `fc2.hdf5` and `fc3.hdf5`{.docutils
-.literal}
+resulting in `fc2.hdf5` and `fc3.hdf5`
### Thermal conductivity
- The phonon lifetime calculations takes some time,
-however is independent on grid points, so could be splitted:
+The phonon lifetime calculations takes some time, however is independent on grid points, so could be splitted:
-`
+```bash
$ phono3py --fc3 --fc2 --dim="2 2 2" --mesh="9 9 9" --sigma 0.1 --wgp
-`
+```
### Inspecting ir_grid_points.yaml
-`
+```bash
$ grep grid_point ir_grid_points.yaml
num_reduced_ir_grid_points: 35
ir_grid_points:Â # [address, weight]
@@ -176,27 +145,22 @@ ir_grid_points:Â # [address, weight]
- grid_point: 274
- grid_point: 283
- grid_point: 364
-`
+```
-one finds which grid points needed to be calculated, for instance using
-following
+one finds which grid points needed to be calculated, for instance using following
-`
+```bash
$ phono3py --fc3 --fc2 --dim="2 2 2" --mesh="9 9 9" -c POSCARÂ --sigma 0.1 --br --write-gamma --gp="0 1 2
-`
+```
- one calculates grid points 0, 1, 2. To automize one can
-use for instance scripts to submit 5 points in series, see
-[gofree-cond1.sh](phono3py-input/gofree-cond1.sh)
+one calculates grid points 0, 1, 2. To automize one can use for instance scripts to submit 5 points in series, see [gofree-cond1.sh](phono3py-input/gofree-cond1.sh)
-`
+```bash
$ qsub gofree-cond1.sh
-`
+```
- Finally the thermal conductivity result is produced by
-grouping single conductivity per grid calculations usingÂ
+Finally the thermal conductivity result is produced by grouping single conductivity per grid calculations using
-`
+```bash
$ phono3py --fc3 --fc2 --dim="2 2 2" --mesh="9 9 9" --br --read_gamma
-`
-
+```
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/compilers.md b/docs.it4i/salomon/software/compilers.md
index 125d99238..835c86e85 100644
--- a/docs.it4i/salomon/software/compilers.md
+++ b/docs.it4i/salomon/software/compilers.md
@@ -1,12 +1,9 @@
-Compilers
+Compilers
=========
Available compilers, including GNU, INTEL and UPC compilers
-
-
-There are several compilers for different programming languages
-available on the cluster:
+There are several compilers for different programming languages available on the cluster:
- C/C++
- Fortran 77/90/95/HPF
@@ -28,15 +25,14 @@ Commercial licenses:
Intel Compilers
---------------
-For information about the usage of Intel Compilers and other Intel
-products, please read the [Intel Parallel
-studio](intel-suite.html) page.
+For information about the usage of Intel Compilers and other Intel products, please read the [Intel Parallel studio](intel-suite.md) page.
PGI Compilers
-------------
The Portland Group Cluster Development Kit (PGI CDK) is available.
+```bash
$ module load PGI
$ pgcc -v
$ pgc++ -v
@@ -44,50 +40,48 @@ The Portland Group Cluster Development Kit (PGI CDK) is available.
$ pgf90 -v
$ pgf95 -v
$ pghpf -v
+```
The PGI CDK also incudes tools for debugging and profiling.
PGDBG OpenMP/MPI debugger and PGPROF OpenMP/MPI profiler are available
+```bash
$ module load PGI
$ module load Java
$ pgdbg &
$ pgprof &
+```
-For more information, see the [PGI
-page](http://www.pgroup.com/products/pgicdk.htm).
+For more information, see the [PGI page](http://www.pgroup.com/products/pgicdk.htm).
GNU
---
+For compatibility reasons there are still available the original (old 4.4.7-11) versions of GNU compilers as part of the OS. These are accessible in the search path by default.
-For compatibility reasons there are still available the original (old
-4.4.7-11) versions of GNU compilers as part of the OS. These are
-accessible in the search path by default.
-
-It is strongly recommended to use the up to date version which comes
-with the module GCC:
+It is strongly recommended to use the up to date version which comes with the module GCC:
+```bash
$ module load GCC
$ gcc -v
$ g++ -v
$ gfortran -v
+```
-With the module loaded two environment variables are predefined. One for
-maximum optimizations on the cluster's architecture, and the other for
-debugging purposes:
+With the module loaded two environment variables are predefined. One for maximum optimizations on the cluster's architecture, and the other for debugging purposes:
+```bash
$ echo $OPTFLAGS
-O3 -march=native
$ echo $DEBUGFLAGS
-O0 -g
+```
-For more information about the possibilities of the compilers, please
-see the man pages.
+For more information about the possibilities of the compilers, please see the man pages.
Unified Parallel C
------------------
-
UPC is supported by two compiler/runtime implementations:
- GNU - SMP/multi-threading support only
@@ -95,16 +89,18 @@ UPC is supported by two compiler/runtime implementations:
### GNU UPC Compiler
-To use the GNU UPC compiler and run the compiled binaries use the module
-gupc
+To use the GNU UPC compiler and run the compiled binaries use the module gupc
+```bash
$ module add gupc
$ gupc -v
$ g++ -v
+```
Simple program to test the compiler
- $ cat count.upc
+```bash
+ $ cat count.upc
/* hello.upc - a simple UPC example */
#include <upc.h>
@@ -117,38 +113,40 @@ Simple program to test the compiler
 upc_barrier;
 printf(" - Hello from thread %in", MYTHREAD);
 return 0;
- }
+ }
+```
To compile the example use
+```bash
$ gupc -o count.upc.x count.upc
+```
To run the example with 5 threads issue
+```bash
$ ./count.upc.x -fupc-threads-5
+```
For more informations see the man pages.
### Berkley UPC Compiler
-To use the Berkley UPC compiler and runtime environment to run the
-binaries use the module bupc
+To use the Berkley UPC compiler and runtime environment to run the binaries use the module bupc
+```bash
$ module add BerkeleyUPC/2.16.2-gompi-2015b
$ upcc -version
+```
-As default UPC network the "smp" is used. This is very quick and easy
-way for testing/debugging, but limited to one node only.
+As default UPC network the "smp" is used. This is very quick and easy way for testing/debugging, but limited to one node only.
-For production runs, it is recommended to use the native Infiband
-implementation of UPC network "ibv". For testing/debugging using
-multiple nodes, the "mpi" UPC network is recommended. Please note, that
-the selection of the network is done at the compile time** and not at
-runtime (as expected)!
+For production runs, it is recommended to use the native Infiband implementation of UPC network "ibv". For testing/debugging using multiple nodes, the "mpi" UPC network is recommended. Please note, that the selection of the network is done at the compile time and not at runtime (as expected)!
Example UPC code:
- $ cat hello.upc
+```bash
+ $ cat hello.upc
/* hello.upc - a simple UPC example */
#include <upc.h>
@@ -161,34 +159,36 @@ Example UPC code:
 upc_barrier;
 printf(" - Hello from thread %in", MYTHREAD);
 return 0;
- }
+ }
+```
To compile the example with the "ibv" UPC network use
- $ upcc -network=ibv -o hello.upc.x hello.upc
+```bash
+ $ upcc -network=ibv -o hello.upc.x hello.upc
+```
To run the example with 5 threads issue
+```bash
$ upcrun -n 5 ./hello.upc.x
+```
-To run the example on two compute nodes using all 48 cores, with 48
-threads, issue
+To run the example on two compute nodes using all 48 cores, with 48 threads, issue
- $ qsub -I -q qprod -A PROJECT_ID -l select=2:ncpus=24
+```bash
+ $ qsub -I -q qprod -A PROJECT_ID -l select=2:ncpus=24
$ module add bupc
$ upcrun -n 48 ./hello.upc.x
+```
-Â For more informations see the man pages.
+For more informations see the man pages.
Java
----
-For information how to use Java (runtime and/or compiler), please read
-the [Java page](java.html).
+For information how to use Java (runtime and/or compiler), please read the [Java page](java.md).
nVidia CUDA
-For information how to work with nVidia CUDA, please read the [nVidia
-CUDA
-page](../../anselm-cluster-documentation/software/nvidia-cuda.html).
-
+For information how to work with nVidia CUDA, please read the [nVidia CUDA page](../../anselm-cluster-documentation/software/nvidia-cuda.md).
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/comsol/comsol-multiphysics.md b/docs.it4i/salomon/software/comsol/comsol-multiphysics.md
index 5948a933f..c4b69d9f1 100644
--- a/docs.it4i/salomon/software/comsol/comsol-multiphysics.md
+++ b/docs.it4i/salomon/software/comsol/comsol-multiphysics.md
@@ -1,103 +1,51 @@
-COMSOL Multiphysics®
+COMSOL Multiphysics®
====================
-
-
Introduction
-
-------------------------
+[COMSOL](http://www.comsol.com) is a powerful environment for modelling and solving various engineering and scientific problems based on partial differential equations. COMSOL is designed to solve coupled or multiphysics phenomena. For many standard engineering problems COMSOL provides add-on products such as electrical, mechanical, fluid flow, and chemical applications.
-[COMSOL](http://www.comsol.com)
-is a powerful environment for modelling and solving various engineering
-and scientific problems based on partial differential equations. COMSOL
-is designed to solve coupled or multiphysics phenomena. For many
-standard engineering problems COMSOL provides add-on products such as
-electrical, mechanical, fluid flow, and chemical
-applications.
-
-- >[Structural Mechanics
- Module](http://www.comsol.com/structural-mechanics-module),
-
-
-- >[Heat Transfer
- Module](http://www.comsol.com/heat-transfer-module),
-
-
-- >[CFD
- Module](http://www.comsol.com/cfd-module),
-
-
-- >[Acoustics
- Module](http://www.comsol.com/acoustics-module),
-
+- [Structural Mechanics Module](http://www.comsol.com/structural-mechanics-module),
+- [Heat Transfer Module](http://www.comsol.com/heat-transfer-module),
+- [CFD Module](http://www.comsol.com/cfd-module),
+- [Acoustics Module](http://www.comsol.com/acoustics-module),
+- and [many others](http://www.comsol.com/products)
-- >and [many
- others](http://www.comsol.com/products)
-
-COMSOL also allows an
-interface support for
-equation-based modelling of
-partial differential
-equations.
+COMSOL also allows an interface support for equation-based modelling of partial differential equations.
Execution
-
----------------------
-On the clusters COMSOL is available in the latest stable
-version. There are two variants of the release:
-
-- >**Non commercial** or so
- called >**EDU
- variant**>, which can be used for research
- and educational purposes.
-
-- >**Commercial** or so called
- >**COM variant**,
- which can used also for commercial activities.
- >**COM variant**
- has only subset of features compared to the
- >**EDU
- variant**> available.
-
- More about
- licensing will be posted here
- soon.
-
+On the clusters COMSOL is available in the latest stable version. There are two variants of the release:
+
+- **Non commercial** or so called >**EDU variant**>, which can be used for research and educational purposes.
+
+- **Commercial** or so called **COM variant**, which can used also for commercial activities. **COM variant** has only subset of features compared to the **EDU variant** available. More about licensing will be posted here soon.
To load the of COMSOL load the module
-`
+```bash
$ module load COMSOL/51-EDU
-`
+```
-By default the **EDU
-variant**> will be loaded. If user needs other
-version or variant, load the particular version. To obtain the list of
-available versions use
+By default the **EDU variant** will be loaded. If user needs other version or variant, load the particular version. To obtain the list of available versions use
-`
+```bash
$ module avail COMSOL
-`
+```
-If user needs to prepare COMSOL jobs in the interactive mode
-it is recommend to use COMSOL on the compute nodes via PBS Pro
-scheduler. In order run the COMSOL Desktop GUI on Windows is recommended
-to use the [Virtual Network Computing
-(VNC)](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-system/x-window-and-vnc.html).
+If user needs to prepare COMSOL jobs in the interactive mode it is recommend to use COMSOL on the compute nodes via PBS Pro scheduler. In order run the COMSOL Desktop GUI on Windows is recommended to use the [Virtual Network Computing (VNC)](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-and-vnc.md).
-`
+```bash
$ xhost +
$ qsub -I -X -A PROJECT_ID -q qprod -l select=1:ppn=24
$ module load COMSOL
$ comsol
-`
+```
-To run COMSOL in batch mode, without the COMSOL Desktop GUI
-environment, user can utilized the default (comsol.pbs) job script and
-execute it via the qsub command.
+To run COMSOL in batch mode, without the COMSOL Desktop GUI environment, user can utilized the default (comsol.pbs) job script and execute it via the qsub command.
-`
+```bash
#!/bin/bash
#PBS -l select=3:ppn=24
#PBS -q qprod
@@ -120,54 +68,29 @@ module load COMSOL
ntask=$(wc -l $PBS_NODEFILE)
comsol -nn ${ntask} batch -configuration /tmp –mpiarg –rmk –mpiarg pbs -tmpdir /scratch/$USER/ -inputfile name_input_f.mph -outputfile name_output_f.mph -batchlog name_log_f.log
-`
+```
-Working directory has to be created before sending the
-(comsol.pbs) job script into the queue. Input file (name_input_f.mph)
-has to be in working directory or full path to input file has to be
-specified. The appropriate path to the temp directory of the job has to
-be set by command option (-tmpdir).
+Working directory has to be created before sending the (comsol.pbs) job script into the queue. Input file (name_input_f.mph) has to be in working directory or full path to input file has to be specified. The appropriate path to the temp directory of the job has to be set by command option (-tmpdir).
-LiveLink™* *for MATLAB®^
+LiveLink™* *for MATLAB®
-------------------------
+COMSOL is the software package for the numerical solution of the partial differential equations. LiveLink for MATLAB allows connection to the COMSOL®API (Application Programming Interface) with the benefits of the programming language and computing environment of the MATLAB.
+
+LiveLink for MATLAB is available in both **EDU** and **COM** **variant** of the COMSOL release. On the clusters 1 commercial (**COM**) license and the 5 educational (**EDU**) licenses of LiveLink for MATLAB (please see the [ISV Licenses](../isv_licenses.md)) are available. Following example shows how to start COMSOL model from MATLAB via LiveLink in the interactive mode.
-COMSOL is the software package for the numerical solution of
-the partial differential equations. LiveLink for MATLAB allows
-connection to the
-COMSOL>><span><span><span><span>**®**</span>^
-API (Application Programming Interface) with the benefits of the
-programming language and computing environment of the MATLAB.
-
-LiveLink for MATLAB is available in both
-**EDU** and
-**COM**
-**variant** of the
-COMSOL release. On the clusters 1 commercial
-(>**COM**) license
-and the 5 educational
-(>**EDU**) licenses
-of LiveLink for MATLAB (please see the [ISV
-Licenses](../isv_licenses.html)) are available.
-Following example shows how to start COMSOL model from MATLAB via
-LiveLink in the interactive mode.
-
-`
+```bash
$ xhost +
$ qsub -I -X -A PROJECT_ID -q qexp -l select=1:ppn=24
$ module load MATLAB
$ module load COMSOL
$ comsol server MATLAB
-`
+```
-At the first time to launch the LiveLink for MATLAB
-(client-MATLAB/server-COMSOL connection) the login and password is
-requested and this information is not requested again.
+At the first time to launch the LiveLink for MATLAB (client-MATLAB/server-COMSOL connection) the login and password is requested and this information is not requested again.
-To run LiveLink for MATLAB in batch mode with
-(comsol_matlab.pbs) job script you can utilize/modify the following
-script and execute it via the qsub command.
+To run LiveLink for MATLAB in batch mode with (comsol_matlab.pbs) job script you can utilize/modify the following script and execute it via the qsub command.
-`
+```bash
#!/bin/bash
#PBS -l select=3:ppn=24
#PBS -q qprod
@@ -192,13 +115,6 @@ ntask=$(wc -l $PBS_NODEFILE)
comsol -nn ${ntask} server -configuration /tmp -mpiarg -rmk -mpiarg pbs -tmpdir /scratch/work/user/$USER/work &
cd /apps/cae/COMSOL/51/mli
matlab -nodesktop -nosplash -r "mphstart; addpath /scratch/work/user/$USER/work; test_job"
-`
-
-This example shows how to run Livelink for MATLAB with following
-configuration: 3 nodes and 16 cores per node. Working directory has to
-be created before submitting (comsol_matlab.pbs) job script into the
-queue. Input file (test_job.m) has to be in working directory or full
-path to input file has to be specified. The Matlab command option (-r
-”mphstart”) created a connection with a COMSOL server using the default
-port number.
+```
+This example shows how to run Livelink for MATLAB with following configuration: 3 nodes and 16 cores per node. Working directory has to be created before submitting (comsol_matlab.pbs) job script into the queue. Input file (test_job.m) has to be in working directory or full path to input file has to be specified. The Matlab command option (-r ”mphstart”) created a connection with a COMSOL server using the default port number.
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/comsol/licensing-and-available-versions.md b/docs.it4i/salomon/software/comsol/licensing-and-available-versions.md
index ffafbd5d1..e3a4950a7 100644
--- a/docs.it4i/salomon/software/comsol/licensing-and-available-versions.md
+++ b/docs.it4i/salomon/software/comsol/licensing-and-available-versions.md
@@ -1,40 +1,23 @@
-Licensing and Available Versions
+Licensing and Available Versions
================================
Comsol licence can be used by:
------------------------------
-- all persons in the carrying out of the CE IT4Innovations Project (In
- addition to the primary licensee, which is VSB - Technical
- University of Ostrava, users are CE IT4Innovations third parties -
- CE IT4Innovations project partners, particularly the University of
- Ostrava, the Brno University of Technology - Faculty of Informatics,
- the Silesian University in Opava, Institute of Geonics AS CR.)
-- all
- persons who have a valid
- license
-- students
- of the Technical University
+- all persons in the carrying out of the CE IT4Innovations Project (In addition to the primary licensee, which is VSB - Technical University of Ostrava, users are CE IT4Innovations third parties - CE IT4Innovations project partners, particularly the University of Ostrava, the Brno University of Technology - Faculty of Informatics, the Silesian University in Opava, Institute of Geonics AS CR.)
+- all persons who have a valid license
+- students of the Technical University
Comsol EDU Network Licence
--------------------------
-The licence intended to be used for science and research, publications,
-students’ projects, teaching (academic licence).
+The licence intended to be used for science and research, publications, students’ projects, teaching (academic licence).
Comsol COM Network Licence
--------------------------
-The licence intended to be used for science and research, publications,
-students’ projects, commercial research with no commercial use
-restrictions. > E
-nables the solution
-of at least one job
-by one user in one
-program start.
+The licence intended to be used for science and research, publications, students’ projects, commercial research with no commercial use restrictions. Enables the solution of at least one job by one user in one program start.
Available Versions
------------------
-
-- ver. 51
-
+- ver. 51
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/debuggers.md b/docs.it4i/salomon/software/debuggers.md
deleted file mode 100644
index 2288b8716..000000000
--- a/docs.it4i/salomon/software/debuggers.md
+++ /dev/null
@@ -1,90 +0,0 @@
-Debuggers and profilers summary
-===============================
-
-
-
-Introduction
-------------
-
-We provide state of the art programms and tools to develop, profile and
-debug HPC codes at IT4Innovations.
-On these pages, we provide an overview of the profiling and debugging
-tools available on Anslem at IT4I.
-
-Intel debugger
---------------
-
-Intel debugger is no longer available since Parallel Studio version 2015
-
-The intel debugger version 13.0 is available, via module intel. The
-debugger works for applications compiled with C and C++ compiler and the
-ifort fortran 77/90/95 compiler. The debugger provides java GUI
-environment. Use [X
-display](../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-system/x-window-and-vnc.html)
-for running the GUI.
-
- $ module load intel
- $ idb
-
-Read more at the [Intel
-Debugger](intel-suite/intel-debugger.html) page.
-
-Allinea Forge (DDT/MAP)
------------------------
-
-Allinea DDT, is a commercial debugger primarily for debugging parallel
-MPI or OpenMP programs. It also has a support for GPU (CUDA) and Intel
-Xeon Phi accelerators. DDT provides all the standard debugging features
-(stack trace, breakpoints, watches, view variables, threads etc.) for
-every thread running as part of your program, or for every process -
-even if these processes are distributed across a cluster using an MPI
-implementation.
-
- $ module load Forge
- $ forge
-
-Read more at the [Allinea
-DDT](debuggers/allinea-ddt.html) page.
-
-Allinea Performance Reports
----------------------------
-
-Allinea Performance Reports characterize the performance of HPC
-application runs. After executing your application through the tool, a
-synthetic HTML report is generated automatically, containing information
-about several metrics along with clear behavior statements and hints to
-help you improve the efficiency of your runs. Our license is limited to
-64 MPI processes.
-
- $ module load PerformanceReports/6.0
- $ perf-report mpirun -n 64 ./my_application argument01 argument02
-
-Read more at the [Allinea Performance
-Reports](debuggers/allinea-performance-reports.html)
-page.
-
-RougeWave Totalview
--------------------
-
-TotalView is a source- and machine-level debugger for multi-process,
-multi-threaded programs. Its wide range of tools provides ways to
-analyze, organize, and test programs, making it easy to isolate and
-identify problems in individual threads and processes in programs of
-great complexity.
-
- $ module load TotalView/8.15.4-6-linux-x86-64
- $ totalview
-
-Read more at the [Totalview](debuggers/total-view.html)
-page.
-
-Vampir trace analyzer
----------------------
-
-Vampir is a GUI trace analyzer for traces in OTF format.
-
- $ module load Vampir/8.5.0
- $ vampir
-
-Read more at the [Vampir](debuggers/vampir.html) page.
-
diff --git a/docs.it4i/salomon/software/debuggers/Introduction.md b/docs.it4i/salomon/software/debuggers/Introduction.md
new file mode 100644
index 000000000..c150402ae
--- /dev/null
+++ b/docs.it4i/salomon/software/debuggers/Introduction.md
@@ -0,0 +1,72 @@
+Debuggers and profilers summary
+===============================
+
+Introduction
+------------
+
+We provide state of the art programms and tools to develop, profile and debug HPC codes at IT4Innovations. On these pages, we provide an overview of the profiling and debugging tools available on Anslem at IT4I.
+
+Intel debugger
+--------------
+
+Intel debugger is no longer available since Parallel Studio version 2015
+
+The intel debugger version 13.0 is available, via module intel. The debugger works for applications compiled with C and C++ compiler and the ifort fortran 77/90/95 compiler. The debugger provides java GUI environment. Use [X display](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-and-vnc.md)
+for running the GUI.
+
+```bash
+ $ module load intel
+ $ idb
+```
+
+Read more at the [Intel Debugger](../intel-suite/intel-debugger.md) page.
+
+Allinea Forge (DDT/MAP)
+-----------------------
+Allinea DDT, is a commercial debugger primarily for debugging parallel
+MPI or OpenMP programs. It also has a support for GPU (CUDA) and Intel
+Xeon Phi accelerators. DDT provides all the standard debugging features
+(stack trace, breakpoints, watches, view variables, threads etc.) for
+every thread running as part of your program, or for every process -
+even if these processes are distributed across a cluster using an MPI
+implementation.
+
+```bash
+ $ module load Forge
+ $ forge
+```
+
+Read more at the [Allinea DDT](allinea-ddt.md) page.
+
+Allinea Performance Reports
+---------------------------
+Allinea Performance Reports characterize the performance of HPC application runs. After executing your application through the tool, a synthetic HTML report is generated automatically, containing information about several metrics along with clear behavior statements and hints to help you improve the efficiency of your runs. Our license is limited to 64 MPI processes.
+
+```bash
+ $ module load PerformanceReports/6.0
+ $ perf-report mpirun -n 64 ./my_application argument01 argument02
+```
+
+Read more at the [Allinea Performance Reports](allinea-performance-reports.md) page.
+
+RougeWave Totalview
+-------------------
+TotalView is a source- and machine-level debugger for multi-process, multi-threaded programs. Its wide range of tools provides ways to analyze, organize, and test programs, making it easy to isolate and identify problems in individual threads and processes in programs of great complexity.
+
+```bash
+ $ module load TotalView/8.15.4-6-linux-x86-64
+ $ totalview
+```
+
+Read more at the [Totalview](total-view.md) page.
+
+Vampir trace analyzer
+---------------------
+Vampir is a GUI trace analyzer for traces in OTF format.
+
+```bash
+ $ module load Vampir/8.5.0
+ $ vampir
+```
+
+Read more at the [Vampir](vampir.md) page.
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/debuggers/aislinn.md b/docs.it4i/salomon/software/debuggers/aislinn.md
index ca9d44c11..d363c176f 100644
--- a/docs.it4i/salomon/software/debuggers/aislinn.md
+++ b/docs.it4i/salomon/software/debuggers/aislinn.md
@@ -1,28 +1,19 @@
-Aislinn
+Aislinn
=======
-- Aislinn is a dynamic verifier for MPI programs. For a fixed input it
- covers all possible runs with respect to nondeterminism introduced
- by MPI. It allows to detect bugs (for sure) that occurs very rare in
- normal runs.
-- Aislinn detects problems like invalid memory accesses, deadlocks,
- misuse of MPI, and resource leaks.
-- Aislinn is open-source software; you can use it without any
- licensing limitations.
+- Aislinn is a dynamic verifier for MPI programs. For a fixed input it covers all possible runs with respect to nondeterminism introduced by MPI. It allows to detect bugs (for sure) that occurs very rare in normal runs.
+- Aislinn detects problems like invalid memory accesses, deadlocks, misuse of MPI, and resource leaks.
+- Aislinn is open-source software; you can use it without any licensing limitations.
- Web page of the project: <http://verif.cs.vsb.cz/aislinn/>
-Note
-
-Aislinn is software developed at IT4Innovations and some parts are still
-considered experimental. If you have any questions or experienced any
-problems, please contact the author: <stanislav.bohm@vsb.cz>.
+>Note
+Aislinn is software developed at IT4Innovations and some parts are still considered experimental. If you have any questions or experienced any problems, please contact the author: <stanislav.bohm@vsb.cz>.
### Usage
-Let us have the following program that contains a bug that is not
-manifested in all runs:
+Let us have the following program that contains a bug that is not manifested in all runs:
-`
+```cpp
#include <mpi.h>
#include <stdlib.h>
@@ -53,98 +44,59 @@ int main(int argc, char **argv) {
MPI_Finalize();
return 0;
}
-`
+```
-The program does the following: process 0 receives two messages from
-anyone and processes 1 and 2 send a message to process 0. If a message
-from process 1 is received first, then the run does not expose the
-error. If a message from process 2 is received first, then invalid
-memory write occurs at line 16.
+The program does the following: process 0 receives two messages from anyone and processes 1 and 2 send a message to process 0. If a message from process 1 is received first, then the run does not expose the error. If a message from process 2 is received first, then invalid memory write occurs at line 16.
To verify this program by Aislinn, we first load Aislinn itself:
-`
+```bash
$ module load aislinn
-`
+```
-Now we compile the program by Aislinn implementation of MPI. There are
-`mpicc` for C programs and `mpicxx`{.docutils
-.literal} for C++ programs. Only MPI parts of the verified application
-has to be recompiled; non-MPI parts may remain untouched. Let us assume
-that our program is in `test.cpp`.
+Now we compile the program by Aislinn implementation of MPI. There are `mpicc` for C programs and `mpicxx` for C++ programs. Only MPI parts of the verified application has to be recompiled; non-MPI parts may remain untouched. Let us assume that our program is in `test.cpp`.
-`
+```bash
$ mpicc -g test.cpp -o test
-`
+```
-The `-g` flag is not necessary, but it puts more
-debugging information into the program, hence Aislinn may provide more
-detailed report. The command produces executable file `test`{.docutils
-.literal}.
+The `-g` flag is not necessary, but it puts more debugging information into the program, hence Aislinn may provide more detailed report. The command produces executable file `test`.
-Now we run the Aislinn itself. The argument `-p 3`
-specifies that we want to verify our program for the case of three MPI
-processes
+Now we run the Aislinn itself. The argument `-p 3` specifies that we want to verify our program for the case of three MPI processes
-`
+```bash
$ aislinn -p 3 ./test
==AN== INFO: Aislinn v0.3.0
==AN== INFO: Found error 'Invalid write'
==AN== INFO: 1 error(s) found
==AN== INFO: Report written into 'report.html'
-`
+```
-Aislinn found an error and produced HTML report. To view it, we can use
-any browser, e.g.:
+Aislinn found an error and produced HTML report. To view it, we can use any browser, e.g.:
+```bash
$ firefox report.html
+```
-At the beginning of the report there are some basic summaries of the
-verification. In the second part (depicted in the following picture),
-the error is described.
+At the beginning of the report there are some basic summaries of the verification. In the second part (depicted in the following picture), the error is described.
-It shows us:
+It shows us:
- Error occurs in process 0 in test.cpp on line 16.
- - Stdout and stderr streams are empty. (The program does not
- write anything).
- - The last part shows MPI calls for each process that occurs in the
- invalid run. The more detailed information about each call can be
- obtained by mouse cursor.
+ - Stdout and stderr streams are empty. (The program does not write anything).
+ - The last part shows MPI calls for each process that occurs in the invalid run. The more detailed information about each call can be obtained by mouse cursor.
### Limitations
-Since the verification is a non-trivial process there are some of
-limitations.
-
-- The verified process has to terminate in all runs, i.e. we cannot
- answer the halting problem.
-- The verification is a computationally and memory demanding process.
- We put an effort to make it efficient and it is an important point
- for further research. However covering all runs will be always more
- demanding than techniques that examines only a single run. The good
- practise is to start with small instances and when it is feasible,
- make them bigger. The Aislinn is good to find bugs that are hard to
- find because they occur very rarely (only in a rare scheduling).
- Such bugs often do not need big instances.
-- Aislinn expects that your program is a "standard MPI" program, i.e.
- processes communicate only through MPI, the verified program does
- not interacts with the system in some unusual ways (e.g.
- opening sockets).
-
-There are also some limitations bounded to the current version and they
-will be removed in the future:
-
-- All files containing MPI calls have to be recompiled by MPI
- implementation provided by Aislinn. The files that does not contain
- MPI calls, they do not have to recompiled. Aislinn MPI
- implementation supports many commonly used calls from MPI-2 and
- MPI-3 related to point-to-point communication, collective
- communication, and communicator management. Unfortunately, MPI-IO
- and one-side communication is not implemented yet.
-- Each MPI can use only one thread (if you use OpenMP, set
- `OMP_NUM_THREADS` to 1).
-- There are some limitations for using files, but if the program just
- reads inputs and writes results, it is ok.
+Since the verification is a non-trivial process there are some of limitations.
+
+- The verified process has to terminate in all runs, i.e. we cannot answer the halting problem.
+- The verification is a computationally and memory demanding process. We put an effort to make it efficient and it is an important point for further research. However covering all runs will be always more demanding than techniques that examines only a single run. The good practise is to start with small instances and when it is feasible, make them bigger. The Aislinn is good to find bugs that are hard to find because they occur very rarely (only in a rare scheduling). Such bugs often do not need big instances.
+- Aislinn expects that your program is a "standard MPI" program, i.e. processes communicate only through MPI, the verified program does not interacts with the system in some unusual ways (e.g. opening sockets).
+
+There are also some limitations bounded to the current version and they will be removed in the future:
+- All files containing MPI calls have to be recompiled by MPI implementation provided by Aislinn. The files that does not contain MPI calls, they do not have to recompiled. Aislinn MPI implementation supports many commonly used calls from MPI-2 and MPI-3 related to point-to-point communication, collective communication, and communicator management. Unfortunately, MPI-IO and one-side communication is not implemented yet.
+- Each MPI can use only one thread (if you use OpenMP, set OMP_NUM_THREADS to 1).
+- There are some limitations for using files, but if the program just reads inputs and writes results, it is ok.
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/debuggers/allinea-ddt.md b/docs.it4i/salomon/software/debuggers/allinea-ddt.md
index e84881fc2..e929ddf3b 100644
--- a/docs.it4i/salomon/software/debuggers/allinea-ddt.md
+++ b/docs.it4i/salomon/software/debuggers/allinea-ddt.md
@@ -1,220 +1,97 @@
-Allinea Forge (DDT,MAP)
+Allinea Forge (DDT,MAP)
=======================
-
-
Allinea Forge consist of two tools - debugger DDT and profiler MAP.
-Allinea DDT, is a commercial debugger primarily for debugging parallel
-MPI or OpenMP programs. It also has a support for GPU (CUDA) and Intel
-Xeon Phi accelerators. DDT provides all the standard debugging features
-(stack trace, breakpoints, watches, view variables, threads etc.) for
-every thread running as part of your program, or for every process -
-even if these processes are distributed across a cluster using an MPI
-implementation.
-
-Allinea MAP is a profiler for C/C++/Fortran HPC codes. It is designed
-for profiling parallel code, which uses pthreads, OpenMP or MPI.
+Allinea DDT, is a commercial debugger primarily for debugging parallel MPI or OpenMP programs. It also has a support for GPU (CUDA) and Intel Xeon Phi accelerators. DDT provides all the standard debugging features (stack trace, breakpoints, watches, view variables, threads etc.) for every thread running as part of your program, or for every process - even if these processes are distributed across a cluster using an MPI implementation.
-License and Limitations for the clusters Users
-----------------------------------------------
+Allinea MAP is a profiler for C/C++/Fortran HPC codes. It is designed for profiling parallel code, which uses pthreads, OpenMP or MPI.
-On the clusters users can debug OpenMP or MPI code that runs up to 64
-parallel processes. In case of debugging GPU or Xeon Phi accelerated
-codes the limit is 8 accelerators. These limitation means that:
+License and Limitations for Anselm Users
+----------------------------------------
+On Anselm users can debug OpenMP or MPI code that runs up to 64 parallel processes. In case of debugging GPU or Xeon Phi accelerated codes the limit is 8 accelerators. These limitation means that:
- 1 user can debug up 64 processes, or
- 32 users can debug 2 processes, etc.
In case of debugging on accelerators:
-- 1 user can debug on up to 8 accelerators, orÂ
-- 8 users can debug on single accelerator.Â
+- 1 user can debug on up to 8 accelerators, or
+- 8 users can debug on single accelerator.
-Compiling Code to run with Forge
---------------------------------
+Compiling Code to run with DDT
+------------------------------
### Modules
-Load all necessary modules to compile the code. For example:Â
+Load all necessary modules to compile the code. For example:
+```bash
$ module load intel
- $ module load impi ... or ... module load OpenMPI
+ $ module load impi ... or ... module load openmpi/X.X.X-icc
+```
Load the Allinea DDT module:
+```bash
$ module load Forge
+```
Compile the code:
-`
+```bash
$ mpicc -g -O0 -o test_debug test.c
$ mpif90 -g -O0 -o test_debug test.f
-`
+```
### Compiler flags
Before debugging, you need to compile your code with theses flags:
--g** : Generates extra debugging information usable by GDB. -g3**
-includes even more debugging information. This option is available for
+>- **g** : Generates extra debugging information usable by GDB. -g3 includes even more debugging information. This option is available for
GNU and INTEL C/C++ and Fortran compilers.
--O0** : Suppress all optimizations.**
-
-Â
+>- **O0** : Suppress all optimizations.
-Direct starting a Job with Forge
---------------------------------
+Starting a Job with DDT
+-----------------------
+Be sure to log in with an X window forwarding enabled. This could mean using the -X in the ssh:
-Be sure to log in with an [ X window
-forwarding
-enabled](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-system/x-window-and-vnc.html).
-This could mean using the -X in the ssh: Â
+```bash
+ $ ssh -X username@anselm.it4i.cz
+```
- $ ssh -X username@clustername.it4i.cz
+Other options is to access login node using VNC. Please see the detailed information on how to [use graphic user interface on Anselm](https://docs.it4i.cz/anselm-cluster-documentation/software/debuggers/resolveuid/11e53ad0d2fd4c5187537f4baeedff33)
-Other options is to access login node using VNC. Please see the detailed
-information on [how to
-use graphic user interface on the
-clusters](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-system/x-window-and-vnc.html)
-.
+From the login node an interactive session **with X windows forwarding** (-X option) can be started by following command:
-From the login node an interactive session **with X windows forwarding**
-(-X option) can be started by following command:Â
+```bash
+ $ qsub -I -X -A NONE-0-0 -q qexp -lselect=1:ncpus=16:mpiprocs=16,walltime=01:00:00
+```
- $ qsub -I -X -A NONE-0-0 -q qexp -lselect=1:ncpus=24:mpiprocs=24,walltime=01:00:00Â
-
-Then launch the debugger with the ddt command followed by the name of
-the executable to debug:
+Then launch the debugger with the ddt command followed by the name of the executable to debug:
+```bash
$ ddt test_debug
+```
-Forge now has common GUI for both DDT and MAP. In interactive mode, you
-can launch Forge using forge, ddt or map,
-the latter two will just launch forge and swith to the respective
-tab in the common GUI.
-
-A submission window that appears have
-a prefilled path to the executable to debug. You can select the number
-of MPI processors and/or OpenMP threads on which to run and press run.
-Command line arguments to a program can be entered to the
-"Arguments "
-box.
+A submission window that appears have a prefilled path to the executable to debug. You can select the number of MPI processors and/or OpenMP threads on which to run and press run. Command line arguments to a program can be entered to the "Arguments " box.
-Â Â
-To start the debugging directly without the submission window, user can
-specify the debugging and execution parameters from the command line.
-For example the number of MPI processes is set by option "-np 4".
-Skipping the dialog is done by "-start" option. To see the list of the
-"ddt" command line parameters, run "ddt --help". Â
+To start the debugging directly without the submission window, user can specify the debugging and execution parameters from the command line. For example the number of MPI processes is set by option "-np 4". Skipping the dialog is done by "-start" option. To see the list of the "ddt" command line parameters, run "ddt --help".
+```bash
ddt -start -np 4 ./hello_debug_impi
-
-All of the above text also applies for MAP, just replace ddt command
-with map.
-
-Reverse connect
----------------
-
-Forge now provides a new convenient mode of operation, called Reverse
-connect. Instead of launching a job from the GUI, the process is
-reserved - DDT/MAP is launched as a server in the job which then
-connects to a running instance of your GUI.
-
-To use Reverse connect, use a jobscript that you would normally use to
-launch your application, just prepend ddt/map --connect to your
-application:
-
- map --connect mpirun -np 24 ./mpi-test
- ddt --connect mpirun -np 24 ./mpi-test
-
-Launch Forge GUI on login node and submit the job using qsub. When the
-job starts running, Forge will ask you to accept the connection:
-
-
-
-After accepting the request, you can start remote profiling/debugging.
-
-Xeon Phi
---------
-
-Forge allows debugging and profiling of both offload and native mode
-Xeon Phi programs.
-
-### Offload mode
-
-It is recommended to set the following environment values on the offload
-host:
-
- export MYO_WATCHDOG_MONITOR=-1 # To make sure the host process isn't killed when we enter a debugging session
- export AMPLXE_COI_DEBUG_SUPPORT=true # To make sure that debugging symbols are accessible on the host and the card
- unset OFFLOAD_MAIN # To make sure allinea DDT can attach to offloaded codes
-
-Then use one of the above mentioned methods to launch Forge. (Reverse
-connect also works.)
-
-### Native mode
-
-Native mode programs can be profiled/debugged using the remote launch
-feature. First, you need to create a script that will setup the
-environment on the Phi card. An example:
-
- #!/bin/bash
- # adjust PATH and LD_LIBRARY_PATH according to the toolchain/libraries your app is using.
- export PATH=/apps/all/impi/5.0.3.048-iccifort-2015.3.187/mic/bin:$PATH
- export LD_LIBRARY_PATH=/apps/all/impi/5.0.3.048-iccifort-2015.3.187/mic/lib:/apps/all/ifort/2015.3.187/lib/mic:/apps/all/icc/2015.3.187/lib/mic:$LD_LIBRARY_PATH
- export MIC_OMP_NUM_THREADS=60
- export MYO_WATCHDOG_MONITOR=-1
- export AMPLXE_COI_DEBUG_SUPPORT=true
- unset OFFLOAD_MAIN
- export I_MPI_MIC=1
-
-Save the script in eg. ~/remote-mic.sh.
-Now, start an interactive graphical session on a node with
-accelerator:
-
- $ qsub â€IX â€q qexp â€l select=1:ncpus=24:accelerator=True
-
-Launch Forge :
-
- $ module load Forge
- $ forge&
-
-Now click on the remote launch drop-down list, select "Configure..." and
-Add a new remote connection with the following parameters:
-
-Connection name: mic0
-
-Hostname: mic0
-
-Remote Installation Directory:Â /apps/all/Forge/5.1-43967/
-
-Remote script: ~/remote-mic.sh
-
-You can click Test Remote Launch to verify the configuration. After you
-save the remote launch configuration and select it in the dropdown list,
-you can use the Run button in the main windows to remotely launch your
-program on mic0.
+```
Documentation
-------------
+Users can find original User Guide after loading the DDT module:
-Users can find original User Guide after loading the Forge module:Â
-
- $EBROOTFORGE/doc/userguide-forge.pdf
-
-Â
-
-Â
-
-Â [1] Discipline, Magic, Inspiration and Science: Best Practice
-Debugging with Allinea DDT, Workshop conducted at LLNL by Allinea on May
-10, 2013,
-[link](https://computing.llnl.gov/tutorials/allineaDDT/index.html)
-
-Â
+```bash
+ $DDTPATH/doc/userguide.pdf
+```
+[1] Discipline, Magic, Inspiration and Science: Best Practice Debugging with Allinea DDT, Workshop conducted at LLNL by Allinea on May 10, 2013, [link](https://computing.llnl.gov/tutorials/allineaDDT/index.html)
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/debuggers/debuggers.md b/docs.it4i/salomon/software/debuggers/debuggers.md
new file mode 100644
index 000000000..208abd52d
--- /dev/null
+++ b/docs.it4i/salomon/software/debuggers/debuggers.md
@@ -0,0 +1,61 @@
+Debuggers and profilers summary
+===============================
+
+Introduction
+------------
+We provide state of the art programms and tools to develop, profile and debug HPC codes at IT4Innovations. On these pages, we provide an overview of the profiling and debugging tools available on Anslem at IT4I.
+
+Intel debugger
+--------------
+The intel debugger version 13.0 is available, via module intel. The debugger works for applications compiled with C and C++ compiler and the ifort fortran 77/90/95 compiler. The debugger provides java GUI environment. Use [X display](https://docs.it4i.cz/anselm-cluster-documentation/software/debuggers/resolveuid/11e53ad0d2fd4c5187537f4baeedff33) for running the GUI.
+
+```bash
+ $ module load intel
+ $ idb
+```
+
+Read more at the [Intel Debugger](intel-suite/intel-debugger.html) page.
+
+Allinea Forge (DDT/MAP)
+-----------------------
+Allinea DDT, is a commercial debugger primarily for debugging parallel MPI or OpenMP programs. It also has a support for GPU (CUDA) and Intel Xeon Phi accelerators. DDT provides all the standard debugging features (stack trace, breakpoints, watches, view variables, threads etc.) for every thread running as part of your program, or for every process even if these processes are distributed across a cluster using an MPI implementation.
+
+```bash
+ $ module load Forge
+ $ forge
+```
+
+Read more at the [Allinea DDT](debuggers/allinea-ddt.html) page.
+
+Allinea Performance Reports
+---------------------------
+Allinea Performance Reports characterize the performance of HPC application runs. After executing your application through the tool, a synthetic HTML report is generated automatically, containing information about several metrics along with clear behavior statements and hints to help you improve the efficiency of your runs. Our license is limited to 64 MPI processes.
+
+```bash
+ $ module load PerformanceReports/6.0
+ $ perf-report mpirun -n 64 ./my_application argument01 argument02
+```
+
+Read more at the [Allinea Performance Reports](debuggers/allinea-performance-reports.html) page.
+
+RougeWave Totalview
+-------------------
+TotalView is a source- and machine-level debugger for multi-process, multi-threaded programs. Its wide range of tools provides ways to analyze, organize, and test programs, making it easy to isolate and identify problems in individual threads and processes in programs of great complexity.
+
+```bash
+ $ module load TotalView/8.15.4-6-linux-x86-64
+ $ totalview
+```
+
+Read more at the [Totalview](debuggers/total-view.html) page.
+
+Vampir trace analyzer
+---------------------
+Vampir is a GUI trace analyzer for traces in OTF format.
+
+```bash
+ $ module load Vampir/8.5.0
+ $ vampir
+```
+
+Read more at the [Vampir](../../salomon/software/debuggers/vampir.html) page.
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/debuggers/intel-vtune-amplifier.md b/docs.it4i/salomon/software/debuggers/intel-vtune-amplifier.md
index efb8cd82b..d04d91449 100644
--- a/docs.it4i/salomon/software/debuggers/intel-vtune-amplifier.md
+++ b/docs.it4i/salomon/software/debuggers/intel-vtune-amplifier.md
@@ -1,137 +1,94 @@
-Intel VTune Amplifier XE
+Intel VTune Amplifier XE
========================
-
-
Introduction
------------
-
-Intel*® *VTune™ >Amplifier, part of Intel Parallel studio, is a GUI
-profiling tool designed for Intel processors. It offers a graphical
-performance analysis of single core and multithreaded applications. A
-highlight of the features:
+Intel*® *VTune™ >Amplifier, part of Intel Parallel studio, is a GUI profiling tool designed for Intel processors. It offers a graphical performance analysis of single core and multithreaded applications. A highlight of the features:
- Hotspot analysis
- Locks and waits analysis
-- Low level specific counters, such as branch analysis and memory
- bandwidth
+- Low level specific counters, such as branch analysis and memory bandwidth
- Power usage analysis - frequency and sleep states.
Usage
-----
+To profile an application with VTune Amplifier, special kernel modules need to be loaded. The modules are not loaded on the login nodes, thus direct profiling on login nodes is not possible. By default, the kernel modules ale not loaded on compute nodes neither. In order to have the modules loaded, you need to specify vtune=version PBS resource at job submit. The version is the same as for environment module. For example to use VTune/2016_update1:
-To profile an application with VTune Amplifier, special kernel
-modules need to be loaded. The modules are not loaded on the login
-nodes, thus direct profiling on login nodes is not possible. By default,
-the kernel modules ale not loaded on compute nodes neither. In order to
-have the modules loaded, you need to specify vtune=version PBS resource
-at job submit. The version is the same as for *environment module*. For
-example to use
-VTune/2016_update1:
-
+```bash
$ qsub -q qexp -A OPEN-0-0 -I -l select=1,vtune=2016_update1
+```
-After that, you can verify the modules sep*, pax and vtsspp are present
-in the kernel :
+After that, you can verify the modules sep*, pax and vtsspp are present in the kernel :
+```bash
$ lsmod | grep -e sep -e pax -e vtsspp
vtsspp 362000 0
sep3_15 546657 0
pax 4312 0
+```
To launch the GUI, first load the module:
+```bash
$ module add VTune/2016_update1
+```
+and launch the GUI :
- class="s1">and launch the GUI :
-
+```bash
$Â amplxe-gui
+```
+
+The GUI will open in new window. Click on "New Project..." to create a new project. After clicking OK, a new window with project properties will appear. At "Application:", select the bath to your binary you want to profile (the binary should be compiled with -g flag). Some additional options such as command line arguments can be selected. At "Managed code profiling mode:" select "Native" (unless you want to profile managed mode .NET/Mono applications). After clicking OK, your project is created.
-The GUI will open in new window. Click on "*New Project...*" to
-create a new project. After clicking *OK*, a new window with project
-properties will appear. Â At "*Application:*", select the bath to your
-binary you want to profile (the binary should be compiled with -g flag).
-Some additional options such as command line arguments can be selected.
-At "*Managed code profiling mode:*" select "*Native*" (unless you want
-to profile managed mode .NET/Mono applications). After clicking *OK*,
-your project is created.
-
-To run a new analysis, click "*New analysis...*". You will see a list of
-possible analysis. Some of them will not be possible on the current CPU
-(eg. Intel Atom analysis is not possible on Sandy bridge CPU), the GUI
-will show an error box if you select the wrong analysis. For example,
-select "*Advanced Hotspots*". Clicking on *Start *will start profiling
-of the application.
+To run a new analysis, click "New analysis...". You will see a list of possible analysis. Some of them will not be possible on the current CPU (eg. Intel Atom analysis is not possible on Sandy bridge CPU), the GUI will show an error box if you select the wrong analysis. For example, select "Advanced Hotspots". Clicking on Start will start profiling of the application.
Remote Analysis
---------------
-VTune Amplifier also allows a form of remote analysis. In this mode,
-data for analysis is collected from the command line without GUI, and
-the results are then loaded to GUI on another machine. This allows
-profiling without interactive graphical jobs. To perform a remote
-analysis, launch a GUI somewhere, open the new analysis window and then
-click the button "*Command line*" in bottom right corner. It will show
-the command line needed to perform the selected analysis.
+VTune Amplifier also allows a form of remote analysis. In this mode, data for analysis is collected from the command line without GUI, and the results are then loaded to GUI on another machine. This allows profiling without interactive graphical jobs. To perform a remote analysis, launch a GUI somewhere, open the new analysis window and then click the button "Command line" in bottom right corner. It will show the command line needed to perform the selected analysis.
The command line will look like this:
+```bash
/apps/all/VTune/2016_update1/vtune_amplifier_xe_2016.1.1.434111/bin64/amplxe-cl -collect advanced-hotspots -app-working-dir /home/sta545/tmp -- /home/sta545/tmp/sgemm
+```
-Copy the line to clipboard and then you can paste it in your jobscript
-or in command line. After the collection is run, open the GUI once
-again, click the menu button in the upper right corner, and select
-"*Open > Result...*". The GUI will load the results from the run.
+Copy the line to clipboard and then you can paste it in your jobscript or in command line. After the collection is run, open the GUI once again, click the menu button in the upper right corner, and select "Open > Result...". The GUI will load the results from the run.
Xeon Phi
--------
-
-It is possible to analyze both native and offloaded Xeon Phi
-applications.Â
+It is possible to analyze both native and offloaded Xeon Phi applications.
### Native mode
-This mode is useful for native Xeon Phi applications launched directly
-on the card. In *Analysis Target* window, select *Intel Xeon Phi
-coprocessor (native), *choose path to the binary and MIC card to run on.
+This mode is useful for native Xeon Phi applications launched directly on the card. In *Analysis Target* window, select *Intel Xeon Phi coprocessor (native), *choose path to the binary and MIC card to run on.
### Offload mode
-This mode is useful for applications that are launched from the host and
-use offload, OpenCL or mpirun. In *Analysis Target* window,
-select *Intel Xeon Phi coprocessor (native), *choose path to the binary
-and MIC card to run on.
-
-Â
+This mode is useful for applications that are launched from the host and use offload, OpenCL or mpirun. In *Analysis Target* window, select *Intel Xeon Phi coprocessor (native), *choose path to the binaryand MIC card to run on.
-If the analysis is interrupted or aborted, further analysis on the card
-might be impossible and you will get errors like "ERROR connecting to
-MIC card". In this case please contact our support to reboot the MIC
-card.
+>If the analysis is interrupted or aborted, further analysis on the card might be impossible and you will get errors like "ERROR connecting to MIC card". In this case please contact our support to reboot the MIC card.
-You may also use remote analysis to collect data from the MIC and then
-analyze it in the GUI later :
+You may also use remote analysis to collect data from the MIC and then analyze it in the GUI later :
Native launch:
+```bash
$ /apps/all/VTune/2016_update1/vtune_amplifier_xe_2016.1.1.434111/bin64/amplxe-cl -target-system mic-native:0 -collect advanced-hotspots -- /home/sta545/tmp/vect-add-mic
+```
Host launch:
+```bash
$ /apps/all/VTune/2016_update1/vtune_amplifier_xe_2016.1.1.434111/bin64/amplxe-cl -target-system mic-host-launch:0 -collect advanced-hotspots -- /home/sta545/tmp/sgemm
+```
-You can obtain this command line by pressing the "Command line..."
-button on Analysis Type screen.Â
+You can obtain this command line by pressing the "Command line..." button on Analysis Type screen.
References
----------
-
-1. ><https://www.rcac.purdue.edu/tutorials/phi/PerformanceTuningXeonPhi-Tullos.pdf>Â Performance
- Tuning for Intel® Xeon Phi™ Coprocessors
-2. ><https://software.intel.com/en-us/intel-vtune-amplifier-xe-support/documentation> >Intel®
- VTune™ Amplifier Support
-3. ><https://software.intel.com/en-us/amplifier_help_linux>Â Linux
- user guide
-
+1. <https://www.rcac.purdue.edu/tutorials/phi/PerformanceTuningXeonPhi-Tullos.pdf> Performance Tuning for Intel® Xeon Phi™ Coprocessors
+2. <https://software.intel.com/en-us/intel-vtune-amplifier-xe-support/documentation> >Intel® VTune™ Amplifier Support
+3. <https://software.intel.com/en-us/amplifier_help_linux>Â Linux user guide
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/debuggers/summary.md b/docs.it4i/salomon/software/debuggers/summary.md
deleted file mode 100644
index a9ec88e50..000000000
--- a/docs.it4i/salomon/software/debuggers/summary.md
+++ /dev/null
@@ -1,87 +0,0 @@
-Debuggers and profilers summary
-===============================
-
-
-
-Introduction
-------------
-
-We provide state of the art programms and tools to develop, profile and
-debug HPC codes at IT4Innovations.
-On these pages, we provide an overview of the profiling and debugging
-tools available on Anslem at IT4I.
-
-Intel debugger
---------------
-
-Intel debugger is no longer available since Parallel Studio version 2015
-
-The intel debugger version 13.0 is available, via module intel. The
-debugger works for applications compiled with C and C++ compiler and the
-ifort fortran 77/90/95 compiler. The debugger provides java GUI
-environment. Use [X
-display](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-system/x-window-and-vnc.html)
-for running the GUI.
-
- $ module load intel
- $ idb
-
-Read more at the [Intel
-Debugger](../intel-suite/intel-debugger.html) page.
-
-Allinea Forge (DDT/MAP)
------------------------
-
-Allinea DDT, is a commercial debugger primarily for debugging parallel
-MPI or OpenMP programs. It also has a support for GPU (CUDA) and Intel
-Xeon Phi accelerators. DDT provides all the standard debugging features
-(stack trace, breakpoints, watches, view variables, threads etc.) for
-every thread running as part of your program, or for every process -
-even if these processes are distributed across a cluster using an MPI
-implementation.
-
- $ module load Forge
- $ forge
-
-Read more at the [Allinea DDT](allinea-ddt.html) page.
-
-Allinea Performance Reports
----------------------------
-
-Allinea Performance Reports characterize the performance of HPC
-application runs. After executing your application through the tool, a
-synthetic HTML report is generated automatically, containing information
-about several metrics along with clear behavior statements and hints to
-help you improve the efficiency of your runs. Our license is limited to
-64 MPI processes.
-
- $ module load PerformanceReports/6.0
- $ perf-report mpirun -n 64 ./my_application argument01 argument02
-
-Read more at the [Allinea Performance
-Reports](allinea-performance-reports.html) page.
-
-RougeWave Totalview
--------------------
-
-TotalView is a source- and machine-level debugger for multi-process,
-multi-threaded programs. Its wide range of tools provides ways to
-analyze, organize, and test programs, making it easy to isolate and
-identify problems in individual threads and processes in programs of
-great complexity.
-
- $ module load TotalView/8.15.4-6-linux-x86-64
- $ totalview
-
-Read more at the [Totalview](total-view.html) page.
-
-Vampir trace analyzer
----------------------
-
-Vampir is a GUI trace analyzer for traces in OTF format.
-
- $ module load Vampir/8.5.0
- $ vampir
-
-Read more at the [Vampir](vampir.html) page.
-
diff --git a/docs.it4i/salomon/software/debuggers/total-view.md b/docs.it4i/salomon/software/debuggers/total-view.md
index 9a7166c43..0568ccc42 100644
--- a/docs.it4i/salomon/software/debuggers/total-view.md
+++ b/docs.it4i/salomon/software/debuggers/total-view.md
@@ -1,22 +1,20 @@
-Total View
+Total View
==========
-TotalView is a GUI-based source code multi-process, multi-thread
-debugger.
+TotalView is a GUI-based source code multi-process, multi-thread debugger.
License and Limitations for cluster Users
-----------------------------------------
+On the cluster users can debug OpenMP or MPI code that runs up to 64 parallel processes. These limitation means that:
-On the cluster users can debug OpenMP or MPI code that runs up to 64
-parallel processes. These limitation means that:
-
+```bash
   1 user can debug up 64 processes, or
   32 users can debug 2 processes, etc.
+```
Debugging of GPU accelerated codes is also supported.
-You can check the status of the licenses
-[here](https://extranet.it4i.cz/rsweb/anselm/license/totalview).
+You can check the status of the licenses [here](https://extranet.it4i.cz/rsweb/anselm/license/totalview).
Compiling Code to run with TotalView
------------------------------------
@@ -25,65 +23,70 @@ Compiling Code to run with TotalView
Load all necessary modules to compile the code. For example:
+```bash
module load intel
module load impi  ... or ... module load OpenMPI/X.X.X-icc
+```
Load the TotalView module:
+```bash
module load TotalView/8.15.4-6-linux-x86-64
+```
Compile the code:
+```bash
mpicc -g -O0 -o test_debug test.c
mpif90 -g -O0 -o test_debug test.f
+```
### Compiler flags
Before debugging, you need to compile your code with theses flags:
--g** : Generates extra debugging information usable by GDB. -g3**
+>**-g** : Generates extra debugging information usable by GDB. -g3
includes even more debugging information. This option is available for
GNU and INTEL C/C++ and Fortran compilers.
--O0** : Suppress all optimizations.**
+>**-O0** : Suppress all optimizations.
Starting a Job with TotalView
-----------------------------
-
Be sure to log in with an X window forwarding enabled. This could mean
-using the -X in the ssh:Â
+using the -X in the ssh:
- ssh -X username@salomon.it4i.cz
+```bash
+ ssh -X username@salomon.it4i.cz
+```
-Other options is to access login node using VNC. Please see the detailed
-information on how to use graphic user interface on Anselm
-[here](https://docs.it4i.cz/salomon/software/debuggers/resolveuid/11e53ad0d2fd4c5187537f4baeedff33#VNC).
+Other options is to access login node using VNC. Please see the detailed information on how to use graphic user interface on Anselm.
-From the login node an interactive session with X windows forwarding (-X
-option) can be started by following command:
+From the login node an interactive session with X windows forwarding (-X option) can be started by following command:
- qsub -I -X -A NONE-0-0 -q qexp -lselect=1:ncpus=24:mpiprocs=24,walltime=01:00:00
+```bash
+ qsub -I -X -A NONE-0-0 -q qexp -lselect=1:ncpus=24:mpiprocs=24,walltime=01:00:00
+```
-Then launch the debugger with the totalview command followed by the name
-of the executable to debug.
+Then launch the debugger with the totalview command followed by the name of the executable to debug.
### Debugging a serial code
To debug a serial code use:
+```bash
totalview test_debug
+```
### Debugging a parallel code - option 1
-To debug a parallel code compiled with >**OpenMPI** you need
-to setup your TotalView environment:Â
+To debug a parallel code compiled with **OpenMPI** you need to setup your TotalView environment:
-Please note:** To be able to run parallel debugging procedure from the
-command line without stopping the debugger in the mpiexec source code
-you have to add the following function to your **~/.tvdrc** file:
+>**Please note:** To be able to run parallel debugging procedure from the command line without stopping the debugger in the mpiexec source code you have to add the following function to your **~/.tvdrc** file:
+```bash
proc mpi_auto_run_starter {loaded_id} {
   set starter_programs {mpirun mpiexec orterun}
   set executable_name [TV::symbol get $loaded_id full_pathname]
@@ -101,57 +104,52 @@ you have to add the following function to your **~/.tvdrc** file:
# TotalView run this program automatically.
dlappend TV::image_load_callbacks mpi_auto_run_starter
+```
The source code of this function can be also found in
+```bash
/apps/all/OpenMPI/1.10.1-GNU-4.9.3-2.25/etc/openmpi-totalview.tcl
+```
You can also add only following line to you ~/.tvdrc file instead of
the entire function:
-source /apps/all/OpenMPI/1.10.1-GNU-4.9.3-2.25/etc/openmpi-totalview.tcl**
+**source /apps/all/OpenMPI/1.10.1-GNU-4.9.3-2.25/etc/openmpi-totalview.tcl**
-You need to do this step only once. See also [OpenMPI FAQ
-entry](https://www.open-mpi.org/faq/?category=running#run-with-tv)
+You need to do this step only once. See also [OpenMPI FAQ entry](https://www.open-mpi.org/faq/?category=running#run-with-tv)
Now you can run the parallel debugger using:
+```bash
mpirun -tv -n 5 ./test_debug
+```
When following dialog appears click on "Yes"
-At this point the main TotalView GUI window will appear and you can
-insert the breakpoints and start debugging:
+At this point the main TotalView GUI window will appear and you can insert the breakpoints and start debugging:
### Debugging a parallel code - option 2
-Other option to start new parallel debugging session from a command line
-is to let TotalView to execute mpirun by itself. In this case user has
-to specify a MPI implementation used to compile the source code.Â
+Other option to start new parallel debugging session from a command line is to let TotalView to execute mpirun by itself. In this case user has to specify a MPI implementation used to compile the source code.
-The following example shows how to start debugging session with Intel
-MPI:
+The following example shows how to start debugging session with Intel MPI:
+```bash
module load intel/2015b-intel-2015b impi/5.0.3.048-iccifort-2015.3.187-GNU-5.1.0-2.25 TotalView/8.15.4-6-linux-x86-64
totalview -mpi "Intel MPI-Hydra" -np 8 ./hello_debug_impi
+```
-After running previous command you will see the same window as shown in
-the screenshot above.
+After running previous command you will see the same window as shown in the screenshot above.
-More information regarding the command line parameters of the TotalView
-can be found TotalView Reference Guide, Chapter 7: TotalView Command
-Syntax. Â
+More information regarding the command line parameters of the TotalView can be found TotalView Reference Guide, Chapter 7: TotalView Command Syntax.
Documentation
-------------
-[1] The [TotalView
-documentation](http://www.roguewave.com/support/product-documentation/totalview-family.aspx#totalview)
-web page is a good resource for learning more about some of the advanced
-TotalView features.
-
+[1] The [TotalView documentation](http://www.roguewave.com/support/product-documentation/totalview-family.aspx#totalview) web page is a good resource for learning more about some of the advanced TotalView features.
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/debuggers/valgrind.md b/docs.it4i/salomon/software/debuggers/valgrind.md
index 1cd426c1d..ddbfd7895 100644
--- a/docs.it4i/salomon/software/debuggers/valgrind.md
+++ b/docs.it4i/salomon/software/debuggers/valgrind.md
@@ -1,71 +1,41 @@
-Valgrind
+Valgrind
========
Valgrind is a tool for memory debugging and profiling.
About Valgrind
--------------
+Valgrind is an open-source tool, used mainly for debuggig memory-related problems, such as memory leaks, use of uninitalized memory etc. in C/C++ applications. The toolchain was however extended over time with more functionality, such as debugging of threaded applications, cache profiling, not limited only to C/C++.
-Valgrind is an open-source tool, used mainly for debuggig memory-related
-problems, such as memory leaks, use of uninitalized memory etc. in C/C++
-applications. The toolchain was however extended over time with more
-functionality, such as debugging of threaded applications, cache
-profiling, not limited only to C/C++.
-
-Valgind is an extremely useful tool for debugging memory errors such as
-[off-by-one](http://en.wikipedia.org/wiki/Off-by-one_error).
-Valgrind uses a virtual machine and dynamic recompilation of binary
-code, because of that, you can expect that programs being debugged by
-Valgrind run 5-100 times slower.
+Valgind is an extremely useful tool for debugging memory errors such as [off-by-one](http://en.wikipedia.org/wiki/Off-by-one_error). Valgrind uses a virtual machine and dynamic recompilation of binary code, because of that, you can expect that programs being debugged by Valgrind run 5-100 times slower.
The main tools available in Valgrind are :
-- **Memcheck**, the original, must used and default tool. Verifies
- memory access in you program and can detect use of unitialized
- memory, out of bounds memory access, memory leaks, double free, etc.
+- **Memcheck**, the original, must used and default tool. Verifies memory access in you program and can detect use of unitialized memory, out of bounds memory access, memory leaks, double free, etc.
- **Massif**, a heap profiler.
-- **Hellgrind** and **DRD** can detect race conditions in
- multi-threaded applications.
+- **Hellgrind** and **DRD** can detect race conditions in multi-threaded applications.
- **Cachegrind**, a cache profiler.
- **Callgrind**, a callgraph analyzer.
-- For a full list and detailed documentation, please refer to the
- [official Valgrind
- documentation](http://valgrind.org/docs/).
+- For a full list and detailed documentation, please refer to the [official Valgrind documentation](http://valgrind.org/docs/).
Installed versions
------------------
-
There are two versions of Valgrind available on the cluster.
-- >Version 3.8.1, installed by operating system vendor
- in /usr/bin/valgrind.
- >This version is available by default, without the need
- to load any module. This version however does not provide additional
- MPI support. Also, it does not support AVX2 instructions,
- **debugging of an AVX2-enabled executable with this version will
- fail**
-- >Version 3.11.0 built by ICC with support for Intel MPI,
- available in
- [module](../../environment-and-modules.html)Â
- Valgrind/3.11.0-intel-2015b. After loading
- the module, this version replaces the default valgrind.
-- Version 3.11.0 built by GCC with support for Open MPI, module
- Valgrind/3.11.0-foss-2015b
+- Version 3.8.1, installed by operating system vendor in /usr/bin/valgrind. This version is available by default, without the need to load any module. This version however does not provide additional MPI support. Also, it does not support AVX2 instructions, debugging of an AVX2-enabled executable with this version will fail
+- Version 3.11.0 built by ICC with support for Intel MPI, available in module Valgrind/3.11.0-intel-2015b. After loading the module, this version replaces the default valgrind.
+- Version 3.11.0 built by GCC with support for Open MPI, module Valgrind/3.11.0-foss-2015b
Usage
-----
+Compile the application which you want to debug as usual. It is advisable to add compilation flags -g (to add debugging information to the binary so that you will see original source code lines in the output) and -O0 (to disable compiler optimizations).
-Compile the application which you want to debug as usual. It is
-advisable to add compilation flags -g (to
-add debugging information to the binary so that you will see original
-source code lines in the output) and -O0
-(to disable compiler optimizations).Â
-
-For example, lets look at this C code, which has two problems :
+For example, lets look at this C code, which has two problems:
+```cpp
#include <stdlib.h>
- void f(void)
+ void f(void)
{
int* x = malloc(10 * sizeof(int));
x[10] = 0; // problem 1: heap block overrun
@@ -76,27 +46,28 @@ For example, lets look at this C code, which has two problems :
f();
return 0;
}
+```
-Now, compile it with Intel compiler :
+Now, compile it with Intel compiler:
+```bash
$ module add intel
- $ icc -g valgrind-example.c -o valgrind-exampleÂ
+ $ icc -g valgrind-example.c -o valgrind-example
+```
-Now, lets run it with Valgrind. The syntax is :
+Now, lets run it with Valgrind. The syntax is:
- valgrind [valgrind options] <your program
-binary> [your program options]
+valgrind [valgrind options] <your program binary> [your program options]
-If no Valgrind options are specified, Valgrind defaults to running
-Memcheck tool. Please refer to the Valgrind documentation for a full
-description of command line options.
+If no Valgrind options are specified, Valgrind defaults to running Memcheck tool. Please refer to the Valgrind documentation for a full description of command line options.
+```bash
$ valgrind ./valgrind-example
==12652== Memcheck, a memory error detector
==12652== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==12652== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
==12652== Command: ./valgrind-example
- ==12652==
+ ==12652==
==12652== Invalid write of size 4
==12652== at 0x40053E: f (valgrind-example.c:6)
==12652== by 0x40054E: main (valgrind-example.c:11)
@@ -104,12 +75,12 @@ description of command line options.
==12652== at 0x4C27AAA: malloc (vg_replace_malloc.c:291)
==12652== by 0x400528: f (valgrind-example.c:5)
==12652== by 0x40054E: main (valgrind-example.c:11)
- ==12652==
- ==12652==
+ ==12652==
+ ==12652==
==12652== HEAP SUMMARY:
==12652== in use at exit: 40 bytes in 1 blocks
==12652== total heap usage: 1 allocs, 0 frees, 40 bytes allocated
- ==12652==
+ ==12652==
==12652== LEAK SUMMARY:
==12652== definitely lost: 40 bytes in 1 blocks
==12652== indirectly lost: 0 bytes in 0 blocks
@@ -117,21 +88,20 @@ description of command line options.
==12652== still reachable: 0 bytes in 0 blocks
==12652== suppressed: 0 bytes in 0 blocks
==12652== Rerun with --leak-check=full to see details of leaked memory
- ==12652==
+ ==12652==
==12652== For counts of detected and suppressed errors, rerun with: -v
==12652== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 6 from 6)
+```
-In the output we can see that Valgrind has detected both errors - the
-off-by-one memory access at line 5 and a memory leak of 40 bytes. If we
-want a detailed analysis of the memory leak, we need to run Valgrind
-with --leak-check=full option :
+In the output we can see that Valgrind has detected both errors - the off-by-one memory access at line 5 and a memory leak of 40 bytes. If we want a detailed analysis of the memory leak, we need to run Valgrind with --leak-check=full option:
+```bash
$ valgrind --leak-check=full ./valgrind-example
==23856== Memcheck, a memory error detector
==23856== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==23856== Using Valgrind-3.6.0 and LibVEX; rerun with -h for copyright info
==23856== Command: ./valgrind-example
- ==23856==
+ ==23856==
==23856== Invalid write of size 4
==23856== at 0x40067E: f (valgrind-example.c:6)
==23856== by 0x40068E: main (valgrind-example.c:11)
@@ -139,42 +109,41 @@ with --leak-check=full option :
==23856== at 0x4C26FDE: malloc (vg_replace_malloc.c:236)
==23856== by 0x400668: f (valgrind-example.c:5)
==23856== by 0x40068E: main (valgrind-example.c:11)
- ==23856==
- ==23856==
+ ==23856==
+ ==23856==
==23856== HEAP SUMMARY:
==23856== in use at exit: 40 bytes in 1 blocks
==23856== total heap usage: 1 allocs, 0 frees, 40 bytes allocated
- ==23856==
+ ==23856==
==23856== 40 bytes in 1 blocks are definitely lost in loss record 1 of 1
==23856== at 0x4C26FDE: malloc (vg_replace_malloc.c:236)
==23856== by 0x400668: f (valgrind-example.c:5)
==23856== by 0x40068E: main (valgrind-example.c:11)
- ==23856==
+ ==23856==
==23856== LEAK SUMMARY:
==23856== definitely lost: 40 bytes in 1 blocks
==23856== indirectly lost: 0 bytes in 0 blocks
==23856== possibly lost: 0 bytes in 0 blocks
==23856== still reachable: 0 bytes in 0 blocks
==23856== suppressed: 0 bytes in 0 blocks
- ==23856==
+ ==23856==
==23856== For counts of detected and suppressed errors, rerun with: -v
==23856== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 6 from 6)
+```
-Now we can see that the memory leak is due to the
-malloc() at line 6.
+Now we can see that the memory leak is due to the malloc() at line 6.
Usage with MPI
---------------------------
+Although Valgrind is not primarily a parallel debugger, it can be used to debug parallel applications as well. When launching your parallel applications, prepend the valgrind command. For example:
-Although Valgrind is not primarily a parallel debugger, it can be used
-to debug parallel applications as well. When launching your parallel
-applications, prepend the valgrind command. For example :
-
+```bash
$ mpirun -np 4 valgrind myapplication
+```
-The default version without MPI support will however report a large
-number of false errors in the MPI library, such as :
+The default version without MPI support will however report a large number of false errors in the MPI library, such as:
+```bash
==30166== Conditional jump or move depends on uninitialised value(s)
==30166== at 0x4C287E8: strlen (mc_replace_strmem.c:282)
==30166== by 0x55443BD: I_MPI_Processor_model_number (init_interface.c:427)
@@ -187,19 +156,19 @@ number of false errors in the MPI library, such as :
==30166== by 0x554650A: MPIR_Init_thread (initthread.c:539)
==30166== by 0x553369F: PMPI_Init (init.c:195)
==30166== by 0x4008BD: main (valgrind-example-mpi.c:18)
+```
-so it is better to use the MPI-enabled valgrind from module. The MPI
-versions requires library :Â
+so it is better to use the MPI-enabled valgrind from module. The MPI versions requires library:
$EBROOTVALGRIND/lib/valgrind/libmpiwrap-amd64-linux.so
-which must be included in the LD_PRELOAD
-environment variable.
+which must be included in the LD_PRELOAD environment variable.
-Lets look at this MPI example :
+Lets look at this MPI example:
+```cpp
#include <stdlib.h>
- #include <mpi.h>Â
+ #include <mpi.h>
int main(int argc, char *argv[])
{
@@ -207,32 +176,34 @@ Lets look at this MPI example :
     MPI_Init(&argc, &argv);
    MPI_Bcast(data, 100, MPI_INT, 0, MPI_COMM_WORLD);
- Â Â Â Â Â MPI_Finalize();Â
+ Â Â Â Â Â MPI_Finalize();
       return 0;
}
+```
-There are two errors - use of uninitialized memory and invalid length of
-the buffer. Lets debug it with valgrind :
+There are two errors - use of uninitialized memory and invalid length of the buffer. Lets debug it with valgrind :
+```bash
$ module add intel impi
$ mpiicc -g valgrind-example-mpi.c -o valgrind-example-mpi
$ module add Valgrind/3.11.0-intel-2015b
$ mpirun -np 2 -env LD_PRELOAD $EBROOTVALGRIND/lib/valgrind/libmpiwrap-amd64-linux.so valgrind ./valgrind-example-mpi
+```
-Prints this output : (note that there is output printed for every
-launched MPI process)
+Prints this output : (note that there is output printed for every launched MPI process)
+```bash
==31318== Memcheck, a memory error detector
==31318== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==31318== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
==31318== Command: ./valgrind-example-mpi
- ==31318==
+ ==31318==
==31319== Memcheck, a memory error detector
==31319== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==31319== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
==31319== Command: ./valgrind-example-mpi
- ==31319==
+ ==31319==
valgrind MPI wrappers 31319: Active for pid 31319
valgrind MPI wrappers 31319: Try MPIWRAP_DEBUG=help for possible options
valgrind MPI wrappers 31318: Active for pid 31318
@@ -244,7 +215,7 @@ launched MPI process)
==31319== Address 0x69291cc is 0 bytes after a block of size 396 alloc'd
==31319== at 0x4C27AAA: malloc (vg_replace_malloc.c:291)
==31319== by 0x4007BC: main (valgrind-example-mpi.c:8)
- ==31319==
+ ==31319==
==31318== Uninitialised byte(s) found during client check request
==31318== at 0x4E3591D: check_mem_is_defined_untyped (libmpiwrap.c:952)
==31318== by 0x4E5D06D: PMPI_Bcast (libmpiwrap.c:908)
@@ -252,7 +223,7 @@ launched MPI process)
==31318== Address 0x6929040 is 0 bytes inside a block of size 396 alloc'd
==31318== at 0x4C27AAA: malloc (vg_replace_malloc.c:291)
==31318== by 0x4007BC: main (valgrind-example-mpi.c:8)
- ==31318==
+ ==31318==
==31318== Unaddressable byte(s) found during client check request
==31318== at 0x4E3591D: check_mem_is_defined_untyped (libmpiwrap.c:952)
==31318== by 0x4E5D06D: PMPI_Bcast (libmpiwrap.c:908)
@@ -260,17 +231,17 @@ launched MPI process)
==31318== Address 0x69291cc is 0 bytes after a block of size 396 alloc'd
==31318== at 0x4C27AAA: malloc (vg_replace_malloc.c:291)
==31318== by 0x4007BC: main (valgrind-example-mpi.c:8)
- ==31318==
- ==31318==
+ ==31318==
+ ==31318==
==31318== HEAP SUMMARY:
==31318== in use at exit: 3,172 bytes in 67 blocks
==31318== total heap usage: 191 allocs, 124 frees, 81,203 bytes allocated
- ==31318==
- ==31319==
+ ==31318==
+ ==31319==
==31319== HEAP SUMMARY:
==31319== in use at exit: 3,172 bytes in 67 blocks
==31319== total heap usage: 175 allocs, 108 frees, 48,435 bytes allocated
- ==31319==
+ ==31319==
==31318== LEAK SUMMARY:
==31318== definitely lost: 408 bytes in 3 blocks
==31318== indirectly lost: 256 bytes in 1 blocks
@@ -278,7 +249,7 @@ launched MPI process)
==31318== still reachable: 2,508 bytes in 63 blocks
==31318== suppressed: 0 bytes in 0 blocks
==31318== Rerun with --leak-check=full to see details of leaked memory
- ==31318==
+ ==31318==
==31318== For counts of detected and suppressed errors, rerun with: -v
==31318== Use --track-origins=yes to see where uninitialised values come from
==31318== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 4 from 4)
@@ -289,11 +260,8 @@ launched MPI process)
==31319== still reachable: 2,508 bytes in 63 blocks
==31319== suppressed: 0 bytes in 0 blocks
==31319== Rerun with --leak-check=full to see details of leaked memory
- ==31319==
+ ==31319==
==31319== For counts of detected and suppressed errors, rerun with: -v
==31319== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 4 from 4)
-We can see that Valgrind has reported use of unitialised memory on the
-master process (which reads the array to be broadcasted) and use of
-unaddresable memory on both processes.
-
+We can see that Valgrind has reported use of unitialised memory on the master process (which reads the array to be broadcasted) and use of unaddresable memory on both processes.
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/debuggers/vampir.md b/docs.it4i/salomon/software/debuggers/vampir.md
index 9d213a421..254aac993 100644
--- a/docs.it4i/salomon/software/debuggers/vampir.md
+++ b/docs.it4i/salomon/software/debuggers/vampir.md
@@ -1,32 +1,24 @@
-Vampir
+Vampir
======
-Vampir is a commercial trace analysis and visualisation tool. It can
-work with traces in OTF and OTF2 formats. It does not have the
-functionality to collect traces, you need to use a trace collection tool
-(such as [Score-P](score-p.html)) first to collect the
-traces.
+Vampir is a commercial trace analysis and visualisation tool. It can work with traces in OTF and OTF2 formats. It does not have the functionality to collect traces, you need to use a trace collection tool (such as [Score-P](score-p.md)) first to collect the traces.
--
Installed versions
------------------
+Version 8.5.0 is currently installed as module Vampir/8.5.0 :
-Version 8.5.0 is currently installed as module
-Vampir/8.5.0 :
-
+```bash
$ module load Vampir/8.5.0
$ vampir &
+```
User manual
-----------
-
-You can find the detailed user manual in PDF format in
-$EBROOTVAMPIR/doc/vampir-manual.pdf
+You can find the detailed user manual in PDF format in $EBROOTVAMPIR/doc/vampir-manual.pdf
References
----------
-
1. <https://www.vampir.eu>
diff --git a/docs.it4i/salomon/software/intel-suite/intel-advisor.md b/docs.it4i/salomon/software/intel-suite/intel-advisor.md
index 805e7d905..7088e53c5 100644
--- a/docs.it4i/salomon/software/intel-suite/intel-advisor.md
+++ b/docs.it4i/salomon/software/intel-suite/intel-advisor.md
@@ -1,54 +1,32 @@
-Intel Advisor
+Intel Advisor
=============
-is tool aiming to assist you in vectorization and threading of your
-code. You can use it to profile your application and identify
-loops, that could benefit from vectorization and/or threading
-parallelism.
+is tool aiming to assist you in vectorization and threading of your code. You can use it to profile your application and identify loops, that could benefit from vectorization and/or threading parallelism.
Installed versions
------------------
-
The following versions are currently available on Salomon as modules:
- --------- |---|---|-------------
- |---|---|
- |2016 Update 2|Advisor/2016_update2|
- --------- |---|---|-------------
+2016 Update 2 - Advisor/2016_update2
Usage
-----
+Your program should be compiled with -g switch to include symbol names. You should compile with -O2 or higher to see code that is already vectorized by the compiler.
-Your program should be compiled with -g switch to include symbol names.
-You should compile with -O2 or higher to see code that is already
-vectorized by the compiler.
-
-Profiling is possible either directly from the GUI, or from command
-line.
+Profiling is possible either directly from the GUI, or from command line.
To profile from GUI, launch Advisor:
+```bash
$ advixe-gui
+```
-Then select menu File -> New -> Project. Choose a directory to
-save project data to. After clicking OK, Project properties window will
-appear, where you can configure path to your binary, launch arguments,
-working directory etc. After clicking OK, the project is ready.
+Then select menu File -> New -> Project. Choose a directory to save project data to. After clicking OK, Project properties window will appear, where you can configure path to your binary, launch arguments, working directory etc. After clicking OK, the project is ready.
-In the left pane, you can switch between Vectorization and Threading
-workflows. Each has several possible steps which you can execute by
-clicking Collect button. Alternatively, you can click on Command Line,
-to see the command line required to run the analysis directly from
-command line.
+In the left pane, you can switch between Vectorization and Threading workflows. Each has several possible steps which you can execute by clicking Collect button. Alternatively, you can click on Command Line, to see the command line required to run the analysis directly from command line.
References
----------
-
-1. [Intel® Advisor 2015 Tutorial: Find Where to Add Parallelism - C++
- Sample](https://software.intel.com/en-us/advisorxe_2015_tut_lin_c)
-2. [Product
- page](https://software.intel.com/en-us/intel-advisor-xe)
-3. [Documentation](https://software.intel.com/en-us/intel-advisor-2016-user-guide-linux)
-
-Â
-
+1. [Intel® Advisor 2015 Tutorial: Find Where to Add Parallelism - C++ Sample](https://software.intel.com/en-us/advisorxe_2015_tut_lin_c)
+2. [Product page](https://software.intel.com/en-us/intel-advisor-xe)
+3. [Documentation](https://software.intel.com/en-us/intel-advisor-2016-user-guide-linux)
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/intel-suite/intel-compilers.md b/docs.it4i/salomon/software/intel-suite/intel-compilers.md
index b846db42a..b496c4feb 100644
--- a/docs.it4i/salomon/software/intel-suite/intel-compilers.md
+++ b/docs.it4i/salomon/software/intel-suite/intel-compilers.md
@@ -1,71 +1,37 @@
-Intel Compilers
+Intel Compilers
===============
-
-
-The Intel compilers in multiple versions are available, via module
-intel. The compilers include the icc C and C++ compiler and the ifort
-fortran 77/90/95 compiler.
+The Intel compilers in multiple versions are available, via module intel. The compilers include the icc C and C++ compiler and the ifort fortran 77/90/95 compiler.
+```bash
$ module load intel
$ icc -v
$ ifort -v
+```
-The intel compilers provide for vectorization of the code, via the AVX2
-instructions and support threading parallelization via OpenMP
+The intel compilers provide for vectorization of the code, via the AVX2 instructions and support threading parallelization via OpenMP
-For maximum performance on the Salomon cluster compute nodes, compile
-your programs using the AVX2 instructions, with reporting where the
-vectorization was used. We recommend following compilation options for
-high performance
+For maximum performance on the Salomon cluster compute nodes, compile your programs using the AVX2 instructions, with reporting where the vectorization was used. We recommend following compilation options for high performance
+```bash
$ icc -ipo -O3 -xCORE-AVX2 -qopt-report1 -qopt-report-phase=vec myprog.c mysubroutines.c -o myprog.x
$ ifort -ipo -O3 -xCORE-AVX2 -qopt-report1 -qopt-report-phase=vec myprog.f mysubroutines.f -o myprog.x
+```
-In this example, we compile the program enabling interprocedural
-optimizations between source files (-ipo), aggresive loop optimizations
-(-O3) and vectorization (-xCORE-AVX2)
+In this example, we compile the program enabling interprocedural optimizations between source files (-ipo), aggresive loop optimizations (-O3) and vectorization (-xCORE-AVX2)
-The compiler recognizes the omp, simd, vector and ivdep pragmas for
-OpenMP parallelization and AVX2 vectorization. Enable the OpenMP
-parallelization by the **-openmp** compiler switch.
+The compiler recognizes the omp, simd, vector and ivdep pragmas for OpenMP parallelization and AVX2 vectorization. Enable the OpenMP parallelization by the **-openmp** compiler switch.
+```bash
$ icc -ipo -O3 -xCORE-AVX2 -qopt-report1 -qopt-report-phase=vec -openmp myprog.c mysubroutines.c -o myprog.x
$ ifort -ipo -O3 -xCORE-AVX2 -qopt-report1 -qopt-report-phase=vec -openmp myprog.f mysubroutines.f -o myprog.x
+```
-Read more
-at <https://software.intel.com/en-us/intel-cplusplus-compiler-16.0-user-and-reference-guide>
+Read more at <https://software.intel.com/en-us/intel-cplusplus-compiler-16.0-user-and-reference-guide>
Sandy Bridge/Ivy Bridge/Haswell binary compatibility
----------------------------------------------------
+ Anselm nodes are currently equipped with Sandy Bridge CPUs, while Salomon compute nodes are equipped with Haswell based architecture. The UV1 SMP compute server has Ivy Bridge CPUs, which are equivalent to Sandy Bridge (only smaller manufacturing technology). The new processors are backward compatible with the Sandy Bridge nodes, so all programs that ran on the Sandy Bridge processors, should also run on the new Haswell nodes. To get optimal performance out of the Haswell processors a program should make use of the special AVX2 instructions for this processor. One can do this by recompiling codes with the compiler flags designated to invoke these instructions. For the Intel compiler suite, there are two ways of doing this:
-Anselm nodes are currently equipped with Sandy Bridge CPUs, while
-Salomon compute nodes are equipped with Haswell based architecture. The
-UV1 SMP compute server has Ivy Bridge CPUs, which are equivalent to
-Sandy Bridge (only smaller manufacturing technology). >The new
-processors are backward compatible with the Sandy Bridge nodes, so all
-programs that ran on the Sandy Bridge processors, should also run on the
-new Haswell nodes. >To get optimal performance out of the
-Haswell processors a program should make use of the
-special >AVX2 instructions for this processor. One can do
-this by recompiling codes with the compiler
-flags >designated to invoke these instructions. For the
-Intel compiler suite, there are two ways of
-doing >this:
-
-- >Using compiler flag (both for Fortran and C):
- -xCORE-AVX2. This will create a
- binary class="s1">with AVX2 instructions, specifically
- for the Haswell processors. Note that the
- executable >will not run on Sandy Bridge/Ivy
- Bridge nodes.
-- >Using compiler flags (both for Fortran and C):
- -xAVX -axCORE-AVX2. This
- will >generate multiple, feature specific auto-dispatch
- code paths for Intel® processors, if there is >a
- performance benefit. So this binary will run both on Sandy
- Bridge/Ivy Bridge and Haswell >processors. During
- runtime it will be decided which path to follow, dependent on
- which >processor you are running on. In general this
- will result in larger binaries.
-
+- Using compiler flag (both for Fortran and C): -xCORE-AVX2. This will create a binary with AVX2 instructions, specifically for the Haswell processors. Note that the executable will not run on Sandy Bridge/Ivy Bridge nodes.
+- Using compiler flags (both for Fortran and C): -xAVX -axCORE-AVX2. This will generate multiple, feature specific auto-dispatch code paths for Intel® processors, if there is a performance benefit. So this binary will run both on Sandy Bridge/Ivy Bridge and Haswell processors. During runtime it will be decided which path to follow, dependent on which processor you are running on. In general this will result in larger binaries.
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/intel-suite/intel-debugger.md b/docs.it4i/salomon/software/intel-suite/intel-debugger.md
index a806174fd..8622cc3fc 100644
--- a/docs.it4i/salomon/software/intel-suite/intel-debugger.md
+++ b/docs.it4i/salomon/software/intel-suite/intel-debugger.md
@@ -1,35 +1,29 @@
-Intel Debugger
+Intel Debugger
==============
-
-
IDB is no longer available since Intel Parallel Studio 2015
Debugging serial applications
-----------------------------
+The intel debugger version 13.0 is available, via module intel. The debugger works for applications compiled with C and C++ compiler and the ifort fortran 77/90/95 compiler. The debugger provides java GUI environment. Use [X display](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-system/x-window-and-vnc.md) for running the GUI.
-The intel debugger version 13.0 is available, via module intel. The
-debugger works for applications compiled with C and C++ compiler and the
-ifort fortran 77/90/95 compiler. The debugger provides java GUI
-environment. Use [X
-display](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-system/x-window-and-vnc.html)
-for running the GUI.
-
+```bash
$ module load intel/2014.06
$ module load Java
$ idb
+```
The debugger may run in text mode. To debug in text mode, use
+```bash
$ idbc
+```
-To debug on the compute nodes, module intel must be loaded.
-The GUI on compute nodes may be accessed using the same way as in [the
-GUI
-section](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-system/x-window-and-vnc.html)
+To debug on the compute nodes, module intel must be loaded. The GUI on compute nodes may be accessed using the same way as in [the GUI section](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-and-vnc.md)
Example:
+```bash
$ qsub -q qexp -l select=1:ncpus=24 -X -I
qsub: waiting for job 19654.srv11 to start
qsub: job 19654.srv11 ready
@@ -38,63 +32,46 @@ Example:
$ module load Java
$ icc -O0 -g myprog.c -o myprog.x
$ idb ./myprog.x
+```
-In this example, we allocate 1 full compute node, compile program
-myprog.c with debugging options -O0 -g and run the idb debugger
-interactively on the myprog.x executable. The GUI access is via X11 port
-forwarding provided by the PBS workload manager.
+In this example, we allocate 1 full compute node, compile program myprog.c with debugging options -O0 -g and run the idb debugger interactively on the myprog.x executable. The GUI access is via X11 port forwarding provided by the PBS workload manager.
Debugging parallel applications
-------------------------------
-
-Intel debugger is capable of debugging multithreaded and MPI parallel
-programs as well.
+ Intel debugger is capable of debugging multithreaded and MPI parallel programs as well.
### Small number of MPI ranks
-For debugging small number of MPI ranks, you may execute and debug each
-rank in separate xterm terminal (do not forget the [X
-display](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-system/x-window-and-vnc.html)).
-Using Intel MPI, this may be done in following way:
+For debugging small number of MPI ranks, you may execute and debug each rank in separate xterm terminal (do not forget the [X display](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-and-vnc.md)). Using Intel MPI, this may be done in following way:
+```bash
$ qsub -q qexp -l select=2:ncpus=24 -X -I
qsub: waiting for job 19654.srv11 to start
qsub: job 19655.srv11 ready
$ module load intel impi
$ mpirun -ppn 1 -hostfile $PBS_NODEFILE --enable-x xterm -e idbc ./mympiprog.x
+```
-In this example, we allocate 2 full compute node, run xterm on each node
-and start idb debugger in command line mode, debugging two ranks of
-mympiprog.x application. The xterm will pop up for each rank, with idb
-prompt ready. The example is not limited to use of Intel MPI
+In this example, we allocate 2 full compute node, run xterm on each node and start idb debugger in command line mode, debugging two ranks of mympiprog.x application. The xterm will pop up for each rank, with idb prompt ready. The example is not limited to use of Intel MPI
### Large number of MPI ranks
-Run the idb debugger from within the MPI debug option. This will cause
-the debugger to bind to all ranks and provide aggregated outputs across
-the ranks, pausing execution automatically just after startup. You may
-then set break points and step the execution manually. Using Intel MPI:
+Run the idb debugger from within the MPI debug option. This will cause the debugger to bind to all ranks and provide aggregated outputs across the ranks, pausing execution automatically just after startup. You may then set break points and step the execution manually. Using Intel MPI:
+```bash
$ qsub -q qexp -l select=2:ncpus=24 -X -I
qsub: waiting for job 19654.srv11 to start
qsub: job 19655.srv11 ready
$ module load intel impi
$ mpirun -n 48 -idb ./mympiprog.x
+```
### Debugging multithreaded application
-Run the idb debugger in GUI mode. The menu Parallel contains number of
-tools for debugging multiple threads. One of the most useful tools is
-the **Serialize Execution** tool, which serializes execution of
-concurrent threads for easy orientation and identification of
-concurrency related bugs.
+Run the idb debugger in GUI mode. The menu Parallel contains number of tools for debugging multiple threads. One of the most useful tools is the **Serialize Execution** tool, which serializes execution of concurrent threads for easy orientation and identification of concurrency related bugs.
Further information
-------------------
-
-Exhaustive manual on idb features and usage is published at Intel
-website,
-<https://software.intel.com/sites/products/documentation/doclib/iss/2013/compiler/cpp-lin/>
-
+Exhaustive manual on idb features and usage is published at Intel website, <https://software.intel.com/sites/products/documentation/doclib/iss/2013/compiler/cpp-lin/>
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/intel-suite/intel-inspector.md b/docs.it4i/salomon/software/intel-suite/intel-inspector.md
index cfbde4808..d3a3dcd80 100644
--- a/docs.it4i/salomon/software/intel-suite/intel-inspector.md
+++ b/docs.it4i/salomon/software/intel-suite/intel-inspector.md
@@ -1,63 +1,40 @@
-Intel Inspector
+Intel Inspector
===============
-Intel Inspector is a dynamic memory and threading error checking tool
-for C/C++/Fortran applications. It can detect issues such as memory
-leaks, invalid memory references, uninitalized variables, race
-conditions, deadlocks etc.
+Intel Inspector is a dynamic memory and threading error checking tool for C/C++/Fortran applications. It can detect issues such as memory leaks, invalid memory references, uninitalized variables, race conditions, deadlocks etc.
Installed versions
------------------
-
The following versions are currently available on Salomon as modules:
- --------- |---|---|---------------
- |---|---|
- |2016 Update 1|Inspector/2016_update1|
- --------- |---|---|---------------
+2016 Update 1 - Inspector/2016_update1
Usage
-----
+Your program should be compiled with -g switch to include symbol names. Optimizations can be turned on.
-Your program should be compiled with -g switch to include symbol names.
-Optimizations can be turned on.
-
-Debugging is possible either directly from the GUI, or from command
-line.
+Debugging is possible either directly from the GUI, or from command line.
### GUI mode
To debug from GUI, launch Inspector:
+```bash
$ inspxe-gui &
+```
-Then select menu File -> New -> Project. Choose a directory to
-save project data to. After clicking OK, Project properties window will
-appear, where you can configure path to your binary, launch arguments,
-working directory etc. After clicking OK, the project is ready.
+Then select menu File -> New -> Project. Choose a directory to save project data to. After clicking OK, Project properties window will appear, where you can configure path to your binary, launch arguments, working directory etc. After clicking OK, the project is ready.
-In the main pane, you can start a predefined analysis type or define
-your own. Click Start to start the analysis. Alternatively, you can
-click on Command Line, to see the command line required to run the
-analysis directly from command line.
+In the main pane, you can start a predefined analysis type or define your own. Click Start to start the analysis. Alternatively, you can click on Command Line, to see the command line required to run the analysis directly from command line.
### Batch mode
-Analysis can be also run from command line in batch mode. Batch mode
-analysis is run with command inspxe-cl.
-To obtain the required parameters, either consult the documentation or
-you can configure the analysis in the GUI and then click "Command Line"
-button in the lower right corner to the respective command line.
+Analysis can be also run from command line in batch mode. Batch mode analysis is run with command inspxe-cl. To obtain the required parameters, either consult the documentation or you can configure the analysis in the GUI and then click "Command Line" button in the lower right corner to the respective command line.
-Results obtained from batch mode can be then viewed in the GUI by
-selecting File -> Open -> Result...
+Results obtained from batch mode can be then viewed in the GUI by selecting File -> Open -> Result...
References
----------
-
-1. [Product
- page](https://software.intel.com/en-us/intel-inspector-xe)
-2. [Documentation and Release
- Notes](https://software.intel.com/en-us/intel-inspector-xe-support/documentation)
-3. [Tutorials](https://software.intel.com/en-us/articles/inspectorxe-tutorials)
-
+1. [Product page](https://software.intel.com/en-us/intel-inspector-xe)
+2. [Documentation and Release Notes](https://software.intel.com/en-us/intel-inspector-xe-support/documentation)
+3. [Tutorials](https://software.intel.com/en-us/articles/inspectorxe-tutorials)
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/intel-suite/intel-integrated-performance-primitives.md b/docs.it4i/salomon/software/intel-suite/intel-integrated-performance-primitives.md
index 5ec3bf70b..87d28155a 100644
--- a/docs.it4i/salomon/software/intel-suite/intel-integrated-performance-primitives.md
+++ b/docs.it4i/salomon/software/intel-suite/intel-integrated-performance-primitives.md
@@ -1,30 +1,22 @@
-Intel IPP
+Intel IPP
=========
-
-
Intel Integrated Performance Primitives
----------------------------------------
-
-Intel Integrated Performance Primitives, version 9.0.1, compiled for
-AVX2 vector instructions is available, via module ipp. The IPP is a very
-rich library of highly optimized algorithmic building blocks for media
-and data applications. This includes signal, image and frame processing
-algorithms, such as FFT, FIR, Convolution, Optical Flow, Hough
-transform, Sum, MinMax, as well as cryptographic functions, linear
-algebra functions and many more.
+--------------------------------------
+Intel Integrated Performance Primitives, version 9.0.1, compiled for AVX2 vector instructions is available, via module ipp. The IPP is a very rich library of highly optimized algorithmic building blocks for media and data applications. This includes signal, image and frame processing algorithms, such as FFT, FIR, Convolution, Optical Flow, Hough transform, Sum, MinMax, as well as cryptographic functions, linear algebra functions and many more.
-Check out IPP before implementing own math functions for data
-processing, it is likely already there.
+Check out IPP before implementing own math functions for data processing, it is likely already there.
+```bash
$ module load ipp
+```
-The module sets up environment variables, required for linking and
-running ipp enabled applications.
+The module sets up environment variables, required for linking and running ipp enabled applications.
IPP example
-----------
+```cpp
#include "ipp.h"
#include <stdio.h>
int main(int argc, char* argv[])
@@ -63,32 +55,28 @@ IPP example
return 0;
}
+```
-Â Compile above example, using any compiler and the ipp module.
+Compile above example, using any compiler and the ipp module.
+```bash
$ module load intel
$ module load ipp
$ icc testipp.c -o testipp.x -lippi -lipps -lippcore
+```
-You will need the ipp module loaded to run the ipp enabled executable.
-This may be avoided, by compiling library search paths into the
-executable
+You will need the ipp module loaded to run the ipp enabled executable. This may be avoided, by compiling library search paths into the executable
+```bash
$ module load intel
$ module load ipp
$ icc testipp.c -o testipp.x -Wl,-rpath=$LIBRARY_PATH -lippi -lipps -lippcore
+```
Code samples and documentation
------------------------------
+Intel provides number of [Code Samples for IPP](https://software.intel.com/en-us/articles/code-samples-for-intel-integrated-performance-primitives-library), illustrating use of IPP.
-Intel provides number of [Code Samples for
-IPP](https://software.intel.com/en-us/articles/code-samples-for-intel-integrated-performance-primitives-library),
-illustrating use of IPP.
-
-Read full documentation on IPP [on Intel
-website,](http://software.intel.com/sites/products/search/search.php?q=&x=15&y=6&product=ipp&version=7.1&docos=lin)
-in particular the [IPP Reference
-manual.](http://software.intel.com/sites/products/documentation/doclib/ipp_sa/71/ipp_manual/index.htm)
-
+Read full documentation on IPP [on Intel website,](http://software.intel.com/sites/products/search/search.php?q=&x=15&y=6&product=ipp&version=7.1&docos=lin) in particular the [IPP Reference manual.](http://software.intel.com/sites/products/documentation/doclib/ipp_sa/71/ipp_manual/index.htm)
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/intel-suite/intel-mkl.md b/docs.it4i/salomon/software/intel-suite/intel-mkl.md
index 57711b34a..4eb6c6bf0 100644
--- a/docs.it4i/salomon/software/intel-suite/intel-mkl.md
+++ b/docs.it4i/salomon/software/intel-suite/intel-mkl.md
@@ -1,200 +1,126 @@
-Intel MKL
+Intel MKL
=========
-
Intel Math Kernel Library
-------------------------
+Intel Math Kernel Library (Intel MKL) is a library of math kernel subroutines, extensively threaded and optimized for maximum performance. Intel MKL provides these basic math kernels:
-Intel Math Kernel Library (Intel MKL) is a library of math kernel
-subroutines, extensively threaded and optimized for maximum performance.
-Intel MKL provides these basic math kernels:
+- BLAS (level 1, 2, and 3) and LAPACK linear algebra routines, offering vector, vector-matrix, and matrix-matrix operations.
+- The PARDISO direct sparse solver, an iterative sparse solver, and supporting sparse BLAS (level 1, 2, and 3) routines for solving sparse systems of equations.
+- ScaLAPACK distributed processing linear algebra routines for Linux* and Windows* operating systems, as well as the Basic Linear Algebra Communications Subprograms (BLACS) and the Parallel Basic Linear Algebra Subprograms (PBLAS).
+- Fast Fourier transform (FFT) functions in one, two, or three dimensions with support for mixed radices (not limited to sizes that are powers of 2), as well as distributed versions of these functions.
+- Vector Math Library (VML) routines for optimized mathematical operations on vectors.
+- Vector Statistical Library (VSL) routines, which offer high-performance vectorized random number generators (RNG) for several probability distributions, convolution and correlation routines, and summary statistics functions.
+- Data Fitting Library, which provides capabilities for spline-based approximation of functions, derivatives and integrals of functions, and search.
+- Extended Eigensolver, a shared memory version of an eigensolver based on the Feast Eigenvalue Solver.
--
-
-
-
- BLAS (level 1, 2, and 3) and LAPACK linear algebra routines,
- offering vector, vector-matrix, and matrix-matrix operations.
--
-
-
-
- The PARDISO direct sparse solver, an iterative sparse solver,
- and supporting sparse BLAS (level 1, 2, and 3) routines for solving
- sparse systems of equations.
--
-
-
-
- ScaLAPACK distributed processing linear algebra routines for
- Linux* and Windows* operating systems, as well as the Basic Linear
- Algebra Communications Subprograms (BLACS) and the Parallel Basic
- Linear Algebra Subprograms (PBLAS).
--
-
-
-
- Fast Fourier transform (FFT) functions in one, two, or three
- dimensions with support for mixed radices (not limited to sizes that
- are powers of 2), as well as distributed versions of
- these functions.
--
-
-
-
- Vector Math Library (VML) routines for optimized mathematical
- operations on vectors.
--
-
-
-
- Vector Statistical Library (VSL) routines, which offer
- high-performance vectorized random number generators (RNG) for
- several probability distributions, convolution and correlation
- routines, and summary statistics functions.
--
-
-
-
- Data Fitting Library, which provides capabilities for
- spline-based approximation of functions, derivatives and integrals
- of functions, and search.
-- Extended Eigensolver, a shared memory version of an eigensolver
- based on the Feast Eigenvalue Solver.
-
-For details see the [Intel MKL Reference
-Manual](http://software.intel.com/sites/products/documentation/doclib/mkl_sa/11/mklman/index.htm).
+For details see the [Intel MKL Reference Manual](http://software.intel.com/sites/products/documentation/doclib/mkl_sa/11/mklman/index.htm).
Intel MKL version 11.2.3.187 is available on the cluster
+```bash
$ module load imkl
+```
-The module sets up environment variables, required for linking and
-running mkl enabled applications. The most important variables are the
-$MKLROOT, $CPATH, $LD_LIBRARY_PATH and $MKL_EXAMPLES
+The module sets up environment variables, required for linking and running mkl enabled applications. The most important variables are the $MKLROOT, $CPATH, $LD_LIBRARY_PATH and $MKL_EXAMPLES
-Intel MKL library may be linked using any compiler.
-With intel compiler use -mkl option to link default threaded MKL.
+Intel MKL library may be linked using any compiler. With intel compiler use -mkl option to link default threaded MKL.
### Interfaces
-Intel MKL library provides number of interfaces. The fundamental once
-are the LP64 and ILP64. The Intel MKL ILP64 libraries use the 64-bit
-integer type (necessary for indexing large arrays, with more than
-231^-1 elements), whereas the LP64 libraries index arrays with the
-32-bit integer type.
+Intel MKL library provides number of interfaces. The fundamental once are the LP64 and ILP64. The Intel MKL ILP64 libraries use the 64-bit integer type (necessary for indexing large arrays, with more than 231^-1 elements), whereas the LP64 libraries index arrays with the 32-bit integer type.
- |Interface|Integer type|
- ----- |---|---|-------------------------------------
- |LP64|32-bit, int, integer(kind=4), MPI_INT|
- ILP64 64-bit, long int, integer(kind=8), MPI_INT64
+|Interface|Integer type|
+|---|---|
+|LP64|32-bit, int, integer(kind=4), MPI_INT|
+|ILP64|64-bit, long int, integer(kind=8), MPI_INT64|
### Linking
-Linking Intel MKL libraries may be complex. Intel [mkl link line
-advisor](http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor)
-helps. See also [examples](intel-mkl.html#examples) below.
+Linking Intel MKL libraries may be complex. Intel [mkl link line advisor](http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor) helps. See also [examples](intel-mkl.md#examples) below.
-You will need the mkl module loaded to run the mkl enabled executable.
-This may be avoided, by compiling library search paths into the
-executable. Include rpath on the compile line:
+You will need the mkl module loaded to run the mkl enabled executable. This may be avoided, by compiling library search paths into the executable. Include rpath on the compile line:
+```bash
$ icc .... -Wl,-rpath=$LIBRARY_PATH ...
+```
### Threading
-Advantage in using Intel MKL library is that it brings threaded
-parallelization to applications that are otherwise not parallel.
+Advantage in using Intel MKL library is that it brings threaded parallelization to applications that are otherwise not parallel.
-For this to work, the application must link the threaded MKL library
-(default). Number and behaviour of MKL threads may be controlled via the
-OpenMP environment variables, such as OMP_NUM_THREADS and
-KMP_AFFINITY. MKL_NUM_THREADS takes precedence over OMP_NUM_THREADS
+For this to work, the application must link the threaded MKL library (default). Number and behaviour of MKL threads may be controlled via the OpenMP environment variables, such as OMP_NUM_THREADS and KMP_AFFINITY. MKL_NUM_THREADS takes precedence over OMP_NUM_THREADS
+```bash
$ export OMP_NUM_THREADS=24
$ export KMP_AFFINITY=granularity=fine,compact,1,0
+```
-The application will run with 24 threads with affinity optimized for
-fine grain parallelization.
+The application will run with 24 threads with affinity optimized for fine grain parallelization.
Examples
------------
-
-Number of examples, demonstrating use of the Intel MKL library and its
-linking is available on clusters, in the $MKL_EXAMPLES directory. In
-the examples below, we demonstrate linking Intel MKL to Intel and GNU
-compiled program for multi-threaded matrix multiplication.
+Number of examples, demonstrating use of the Intel MKL library and its linking is available on clusters, in the $MKL_EXAMPLES directory. In the examples below, we demonstrate linking Intel MKL to Intel and GNU compiled program for multi-threaded matrix multiplication.
### Working with examples
+```bash
$ module load intel
$ module load imkl
$ cp -a $MKL_EXAMPLES/cblas /tmp/
$ cd /tmp/cblas
$ make sointel64 function=cblas_dgemm
+```
-In this example, we compile, link and run the cblas_dgemm example,
-demonstrating use of MKL example suite installed on clusters.
+In this example, we compile, link and run the cblas_dgemm example, demonstrating use of MKL example suite installed on clusters.
### Example: MKL and Intel compiler
+```bash
$ module load intel
$ module load imkl
$ cp -a $MKL_EXAMPLES/cblas /tmp/
$ cd /tmp/cblas
- $
+ $
$ icc -w source/cblas_dgemmx.c source/common_func.c -mkl -o cblas_dgemmx.x
$ ./cblas_dgemmx.x data/cblas_dgemmx.d
+```
-In this example, we compile, link and run the cblas_dgemm example,
-demonstrating use of MKL with icc -mkl option. Using the -mkl option is
-equivalent to:
+In this example, we compile, link and run the cblas_dgemm example, demonstrating use of MKL with icc -mkl option. Using the -mkl option is equivalent to:
- $ icc -w source/cblas_dgemmx.c source/common_func.c -o cblas_dgemmx.x
+```bash
+ $ icc -w source/cblas_dgemmx.c source/common_func.c -o cblas_dgemmx.x
-I$MKL_INC_DIR -L$MKL_LIB_DIR -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5
+```
-In this example, we compile and link the cblas_dgemm example, using
-LP64 interface to threaded MKL and Intel OMP threads implementation.
+In this example, we compile and link the cblas_dgemm example, using LP64 interface to threaded MKL and Intel OMP threads implementation.
### Example: Intel MKL and GNU compiler
+```bash
$ module load GCC
$ module load imkl
$ cp -a $MKL_EXAMPLES/cblas /tmp/
$ cd /tmp/cblas
-
- $ gcc -w source/cblas_dgemmx.c source/common_func.c -o cblas_dgemmx.x
+
+ $ gcc -w source/cblas_dgemmx.c source/common_func.c -o cblas_dgemmx.x
-lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_core -lgomp -lm
$ ./cblas_dgemmx.x data/cblas_dgemmx.d
+```
-In this example, we compile, link and run the cblas_dgemm example,
-using LP64 interface to threaded MKL and gnu OMP threads implementation.
+In this example, we compile, link and run the cblas_dgemm example, using LP64 interface to threaded MKL and gnu OMP threads implementation.
MKL and MIC accelerators
------------------------
-
-The Intel MKL is capable to automatically offload the computations o the
-MIC accelerator. See section [Intel Xeon
-Phi](../intel-xeon-phi.html) for details.
+The Intel MKL is capable to automatically offload the computations o the MIC accelerator. See section [Intel Xeon Phi](../intel-xeon-phi.md) for details.
LAPACKE C Interface
-------------------
-
-MKL includes LAPACKE C Interface to LAPACK. For some reason, although
-Intel is the author of LAPACKE, the LAPACKE header files are not present
-in MKL. For this reason, we have prepared
-LAPACKE module, which includes Intel's LAPACKE
-headers from official LAPACK, which you can use to compile code using
-LAPACKE interface against MKL.
+MKL includes LAPACKE C Interface to LAPACK. For some reason, although Intel is the author of LAPACKE, the LAPACKE header files are not present in MKL. For this reason, we have prepared LAPACKE module, which includes Intel's LAPACKE headers from official LAPACK, which you can use to compile code using LAPACKE interface against MKL.
Further reading
---------------
-
-Read more on [Intel
-website](http://software.intel.com/en-us/intel-mkl), in
-particular the [MKL users
-guide](https://software.intel.com/en-us/intel-mkl/documentation/linux).
-
+Read more on [Intel website](http://software.intel.com/en-us/intel-mkl), in particular the [MKL users guide](https://software.intel.com/en-us/intel-mkl/documentation/linux).
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/intel-suite/intel-parallel-studio-introduction.md b/docs.it4i/salomon/software/intel-suite/intel-parallel-studio-introduction.md
index d77b2d908..429c1d252 100644
--- a/docs.it4i/salomon/software/intel-suite/intel-parallel-studio-introduction.md
+++ b/docs.it4i/salomon/software/intel-suite/intel-parallel-studio-introduction.md
@@ -1,95 +1,71 @@
-Intel Parallel Studio
+Intel Parallel Studio
=====================
-
+The Salomon cluster provides following elements of the Intel Parallel Studio XE
-The Salomon cluster provides following elements of the Intel Parallel
-Studio XE
-
- Intel Parallel Studio XE
- -------------------------------------------------
- Intel Compilers
- Intel Debugger
- Intel MKL Library
- Intel Integrated Performance Primitives Library
- Intel Threading Building Blocks Library
- Intel Trace Analyzer and Collector
- Intel Advisor
- Intel Inspector
+|Intel Parallel Studio XE|
+| -------------------------------------------------|
+|Intel Compilers|
+|Intel Debugger|
+|Intel MKL Library|
+|Intel Integrated Performance Primitives Library|
+|Intel Threading Building Blocks Library|
+|Intel Trace Analyzer and Collector|
+|Intel Advisor|
+|Intel Inspector|
Intel compilers
---------------
-The Intel compilers version 131.3 are available, via module
-iccifort/2013.5.192-GCC-4.8.3. The compilers include the icc C and C++
-compiler and the ifort fortran 77/90/95 compiler.
+The Intel compilers version 131.3 are available, via module iccifort/2013.5.192-GCC-4.8.3. The compilers include the icc C and C++ compiler and the ifort fortran 77/90/95 compiler.
+```bash
$ module load intel
$ icc -v
$ ifort -v
+```
-Read more at the [Intel Compilers](intel-compilers.html)
-page.
+Read more at the [Intel Compilers](intel-compilers.md) page.
Intel debugger
--------------
-
IDB is no longer available since Parallel Studio 2015.
-Â The intel debugger version 13.0 is available, via module intel. The
-debugger works for applications compiled with C and C++ compiler and the
-ifort fortran 77/90/95 compiler. The debugger provides java GUI
-environment. Use [X
-display](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-system/x-window-and-vnc.html)
-for running the GUI.
+The intel debugger version 13.0 is available, via module intel. The debugger works for applications compiled with C and C++ compiler and the ifort fortran 77/90/95 compiler. The debugger provides java GUI environment. Use [X display](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-and-vnc.md) for running the GUI.
+```bash
$ module load intel
$ idb
+```
-Read more at the [Intel Debugger](intel-debugger.html)
-page.
+Read more at the [Intel Debugger](intel-debugger.md) page.
Intel Math Kernel Library
-------------------------
+ Intel Math Kernel Library (Intel MKL) is a library of math kernel subroutines, extensively threaded and optimized for maximum performance. Intel MKL unites and provides these basic components: BLAS, LAPACK, ScaLapack, PARDISO, FFT, VML, VSL, Data fitting, Feast Eigensolver and many more.
-Intel Math Kernel Library (Intel MKL) is a library of math kernel
-subroutines, extensively threaded and optimized for maximum performance.
-Intel MKL unites and provides these basic components: BLAS, LAPACK,
-ScaLapack, PARDISO, FFT, VML, VSL, Data fitting, Feast Eigensolver and
-many more.
-
+```bash
$ module load imkl
+```
-Read more at the [Intel MKL](intel-mkl.html) page.
+Read more at the [Intel MKL](intel-mkl.md) page.
Intel Integrated Performance Primitives
---------------------------------------
+Intel Integrated Performance Primitives, version 7.1.1, compiled for AVX is available, via module ipp. The IPP is a library of highly optimized algorithmic building blocks for media and data applications. This includes signal, image and frame processing algorithms, such as FFT, FIR, Convolution, Optical Flow, Hough transform, Sum, MinMax and many more.
-Intel Integrated Performance Primitives, version 7.1.1, compiled for AVX
-is available, via module ipp. The IPP is a library of highly optimized
-algorithmic building blocks for media and data applications. This
-includes signal, image and frame processing algorithms, such as FFT,
-FIR, Convolution, Optical Flow, Hough transform, Sum, MinMax and many
-more.
-
+```bash
$ module load ipp
+```
-Read more at the [Intel
-IPP](intel-integrated-performance-primitives.html) page.
+Read more at the [Intel IPP](intel-integrated-performance-primitives.md) page.
Intel Threading Building Blocks
-------------------------------
+Intel Threading Building Blocks (Intel TBB) is a library that supports scalable parallel programming using standard ISO C++ code. It does not require special languages or compilers. It is designed to promote scalable data parallel programming. Additionally, it fully supports nested parallelism, so you can build larger parallel components from smaller parallel components. To use the library, you specify tasks, not threads, and let the library map tasks onto threads in an efficient manner.
-Intel Threading Building Blocks (Intel TBB) is a library that supports
-scalable parallel programming using standard ISO C++ code. It does not
-require special languages or compilers. It is designed to promote
-scalable data parallel programming. Additionally, it fully supports
-nested parallelism, so you can build larger parallel components from
-smaller parallel components. To use the library, you specify tasks, not
-threads, and let the library map tasks onto threads in an efficient
-manner.
-
+```bash
$ module load tbb
+```
-Read more at the [Intel TBB](intel-tbb.html) page.
-
+Read more at the [Intel TBB](intel-tbb.md) page.
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/intel-suite/intel-tbb.md b/docs.it4i/salomon/software/intel-suite/intel-tbb.md
index 8fea59c78..820a4fcdd 100644
--- a/docs.it4i/salomon/software/intel-suite/intel-tbb.md
+++ b/docs.it4i/salomon/software/intel-suite/intel-tbb.md
@@ -1,54 +1,41 @@
-Intel TBB
+Intel TBB
=========
-
-
Intel Threading Building Blocks
--------------------------------
-
-Intel Threading Building Blocks (Intel TBB) is a library that supports
-scalable parallel programming using standard ISO C++ code. It does not
-require special languages or compilers. To use the library, you specify
-tasks, not threads, and let the library map tasks onto threads in an
-efficient manner. The tasks are executed by a runtime scheduler and may
-be offloaded to [MIC
-accelerator](../intel-xeon-phi.html).
+------------------------------
+Intel Threading Building Blocks (Intel TBB) is a library that supports scalable parallel programming using standard ISO C++ code. It does not require special languages or compilers. To use the library, you specify tasks, not threads, and let the library map tasks onto threads in an efficient manner. The tasks are executed by a runtime scheduler and may be offloaded to [MIC accelerator](../intel-xeon-phi.md).
Intel TBB version 4.3.5.187 is available on the cluster.
+```bash
$ module load tbb
+```
-The module sets up environment variables, required for linking and
-running tbb enabled applications.
+The module sets up environment variables, required for linking and running tbb enabled applications.
Link the tbb library, using -ltbb
Examples
--------
+Number of examples, demonstrating use of TBB and its built-in scheduler is available on Anselm, in the $TBB_EXAMPLES directory.
-Number of examples, demonstrating use of TBB and its built-in schedulerÂ
-is available on Anselm, in the $TBB_EXAMPLES directory.
-
+```bash
$ module load intel
$ module load tbb
$ cp -a $TBB_EXAMPLES/common $TBB_EXAMPLES/parallel_reduce /tmp/
$ cd /tmp/parallel_reduce/primes
$ icc -O2 -DNDEBUG -o primes.x main.cpp primes.cpp -ltbb
$ ./primes.x
+```
-In this example, we compile, link and run the primes example,
-demonstrating use of parallel task-based reduce in computation of prime
-numbers.
+In this example, we compile, link and run the primes example, demonstrating use of parallel task-based reduce in computation of prime numbers.
-You will need the tbb module loaded to run the tbb enabled executable.
-This may be avoided, by compiling library search paths into the
-executable.
+You will need the tbb module loaded to run the tbb enabled executable. This may be avoided, by compiling library search paths into the executable.
+```bash
$ icc -O2 -o primes.x main.cpp primes.cpp -Wl,-rpath=$LIBRARY_PATH -ltbb
+```
Further reading
---------------
-
-Read more on Intel website,
-<http://software.intel.com/sites/products/documentation/doclib/tbb_sa/help/index.htm>
-
+Read more on Intel website, <http://software.intel.com/sites/products/documentation/doclib/tbb_sa/help/index.htm>
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/intel-suite/intel-trace-analyzer-and-collector.md b/docs.it4i/salomon/software/intel-suite/intel-trace-analyzer-and-collector.md
index 510aae8b6..c9123d6cd 100644
--- a/docs.it4i/salomon/software/intel-suite/intel-trace-analyzer-and-collector.md
+++ b/docs.it4i/salomon/software/intel-suite/intel-trace-analyzer-and-collector.md
@@ -1,42 +1,33 @@
-Intel Trace Analyzer and Collector
+Intel Trace Analyzer and Collector
==================================
-Intel Trace Analyzer and Collector (ITAC) is a tool to collect and
-graphicaly analyze behaviour of MPI applications. It helps you to
-analyze communication patterns of your application, identify hotspots,
-perform correctnes checking (identify deadlocks, data corruption etc),
-simulate how your application would run on a different interconnect.Â
+Intel Trace Analyzer and Collector (ITAC) is a tool to collect and graphicaly analyze behaviour of MPI applications. It helps you to analyze communication patterns of your application, identify hotspots, perform correctnes checking (identify deadlocks, data corruption etc), simulate how your application would run on a different interconnect.
-ITAC is a offline analysis tool - first you run your application to
-collect a trace file, then you can open the trace in a GUI analyzer to
-view it.
+ITAC is a offline analysis tool - first you run your application to collect a trace file, then you can open the trace in a GUI analyzer to view it.
Installed version
-----------------
-
-Currently on Salomon is version 9.1.2.024 available as moduleÂ
-itac/9.1.2.024
+Currently on Salomon is version 9.1.2.024 available as module itac/9.1.2.024
Collecting traces
-----------------
+ITAC can collect traces from applications that are using Intel MPI. To generate a trace, simply add -trace option to your mpirun command :
-ITAC can collect traces from applications that are using Intel MPI. To
-generate a trace, simply add -trace option to your mpirun command :
-
+```bash
$ module load itac/9.1.2.024
$ mpirun -trace myapp
+```
The trace will be saved in file myapp.stf in the current directory.
Viewing traces
--------------
+To view and analyze the trace, open ITAC GUI in a [graphical environment](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-and-vnc.md):
-To view and analyze the trace, open ITAC GUI in a [graphical
-environment](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-system/x-window-and-vnc.html)
-:
-
+```bash
$ module load itac/9.1.2.024
$ traceanalyzer
+```
The GUI will launch and you can open the produced *.stf file.
@@ -46,9 +37,6 @@ Please refer to Intel documenation about usage of the GUI tool.
References
----------
-
-1. [Getting Started with Intel® Trace Analyzer and
- Collector](https://software.intel.com/en-us/get-started-with-itac-for-linux)
-2. [Intel® Trace Analyzer and Collector -
- Documentation](http://Intel®%20Trace%20Analyzer%20and%20Collector%20-%20Documentation)
+1. [Getting Started with Intel® Trace Analyzer and Collector](https://software.intel.com/en-us/get-started-with-itac-for-linux)
+2. [Intel® Trace Analyzer and Collector - Documentation](http://Intel®%20Trace%20Analyzer%20and%20Collector%20-%20Documentation)
diff --git a/docs.it4i/salomon/software/intel-xeon-phi.md b/docs.it4i/salomon/software/intel-xeon-phi.md
index 55e70e757..d9017bf81 100644
--- a/docs.it4i/salomon/software/intel-xeon-phi.md
+++ b/docs.it4i/salomon/software/intel-xeon-phi.md
@@ -1,165 +1,112 @@
-Intel Xeon Phi
+Intel Xeon Phi
==============
-A guide to Intel Xeon Phi usage
+##A guide to Intel Xeon Phi usage
-
-
-Intel Xeon Phi accelerator can be programmed in several modes. The
-default mode on the cluster is offload mode, but all modes described in
-this document are supported.
+Intel Xeon Phi can be programmed in several modes. The default mode on Anselm is offload mode, but all modes described in this document are supported.
Intel Utilities for Xeon Phi
----------------------------
+To get access to a compute node with Intel Xeon Phi accelerator, use the PBS interactive session
-To get access to a compute node with Intel Xeon Phi accelerator, use the
-PBS interactive session
-
- $ qsub -I -q qprod -l select=1:ncpus=24:accelerator=True:naccelerators=2:accelerator_model=phi7120 -A NONE-0-0
+```bash
+ $ qsub -I -q qmic -A NONE-0-0
+```
-To set up the environment module "intel" has to be loaded, without
-specifying the version, default version is loaded (at time of writing
-this, it's 2015b)
+To set up the environment module "Intel" has to be loaded
- $ module load intel
+```bash
+ $ module load intel/13.5.192
+```
-Information about the hardware can be obtained by running
-the micinfo program on the host.
+Information about the hardware can be obtained by running the micinfo program on the host.
+```bash
$ /usr/bin/micinfo
+```
-The output of the "micinfo" utility executed on one of the cluster node
-is as follows. (note: to get PCIe related details the command has to be
-run with root privileges)
+The output of the "micinfo" utility executed on one of the Anselm node is as follows. (note: to get PCIe related details the command has to be run with root privileges)
+```bash
MicInfo Utility Log
- Created Mon Aug 17 13:55:59 2015
- System Info
- HOST OS : Linux
- OS Version : 2.6.32-504.16.2.el6.x86_64
- Driver Version : 3.4.1-1
- MPSS Version : 3.4.1
- Host Physical Memory : 131930 MB
+ Created Mon Jul 22 00:23:50 2013
+
+ Â Â Â Â Â Â Â System Info
+ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â HOST OSÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â : Linux
+                OS Version             : 2.6.32-279.5.2.bl6.Bull.33.x86_64
+                Driver Version         : 6720-15
+                MPSS Version           : 2.1.6720-15
+                Host Physical Memory   : 98843 MB
Device No: 0, Device Name: mic0
- Version
- Flash Version : 2.1.02.0390
- SMC Firmware Version : 1.16.5078
- SMC Boot Loader Version : 1.8.4326
- uOS Version : 2.6.38.8+mpss3.4.1
- Device Serial Number : ADKC44601414
-
- Board
- Vendor ID : 0x8086
- Device ID : 0x225c
- Subsystem ID : 0x7d95
- Coprocessor Stepping ID : 2
- PCIe Width : x16
- PCIe Speed : 5 GT/s
- PCIe Max payload size : 256 bytes
- PCIe Max read req size : 512 bytes
- Coprocessor Model : 0x01
- Coprocessor Model Ext : 0x00
- Coprocessor Type : 0x00
- Coprocessor Family : 0x0b
- Coprocessor Family Ext : 0x00
- Coprocessor Stepping : C0
- Board SKU : C0PRQ-7120 P/A/X/D
- ECC Mode : Enabled
- SMC HW Revision : Product 300W Passive CS
-
- Cores
- Total No of Active Cores : 61
- Voltage : 1007000 uV
- Frequency : 1238095 kHz
-
- Thermal
- Fan Speed Control : N/A
- Fan RPM : N/A
- Fan PWM : N/A
- Die Temp : 60 C
-
- GDDR
- GDDR Vendor : Samsung
- GDDR Version : 0x6
- GDDR Density : 4096 Mb
- GDDR Size : 15872 MB
- GDDR Technology : GDDR5
- GDDR Speed : 5.500000 GT/s
- GDDR Frequency : 2750000 kHz
- GDDR Voltage : 1501000 uV
-
- Device No: 1, Device Name: mic1
-
- Version
- Flash Version : 2.1.02.0390
- SMC Firmware Version : 1.16.5078
- SMC Boot Loader Version : 1.8.4326
- uOS Version : 2.6.38.8+mpss3.4.1
- Device Serial Number : ADKC44500454
-
- Board
- Vendor ID : 0x8086
- Device ID : 0x225c
- Subsystem ID : 0x7d95
- Coprocessor Stepping ID : 2
- PCIe Width : x16
- PCIe Speed : 5 GT/s
- PCIe Max payload size : 256 bytes
- PCIe Max read req size : 512 bytes
- Coprocessor Model : 0x01
- Coprocessor Model Ext : 0x00
- Coprocessor Type : 0x00
- Coprocessor Family : 0x0b
- Coprocessor Family Ext : 0x00
- Coprocessor Stepping : C0
- Board SKU : C0PRQ-7120 P/A/X/D
- ECC Mode : Enabled
- SMC HW Revision : Product 300W Passive CS
-
- Cores
- Total No of Active Cores : 61
- Voltage : 998000 uV
- Frequency : 1238095 kHz
-
- Thermal
- Fan Speed Control : N/A
- Fan RPM : N/A
- Fan PWM : N/A
- Die Temp : 59 C
-
- GDDR
- GDDR Vendor : Samsung
- GDDR Version : 0x6
- GDDR Density : 4096 Mb
- GDDR Size : 15872 MB
- GDDR Technology : GDDR5
- GDDR Speed : 5.500000 GT/s
- GDDR Frequency : 2750000 kHz
- GDDR Voltage : 1501000 uV
+ Â Â Â Â Â Â Â Version
+                Flash Version           : 2.1.03.0386
+                SMC Firmware Version    : 1.15.4830
+                SMC Boot Loader Version : 1.8.4326
+                uOS Version             : 2.6.38.8-g2593b11
+                Device Serial Number    : ADKC30102482
+
+ Â Â Â Â Â Â Â Board
+ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Vendor IDÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â : 0x8086
+ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Device IDÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â : 0x2250
+ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Subsystem IDÂ Â Â Â Â Â Â Â Â Â Â Â : 0x2500
+ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Coprocessor Stepping IDÂ : 3
+                PCIe Width              : x16
+                PCIe Speed              : 5 GT/s
+                PCIe Max payload size   : 256 bytes
+                PCIe Max read req size  : 512 bytes
+                Coprocessor Model       : 0x01
+                Coprocessor Model Ext   : 0x00
+                Coprocessor Type        : 0x00
+                Coprocessor Family      : 0x0b
+                Coprocessor Family Ext  : 0x00
+                Coprocessor Stepping    : B1
+ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Board SKUÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â : B1PRQ-5110P/5120D
+                ECC Mode                : Enabled
+                SMC HW Revision         : Product 225W Passive CS
+
+ Â Â Â Â Â Â Â Cores
+ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Total No of Active Cores : 60
+                Voltage                 : 1032000 uV
+                Frequency               : 1052631 kHz
+
+ Â Â Â Â Â Â Â Thermal
+                Fan Speed Control       : N/A
+ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Fan RPMÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â : N/A
+ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Fan PWMÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â : N/A
+                Die Temp                : 49 C
+
+ Â Â Â Â Â Â Â GDDR
+                GDDR Vendor             : Elpida
+                GDDR Version            : 0x1
+                GDDR Density            : 2048 Mb
+                GDDR Size               : 7936 MB
+                GDDR Technology         : GDDR5
+                GDDR Speed              : 5.000000 GT/s
+                GDDR Frequency          : 2500000 kHz
+                GDDR Voltage            : 1501000 uV
+```
Offload Mode
------------
+To compile a code for Intel Xeon Phi a MPSS stack has to be installed on the machine where compilation is executed. Currently the MPSS stack is only installed on compute nodes equipped with accelerators.
-To compile a code for Intel Xeon Phi a MPSS stack has to be installed on
-the machine where compilation is executed. Currently the MPSS stack is
-only installed on compute nodes equipped with accelerators.
+```bash
+ $ qsub -I -q qmic -A NONE-0-0
+ $ module load intel/13.5.192
+```
- $ qsub -I -q qprod -l select=1:ncpus=24:accelerator=True:naccelerators=2:accelerator_model=phi7120 -A NONE-0-0
- $ module load intel
-
-For debugging purposes it is also recommended to set environment
-variable "OFFLOAD_REPORT". Value can be set from 0 to 3, where higher
-number means more debugging information.
+For debugging purposes it is also recommended to set environment variable "OFFLOAD_REPORT". Value can be set from 0 to 3, where higher number means more debugging information.
+```bash
export OFFLOAD_REPORT=3
+```
-A very basic example of code that employs offload programming technique
-is shown in the next listing. Please note that this code is sequential
-and utilizes only single core of the accelerator.
+A very basic example of code that employs offload programming technique is shown in the next listing. Please note that this code is sequential and utilizes only single core of the accelerator.
+```bash
$ vim source-offload.cpp
#include <iostream>
@@ -177,22 +124,26 @@ and utilizes only single core of the accelerator.
   result /= niter;
   std::cout << "Pi ~ " << result << 'n';
}
+```
To compile a code using Intel compiler run
+```bash
$ icc source-offload.cpp -o bin-offload
+```
To execute the code, run the following command on the host
+```bash
./bin-offload
+```
### Parallelization in Offload Mode Using OpenMP
-One way of paralelization a code for Xeon Phi is using OpenMP
-directives. The following example shows code for parallel vector
-addition.Â
+One way of paralelization a code for Xeon Phi is using OpenMP directives. The following example shows code for parallel vector addition.
- $ vim ./vect-add
+```bash
+ $ vim ./vect-add
#include <stdio.h>
@@ -267,338 +218,257 @@ addition.Â
 compare(res, res_cpu, SIZE);
}
+```
-During the compilation Intel compiler shows which loops have been
-vectorized in both host and accelerator. This can be enabled with
-compiler option "-vec-report2". To compile and execute the code run
+During the compilation Intel compiler shows which loops have been vectorized in both host and accelerator. This can be enabled with compiler option "-vec-report2". To compile and execute the code run
+```bash
$ icc vect-add.c -openmp_report2 -vec-report2 -o vect-add
- $ ./vect-add
+ $ ./vect-add
+```
Some interesting compiler flags useful not only for code debugging are:
-Debugging
-Â openmp_report[0|1|2] - controls the compiler based vectorization
-diagnostic level
-Â vec-report[0|1|2] - controls the OpenMP parallelizer diagnostic
-level
+>Debugging
+Â openmp_report[0|1|2] - controls the compiler based vectorization diagnostic level
+Â vec-report[0|1|2] - controls the OpenMP parallelizer diagnostic level
-Performance ooptimization
-Â xhost - FOR HOST ONLY - to generate AVX (Advanced Vector Extensions)
-instructions.
+>Performance ooptimization
+Â xhost - FOR HOST ONLY - to generate AVX (Advanced Vector Extensions) instructions.
Automatic Offload using Intel MKL Library
-----------------------------------------
+Intel MKL includes an Automatic Offload (AO) feature that enables computationally intensive MKL functions called in user code to benefit from attached Intel Xeon Phi coprocessors automatically and transparently.
-Intel MKL includes an Automatic Offload (AO) feature that enables
-computationally intensive MKL functions called in user code to benefit
-from attached Intel Xeon Phi coprocessors automatically and
-transparently.
-
-Behavioural of automatic offload mode is controlled by functions called
-within the program or by environmental variables. Complete list of
-controls is listed [
-here](http://software.intel.com/sites/products/documentation/doclib/mkl_sa/11/mkl_userguide_lnx/GUID-3DC4FC7D-A1E4-423D-9C0C-06AB265FFA86.htm).
+Behavioral of automatic offload mode is controlled by functions called within the program or by environmental variables. Complete list of controls is listed [ here](http://software.intel.com/sites/products/documentation/doclib/mkl_sa/11/mkl_userguide_lnx/GUID-3DC4FC7D-A1E4-423D-9C0C-06AB265FFA86.htm).
-The Automatic Offload may be enabled by either an MKL function call
-within the code:
+The Automatic Offload may be enabled by either an MKL function call within the code:
+```cpp
mkl_mic_enable();
+```
or by setting environment variable
+```bash
$ export MKL_MIC_ENABLE=1
+```
-To get more information about automatic offload please refer to "[Using
-Intel® MKL Automatic Offload on Intel ® Xeon Phi™
-Coprocessors](http://software.intel.com/sites/default/files/11MIC42_How_to_Use_MKL_Automatic_Offload_0.pdf)"
-white paper or [ Intel MKL
-documentation](https://software.intel.com/en-us/articles/intel-math-kernel-library-documentation).
+To get more information about automatic offload please refer to "[Using Intel® MKL Automatic Offload on Intel ® Xeon Phi™ Coprocessors](http://software.intel.com/sites/default/files/11MIC42_How_to_Use_MKL_Automatic_Offload_0.pdf)" white paper or [ Intel MKL documentation](https://software.intel.com/en-us/articles/intel-math-kernel-library-documentation).
-### Automatic offload example #1
+### Automatic offload example
-Following example show how to automatically offload an SGEMM (single
-precision - g dir="auto">eneral matrix multiply) function to
-MIC coprocessor.
+At first get an interactive PBS session on a node with MIC accelerator and load "intel" module that automatically loads "mkl" module as well.
-At first get an interactive PBS session on a node with MIC accelerator
-and load "intel" module that automatically loads "mkl" module as well.
-
- $ qsub -I -q qprod -l select=1:ncpus=24:accelerator=True:naccelerators=2:accelerator_model=phi7120 -A NONE-0-0
+```bash
+ $ qsub -I -q qmic -A OPEN-0-0 -l select=1:ncpus=16
$ module load intel
+```
-Â The code can be copied to a file and compiled without any necessary
-modification.Â
+Following example show how to automatically offload an SGEMM (single precision - g dir="auto">eneral matrix multiply) function to MIC coprocessor. The code can be copied to a file and compiled without any necessary modification.
+```bash
$ vim sgemm-ao-short.c
-`
-#include <stdio.h>
-#include <stdlib.h>
-#include <malloc.h>
-#include <stdint.h>
+ #include <stdio.h>
+ #include <stdlib.h>
+ #include <malloc.h>
+ #include <stdint.h>
-#include "mkl.h"
+ #include "mkl.h"
-int main(int argc, char **argv)
-{
-Â Â Â Â Â Â Â float *A, *B, *C; /* Matrices */
+ int main(int argc, char **argv)
+ {
+ Â Â Â Â Â Â Â float *A, *B, *C; /* Matrices */
-Â Â Â Â Â Â Â MKL_INT N = 2560; /* Matrix dimensions */
-Â Â Â Â Â Â Â MKL_INT LD = N; /* Leading dimension */
-Â Â Â Â Â Â Â int matrix_bytes; /* Matrix size in bytes */
-Â Â Â Â Â Â Â int matrix_elements; /* Matrix size in elements */
+ Â Â Â Â Â Â Â MKL_INT N = 2560; /* Matrix dimensions */
+ Â Â Â Â Â Â Â MKL_INT LD = N; /* Leading dimension */
+ Â Â Â Â Â Â Â int matrix_bytes; /* Matrix size in bytes */
+ Â Â Â Â Â Â Â int matrix_elements; /* Matrix size in elements */
-Â Â Â Â Â Â Â float alpha = 1.0, beta = 1.0; /* Scaling factors */
-Â Â Â Â Â Â Â char transa = 'N', transb = 'N'; /* Transposition options */
+ Â Â Â Â Â Â Â float alpha = 1.0, beta = 1.0; /* Scaling factors */
+ Â Â Â Â Â Â Â char transa = 'N', transb = 'N'; /* Transposition options */
-Â Â Â Â Â Â Â int i, j; /* Counters */
+ Â Â Â Â Â Â Â int i, j; /* Counters */
-Â Â Â Â Â Â Â matrix_elements = N * N;
-Â Â Â Â Â Â Â matrix_bytes = sizeof(float) * matrix_elements;
+ Â Â Â Â Â Â Â matrix_elements = N * N;
+ Â Â Â Â Â Â Â matrix_bytes = sizeof(float) * matrix_elements;
-Â Â Â Â Â Â Â /* Allocate the matrices */
-Â Â Â Â Â Â Â A = malloc(matrix_bytes); B = malloc(matrix_bytes); C = malloc(matrix_bytes);
+ Â Â Â Â Â Â Â /* Allocate the matrices */
+ Â Â Â Â Â Â Â A = malloc(matrix_bytes); B = malloc(matrix_bytes); C = malloc(matrix_bytes);
-Â Â Â Â Â Â Â /* Initialize the matrices */
-Â Â Â Â Â Â Â for (i = 0; i < matrix_elements; i++) {
-Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â A[i] = 1.0; B[i] = 2.0; C[i] = 0.0;
-Â Â Â Â Â Â Â }
+ Â Â Â Â Â Â Â /* Initialize the matrices */
+ Â Â Â Â Â Â Â for (i = 0; i < matrix_elements; i++) {
+ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â A[i] = 1.0; B[i] = 2.0; C[i] = 0.0;
+ Â Â Â Â Â Â Â }
-Â Â Â Â Â Â Â printf("Computing SGEMM on the hostn");
-Â Â Â Â Â Â Â sgemm(&transa, &transb, &N, &N, &N, &alpha, A, &N, B, &N, &beta, C, &N);
+ Â Â Â Â Â Â Â printf("Computing SGEMM on the hostn");
+ Â Â Â Â Â Â Â sgemm(&transa, &transb, &N, &N, &N, &alpha, A, &N, B, &N, &beta, C, &N);
-Â Â Â Â Â Â Â printf("Enabling Automatic Offloadn");
-Â Â Â Â Â Â Â /* Alternatively, set environment variable MKL_MIC_ENABLE=1 */
-Â Â Â Â Â Â Â mkl_mic_enable();
-Â Â Â Â Â Â Â
-Â Â Â Â Â Â Â int ndevices = mkl_mic_get_device_count(); /* Number of MIC devices */
-       printf("Automatic Offload enabled: %d MIC devices presentn",  ndevices);
+ Â Â Â Â Â Â Â printf("Enabling Automatic Offloadn");
+ Â Â Â Â Â Â Â /* Alternatively, set environment variable MKL_MIC_ENABLE=1 */
+ Â Â Â Â Â Â Â mkl_mic_enable();
-Â Â Â Â Â Â Â printf("Computing SGEMM with automatic workdivisionn");
-Â Â Â Â Â Â Â sgemm(&transa, &transb, &N, &N, &N, &alpha, A, &N, B, &N, &beta, C, &N);
+ Â Â Â Â Â Â Â int ndevices = mkl_mic_get_device_count(); /* Number of MIC devices */
+        printf("Automatic Offload enabled: %d MIC devices presentn",  ndevices);
-Â Â Â Â Â Â Â /* Free the matrix memory */
-Â Â Â Â Â Â Â free(A); free(B); free(C);
+ Â Â Â Â Â Â Â printf("Computing SGEMM with automatic workdivisionn");
+ Â Â Â Â Â Â Â sgemm(&transa, &transb, &N, &N, &N, &alpha, A, &N, B, &N, &beta, C, &N);
-Â Â Â Â Â Â Â printf("Donen");
+ Â Â Â Â Â Â Â /* Free the matrix memory */
+ Â Â Â Â Â Â Â free(A); free(B); free(C);
-Â Â Â return 0;
-}
-`
+ Â Â Â Â Â Â Â printf("Donen");
-Please note: This example is simplified version of an example from MKL.
-The expanded version can be found here:
-$MKL_EXAMPLES/mic_ao/blasc/source/sgemm.c**
+ Â Â Â return 0;
+ }
+```
+
+>Please note: This example is simplified version of an example from MKL. The expanded version can be found here: **$MKL_EXAMPLES/mic_ao/blasc/source/sgemm.c**
To compile a code using Intel compiler use:
+```bash
$ icc -mkl sgemm-ao-short.c -o sgemm
+```
-For debugging purposes enable the offload report to see more information
-about automatic offloading.
+For debugging purposes enable the offload report to see more information about automatic offloading.
+```bash
$ export OFFLOAD_REPORT=2
+```
-The output of a code should look similar to following listing, where
-lines starting with [MKL] are generated by offload reporting:
+The output of a code should look similar to following listing, where lines starting with [MKL] are generated by offload reporting:
- [user@r31u03n799 ~]$ ./sgemm
+```bash
Computing SGEMM on the host
Enabling Automatic Offload
- Automatic Offload enabled: 2 MIC devices present
+ Automatic Offload enabled: 1 MIC devices present
Computing SGEMM with automatic workdivision
- [MKL] [MIC --] [AO Function]Â Â Â SGEMM
- [MKL] [MIC --] [AO SGEMM Workdivision]Â Â Â 0.44 0.28 0.28
- [MKL] [MIC 00] [AO SGEMM CPU Time]Â Â Â 0.252427 seconds
- [MKL] [MIC 00] [AO SGEMM MIC Time]Â Â Â 0.091001 seconds
- [MKL] [MIC 00] [AO SGEMM CPU->MIC Data]Â Â Â 34078720 bytes
- [MKL] [MIC 00] [AO SGEMM MIC->CPU Data]Â Â Â 7864320 bytes
- [MKL] [MIC 01] [AO SGEMM CPU Time]Â Â Â 0.252427 seconds
- [MKL] [MIC 01] [AO SGEMM MIC Time]Â Â Â 0.094758 seconds
- [MKL] [MIC 01] [AO SGEMM CPU->MIC Data]Â Â Â 34078720 bytes
- [MKL] [MIC 01] [AO SGEMM MIC->CPU Data]Â Â Â 7864320 bytes
+ [MKL] [MIC --] [AO Function]Â Â Â SGEMM
+ [MKL] [MIC --] [AO SGEMM Workdivision]Â 0.00 1.00
+ [MKL] [MIC 00] [AO SGEMM CPU Time]Â Â Â Â Â 0.463351 seconds
+ [MKL] [MIC 00] [AO SGEMM MIC Time]Â Â Â Â Â 0.179608 seconds
+ [MKL] [MIC 00] [AO SGEMM CPU->MIC Data] 52428800 bytes
+ [MKL] [MIC 00] [AO SGEMM MIC->CPU Data] 26214400 bytes
Done
-
-Behavioral of automatic offload mode is controlled by functions called
-within the program or by environmental variables. Complete list of
-controls is listed [
-here](http://software.intel.com/sites/products/documentation/doclib/mkl_sa/11/mkl_userguide_lnx/GUID-3DC4FC7D-A1E4-423D-9C0C-06AB265FFA86.htm).
-
-To get more information about automatic offload please refer to "[Using
-Intel® MKL Automatic Offload on Intel ® Xeon Phi™
-Coprocessors](http://software.intel.com/sites/default/files/11MIC42_How_to_Use_MKL_Automatic_Offload_0.pdf)"
-white paper or [ Intel MKL
-documentation](https://software.intel.com/en-us/articles/intel-math-kernel-library-documentation).
-
-### Automatic offload example #2
-
-In this example, we will demonstrate automatic offload control via an
-environment vatiable MKL_MIC_ENABLE. The function DGEMM will be
-offloaded.
-
-At first get an interactive PBS session on a node with MIC accelerator.
-
- $ qsub -I -q qprod -l select=1:ncpus=24:accelerator=True:naccelerators=2:accelerator_model=phi7120 -A NONE-0-0
-
-Once in, we enable the offload and run the Octave software. In octave,
-we generate two large random matrices and let them multiply together.
-
- $ export MKL_MIC_ENABLE=1
- $ export OFFLOAD_REPORT=2
- $ module load Octave/3.8.2-intel-2015b
-
- $ octave -q
- octave:1> A=rand(10000);
- octave:2> B=rand(10000);
- octave:3> C=A*B;
- [MKL] [MIC --] [AO Function]Â Â Â DGEMM
- [MKL] [MIC --] [AO DGEMM Workdivision]Â Â Â 0.14 0.43 0.43
- [MKL] [MIC 00] [AO DGEMM CPU Time]Â Â Â 3.814714 seconds
- [MKL] [MIC 00] [AO DGEMM MIC Time]Â Â Â 2.781595 seconds
- [MKL] [MIC 00] [AO DGEMM CPU->MIC Data]Â Â Â 1145600000 bytes
- [MKL] [MIC 00] [AO DGEMM MIC->CPU Data]Â Â Â 1382400000 bytes
- [MKL] [MIC 01] [AO DGEMM CPU Time]Â Â Â 3.814714 seconds
- [MKL] [MIC 01] [AO DGEMM MIC Time]Â Â Â 2.843016 seconds
- [MKL] [MIC 01] [AO DGEMM CPU->MIC Data]Â Â Â 1145600000 bytes
- [MKL] [MIC 01] [AO DGEMM MIC->CPU Data]Â Â Â 1382400000 bytes
- octave:4> exit
-
-On the example above we observe, that the DGEMM function workload was
-split over CPU, MIC 0 and MIC 1, in the ratio 0.14 0.43 0.43. The matrix
-multiplication was done on the CPU, accelerated by two Xeon Phi
-accelerators.
+```
Native Mode
-----------
+In the native mode a program is executed directly on Intel Xeon Phi without involvement of the host machine. Similarly to offload mode, the code is compiled on the host computer with Intel compilers.
-In the native mode a program is executed directly on Intel Xeon Phi
-without involvement of the host machine. Similarly to offload mode, the
-code is compiled on the host computer with Intel compilers.
-
-To compile a code user has to be connected to a compute with MIC and
-load Intel compilers module. To get an interactive session on a compute
-node with an Intel Xeon Phi and load the module use following commands:Â
+To compile a code user has to be connected to a compute with MIC and load Intel compilers module. To get an interactive session on a compute node with an Intel Xeon Phi and load the module use following commands:
- $ qsub -I -q qprod -l select=1:ncpus=24:accelerator=True:naccelerators=2:accelerator_model=phi7120 -A NONE-0-0
+```bash
+ $ qsub -I -q qmic -A NONE-0-0
- $ module load intel
+ $ module load intel/13.5.192
+```
-Please note that particular version of the Intel module is specified.
-This information is used later to specify the correct library paths.
+>Please note that particular version of the Intel module is specified. This information is used later to specify the correct library paths.
-To produce a binary compatible with Intel Xeon Phi architecture user has
-to specify "-mmic" compiler flag. Two compilation examples are shown
-below. The first example shows how to compile OpenMP parallel code
-"vect-add.c" for host only:
+To produce a binary compatible with Intel Xeon Phi architecture user has to specify "-mmic" compiler flag. Two compilation examples are shown below. The first example shows how to compile OpenMP parallel code "vect-add.c" for host only:
+```bash
$ icc -xhost -no-offload -fopenmp vect-add.c -o vect-add-host
+```
To run this code on host, use:
+```bash
$ ./vect-add-host
+```
-The second example shows how to compile the same code for Intel Xeon
-Phi:
+The second example shows how to compile the same code for Intel Xeon Phi:
+```bash
$ icc -mmic -fopenmp vect-add.c -o vect-add-mic
+```
### Execution of the Program in Native Mode on Intel Xeon Phi
-The user access to the Intel Xeon Phi is through the SSH. Since user
-home directories are mounted using NFS on the accelerator, users do not
-have to copy binary files or libraries between the host and accelerator.
-Â
-
-Get the PATH of MIC enabled libraries for currently used Intel Compiler
-(here was icc/2015.3.187-GNU-5.1.0-2.25 used) :
-
- $ echo $MIC_LD_LIBRARY_PATH
- /apps/all/icc/2015.3.187-GNU-5.1.0-2.25/composer_xe_2015.3.187/compiler/lib/mic
+The user access to the Intel Xeon Phi is through the SSH. Since user home directories are mounted using NFS on the accelerator, users do not have to copy binary files or libraries between the host and accelerator.
To connect to the accelerator run:
+```bash
$ ssh mic0
+```
If the code is sequential, it can be executed directly:
+```bash
mic0 $ ~/path_to_binary/vect-add-seq-mic
+```
-If the code is parallelized using OpenMP a set of additional libraries
-is required for execution. To locate these libraries new path has to be
-added to the LD_LIBRARY_PATH environment variable prior to the
-execution:
+If the code is parallelized using OpenMP a set of additional libraries is required for execution. To locate these libraries new path has to be added to the LD_LIBRARY_PATH environment variable prior to the execution:
- mic0 $ export LD_LIBRARY_PATH=/apps/all/icc/2015.3.187-GNU-5.1.0-2.25/composer_xe_2015.3.187/compiler/lib/mic:$LD_LIBRARY_PATH
+```bash
+ mic0 $ export LD_LIBRARY_PATH=/apps/intel/composer_xe_2013.5.192/compiler/lib/mic:$LD_LIBRARY_PATH
+```
-Please note that the path exported in the previous example contains path
-to a specific compiler (here the version is 2015.3.187-GNU-5.1.0-2.25).
-This version number has to match with the version number of the Intel
-compiler module that was used to compile the code on the host computer.
+>Please note that the path exported in the previous example contains path to a specific compiler (here the version is 5.192). This version number has to match with the version number of the Intel compiler module that was used to compile the code on the host computer.
-For your information the list of libraries and their location required
-for execution of an OpenMP parallel code on Intel Xeon Phi is:
+For your information the list of libraries and their location required for execution of an OpenMP parallel code on Intel Xeon Phi is:
-/apps/all/icc/2015.3.187-GNU-5.1.0-2.25/composer_xe_2015.3.187/compiler/lib/mic
+>/apps/intel/composer_xe_2013.5.192/compiler/lib/mic
-libiomp5.so
+>libiomp5.so
libimf.so
libsvml.so
libirng.so
libintlc.so.5
-Finally, to run the compiled code use:
+Finally, to run the compiled code use:
+
+```bash
$ ~/path_to_binary/vect-add-mic
+```
OpenCL
-------------------
+OpenCL (Open Computing Language) is an open standard for general-purpose parallel programming for diverse mix of multi-core CPUs, GPU coprocessors, and other parallel processors. OpenCL provides a flexible execution model and uniform programming environment for software developers to write portable code for systems running on both the CPU and graphics processors or accelerators like the Intel® Xeon Phi.
-OpenCL (Open Computing Language) is an open standard for
-general-purpose parallel programming for diverse mix of multi-core CPUs,
-GPU coprocessors, and other parallel processors. OpenCL provides a
-flexible execution model and uniform programming environment for
-software developers to write portable code for systems running on both
-the CPU and graphics processors or accelerators like the Intel® Xeon
-Phi.
-
-On Anselm OpenCL is installed only on compute nodes with MIC
-accelerator, therefore OpenCL code can be compiled only on these nodes.
+On Anselm OpenCL is installed only on compute nodes with MIC accelerator, therefore OpenCL code can be compiled only on these nodes.
+```bash
module load opencl-sdk opencl-rt
+```
-Always load "opencl-sdk" (providing devel files like headers) and
-"opencl-rt" (providing dynamic library libOpenCL.so) modules to compile
-and link OpenCL code. Load "opencl-rt" for running your compiled code.
+Always load "opencl-sdk" (providing devel files like headers) and "opencl-rt" (providing dynamic library libOpenCL.so) modules to compile and link OpenCL code. Load "opencl-rt" for running your compiled code.
-There are two basic examples of OpenCL code in the following
-directory:
+There are two basic examples of OpenCL code in the following directory:
+```bash
/apps/intel/opencl-examples/
+```
-First example "CapsBasic" detects OpenCL compatible hardware, here
-CPU and MIC, and prints basic information about the capabilities of
-it.Â
+First example "CapsBasic" detects OpenCL compatible hardware, here CPU and MIC, and prints basic information about the capabilities of it.
+```bash
/apps/intel/opencl-examples/CapsBasic/capsbasic
+```
-To compile and run the example copy it to your home directory, get
-a PBS interactive session on of the nodes with MIC and run make for
-compilation. Make files are very basic and shows how the OpenCL code can
-be compiled on Anselm.
+To compile and run the example copy it to your home directory, get a PBS interactive session on of the nodes with MIC and run make for compilation. Make files are very basic and shows how the OpenCL code can be compiled on Anselm.
+```bash
$ cp /apps/intel/opencl-examples/CapsBasic/* .
- $ qsub -I -q qprod -l select=1:ncpus=24:accelerator=True:naccelerators=2:accelerator_model=phi7120 -A NONE-0-0
+ $ qsub -I -q qmic -A NONE-0-0
$ make
+```
-The compilation command for this example is:
+The compilation command for this example is:
+```bash
$ g++ capsbasic.cpp -lOpenCL -o capsbasic -I/apps/intel/opencl/include/
+```
-After executing the complied binary file, following output should
-be displayed.
+After executing the complied binary file, following output should be displayed.
+```bash
./capsbasic
Number of available platforms: 1
@@ -622,27 +492,28 @@ be displayed.
   CL_DEVICE_AVAILABLE: 1
...
+```
-More information about this example can be found on Intel website:
-<http://software.intel.com/en-us/vcsource/samples/caps-basic/>
+>More information about this example can be found on Intel website: <http://software.intel.com/en-us/vcsource/samples/caps-basic/>
-The second example that can be found in
-"/apps/intel/opencl-examples" >directory is General Matrix
-Multiply. You can follow the the same procedure to download the example
-to your directory and compile it.Â
+The second example that can be found in "/apps/intel/opencl-examples" directory is General Matrix Multiply. You can follow the the same procedure to download the example to your directory and compile it.
+```bash
$ cp -r /apps/intel/opencl-examples/* .
- $ qsub -I -q qprod -l select=1:ncpus=24:accelerator=True:naccelerators=2:accelerator_model=phi7120 -A NONE-0-0
- $ cd GEMM
+ $ qsub -I -q qmic -A NONE-0-0
+ $ cd GEMM
$ make
+```
-The compilation command for this example is:
+The compilation command for this example is:
+```bash
$ g++ cmdoptions.cpp gemm.cpp ../common/basic.cpp ../common/cmdparser.cpp ../common/oclobject.cpp -I../common -lOpenCL -o gemm -I/apps/intel/opencl/include/
+```
-To see the performance of Intel Xeon Phi performing the DGEMM run
-the example as follows:
+To see the performance of Intel Xeon Phi performing the DGEMM run the example as follows:
+```bash
./gemm -d 1
Platforms (1):
[0] Intel(R) OpenCL [Selected]
@@ -660,403 +531,357 @@ the example as follows:
Host time: 0.293334 sec.
Host perf: 426.081 GFLOPS
...
+```
-Please note: GNU compiler is used to compile the OpenCL codes for
-Intel MIC. You do not need to load Intel compiler module.
+>Please note: GNU compiler is used to compile the OpenCL codes for Intel MIC. You do not need to load Intel compiler module.
MPI
-----------------
+-----------------
### Environment setup and compilation
-To achieve best MPI performance always use following setup for Intel MPI
-on Xeon Phi accelerated nodes:
-
- $ export I_MPI_FABRICS=shm:dapl
- $ export I_MPI_DAPL_PROVIDER_LIST=ofa-v2-mlx4_0-1u,ofa-v2-scif0,ofa-v2-mcm-1
-
-This ensures, that MPI inside node will use SHMEM communication, between
-HOST and Phi the IB SCIF will be used and between different nodes or
-Phi's on diferent nodes a CCL-Direct proxy will be used.
-
-Please note: Other FABRICS like tcp,ofa may be used (even combined with
-shm) but there's severe loss of performance (by order of magnitude).
-Usage of single DAPL PROVIDER (e. g.
-I_MPI_DAPL_PROVIDER=ofa-v2-mlx4_0-1u) will cause failure of
-Host<->Phi and/or Phi<->Phi communication.
-Usage of the I_MPI_DAPL_PROVIDER_LIST on non-accelerated node will
-cause failure of any MPI communication, since those nodes don't have
-SCIF device and there's no CCL-Direct proxy runnig.
+Again an MPI code for Intel Xeon Phi has to be compiled on a compute node with accelerator and MPSS software stack installed. To get to a compute node with accelerator use:
-Again an MPI code for Intel Xeon Phi has to be compiled on a compute
-node with accelerator and MPSS software stack installed. To get to a
-compute node with accelerator use:
+```bash
+ $ qsub -I -q qmic -A NONE-0-0
+```
- $ qsub -I -q qprod -l select=1:ncpus=24:accelerator=True:naccelerators=2:accelerator_model=phi7120 -A NONE-0-0
+The only supported implementation of MPI standard for Intel Xeon Phi is Intel MPI. To setup a fully functional development environment a combination of Intel compiler and Intel MPI has to be used. On a host load following modules before compilation:
-The only supported implementation of MPI standard for Intel Xeon Phi is
-Intel MPI. To setup a fully functional development environment a
-combination of Intel compiler and Intel MPI has to be used. On a host
-load following modules before compilation:
-
- $ module load intel impi
+```bash
+ $ module load intel/13.5.192 impi/4.1.1.036
+```
To compile an MPI code for host use:
+```bash
$ mpiicc -xhost -o mpi-test mpi-test.c
+```
To compile the same code for Intel Xeon Phi architecture use:
+```bash
$ mpiicc -mmic -o mpi-test-mic mpi-test.c
+```
-Or, if you are using Fortran :
-
- $ mpiifort -mmic -o mpi-test-mic mpi-test.f90
-
-An example of basic MPI version of "hello-world" example in C language,
-that can be executed on both host and Xeon Phi is (can be directly copy
-and pasted to a .c file)
+An example of basic MPI version of "hello-world" example in C language, that can be executed on both host and Xeon Phi is (can be directly copy and pasted to a .c file)
-`
-#include <stdio.h>
-#include <mpi.h>
+```cpp
+ #include <stdio.h>
+ #include <mpi.h>
-int main (argc, argv)
-Â Â Â Â int argc;
-Â Â Â Â char *argv[];
-{
-Â int rank, size;
+ int main (argc, argv)
+ Â Â Â Â int argc;
+ Â Â Â Â char *argv[];
+ {
+ Â int rank, size;
-Â int len;
-Â char node[MPI_MAX_PROCESSOR_NAME];
+ Â int len;
+ Â char node[MPI_MAX_PROCESSOR_NAME];
-Â MPI_Init (&argc, &argv);Â Â Â Â Â /* starts MPI */
-Â MPI_Comm_rank (MPI_COMM_WORLD, &rank);Â Â Â Â Â Â Â /* get current process id */
-Â MPI_Comm_size (MPI_COMM_WORLD, &size);Â Â Â Â Â Â Â /* get number of processes */
+ Â MPI_Init (&argc, &argv);Â Â Â Â Â /* starts MPI */
+ Â MPI_Comm_rank (MPI_COMM_WORLD, &rank);Â Â Â Â Â Â Â /* get current process id */
+ Â MPI_Comm_size (MPI_COMM_WORLD, &size);Â Â Â Â Â Â Â /* get number of processes */
-Â MPI_Get_processor_name(node,&len);
+ Â MPI_Get_processor_name(node,&len);
-Â printf( "Hello world from process %d of %d on host %s n", rank, size, node );
-Â MPI_Finalize();
-Â return 0;
-}
-`
+ Â printf( "Hello world from process %d of %d on host %s n", rank, size, node );
+ Â MPI_Finalize();
+ Â return 0;
+ }
+```
### MPI programming models
-Intel MPI for the Xeon Phi coprocessors offers different MPI
-programming models:
+Intel MPI for the Xeon Phi coprocessors offers different MPI programming models:
-Host-only model** - all MPI ranks reside on the host. The coprocessors
-can be used by using offload pragmas. (Using MPI calls inside offloaded
-code is not supported.)**
+>**Host-only model** - all MPI ranks reside on the host. The coprocessors can be used by using offload pragmas. (Using MPI calls inside offloaded code is not supported.)
-Coprocessor-only model** - all MPI ranks reside only on the
-coprocessors.
+>**Coprocessor-only model** - all MPI ranks reside only on the coprocessors.
-Symmetric model** - the MPI ranks reside on both the host and the
-coprocessor. Most general MPI case.
+>**Symmetric model** - the MPI ranks reside on both the host and the coprocessor. Most general MPI case.
###Host-only model
-In this case all environment variables are set by modules,
-so to execute the compiled MPI program on a single node, use:
+In this case all environment variables are set by modules, so to execute the compiled MPI program on a single node, use:
+```bash
$ mpirun -np 4 ./mpi-test
+```
The output should be similar to:
- Hello world from process 1 of 4 on host r38u31n1000
- Hello world from process 3 of 4 on host r38u31n1000
- Hello world from process 2 of 4 on host r38u31n1000
- Hello world from process 0 of 4 on host r38u31n1000
+```bash
+ Hello world from process 1 of 4 on host cn207
+ Hello world from process 3 of 4 on host cn207
+ Hello world from process 2 of 4 on host cn207
+ Hello world from process 0 of 4 on host cn207
+```
### Coprocessor-only model
-There are two ways how to execute an MPI code on a single
-coprocessor: 1.) lunch the program using "**mpirun**" from the
-coprocessor; or 2.) lunch the task using "**mpiexec.hydra**" from a
-host.
-
-Execution on coprocessor**Â
+There are two ways how to execute an MPI code on a single coprocessor: 1.) lunch the program using "**mpirun**" from the
+coprocessor; or 2.) lunch the task using "**mpiexec.hydra**" from a host.
-Similarly to execution of OpenMP programs in native mode, since the
-environmental module are not supported on MIC, user has to setup paths
-to Intel MPI libraries and binaries manually. One time setup can be done
-by creating a "**.profile**" file in user's home directory. This file
-sets up the environment on the MIC automatically once user access to the
-accelerator through the SSH.
+**Execution on coprocessor**
-At first get the LD_LIBRARY_PATH for currenty used Intel Compiler and
-Intel MPI:
+Similarly to execution of OpenMP programs in native mode, since the environmental module are not supported on MIC, user has to setup paths to Intel MPI libraries and binaries manually. One time setup can be done by creating a "**.profile**" file in user's home directory. This file sets up the environment on the MIC automatically once user access to the accelerator through the SSH.
- $ echo $MIC_LD_LIBRARY_PATH
- /apps/all/imkl/11.2.3.187-iimpi-7.3.5-GNU-5.1.0-2.25/mkl/lib/mic:/apps/all/imkl/11.2.3.187-iimpi-7.3.5-GNU-5.1.0-2.25/lib/mic:/apps/all/icc/2015.3.187-GNU-5.1.0-2.25/composer_xe_2015.3.187/compiler/lib/mic/
-
-Use it in your ~/.profile:
-
- $ vim ~/.profile
+```bash
+ $ vim ~/.profile
PS1='[u@h W]$ '
export PATH=/usr/bin:/usr/sbin:/bin:/sbin
- #IMPI
- export PATH=/apps/all/impi/5.0.3.048-iccifort-2015.3.187-GNU-5.1.0-2.25/mic/bin/:$PATH
+ #OpenMP
+ export LD_LIBRARY_PATH=/apps/intel/composer_xe_2013.5.192/compiler/lib/mic:$LD_LIBRARY_PATH
- #OpenMP (ICC, IFORT), IMKL and IMPI
- export LD_LIBRARY_PATH=/apps/all/imkl/11.2.3.187-iimpi-7.3.5-GNU-5.1.0-2.25/mkl/lib/mic:/apps/all/imkl/11.2.3.187-iimpi-7.3.5-GNU-5.1.0-2.25/lib/mic:/apps/all/icc/2015.3.187-GNU-5.1.0-2.25/composer_xe_2015.3.187/compiler/lib/mic:$LD_LIBRARY_PATH
+ #Intel MPI
+ export LD_LIBRARY_PATH=/apps/intel/impi/4.1.1.036/mic/lib/:$LD_LIBRARY_PATH
+ export PATH=/apps/intel/impi/4.1.1.036/mic/bin/:$PATH
+```
-Please note:
-Â - this file sets up both environmental variable for both MPI and OpenMP
-libraries.
-Â - this file sets up the paths to a particular version of Intel MPI
-library and particular version of an Intel compiler. These versions have
-to match with loaded modules.
+>Please note:
+Â - this file sets up both environmental variable for both MPI and OpenMP libraries.
+Â - this file sets up the paths to a particular version of Intel MPI library and particular version of an Intel compiler. These versions have to match with loaded modules.
-To access a MIC accelerator located on a node that user is currently
-connected to, use:
+To access a MIC accelerator located on a node that user is currently connected to, use:
+```bash
$ ssh mic0
+```
or in case you need specify a MIC accelerator on a particular node, use:
- $ ssh r38u31n1000-mic0
+```bash
+ $ ssh cn207-mic0
+```
-To run the MPI code in parallel on multiple core of the accelerator,
-use:
+To run the MPI code in parallel on multiple core of the accelerator, use:
+```bash
$ mpirun -np 4 ./mpi-test-mic
+```
The output should be similar to:
- Hello world from process 1 of 4 on host r38u31n1000-mic0
- Hello world from process 2 of 4 on host r38u31n1000-mic0
- Hello world from process 3 of 4 on host r38u31n1000-mic0
- Hello world from process 0 of 4 on host r38u31n1000-mic0
-
-Â **
+```bash
+ Hello world from process 1 of 4 on host cn207-mic0
+ Hello world from process 2 of 4 on host cn207-mic0
+ Hello world from process 3 of 4 on host cn207-mic0
+ Hello world from process 0 of 4 on host cn207-mic0
+```
**Execution on host**
-If the MPI program is launched from host instead of the coprocessor, the
-environmental variables are not set using the ".profile" file. Therefore
-user has to specify library paths from the command line when calling
-"mpiexec".
+If the MPI program is launched from host instead of the coprocessor, the environmental variables are not set using the ".profile" file. Therefore user has to specify library paths from the command line when calling "mpiexec".
-First step is to tell mpiexec that the MPI should be executed on a local
-accelerator by setting up the environmental variable "I_MPI_MIC"
+First step is to tell mpiexec that the MPI should be executed on a local accelerator by setting up the environmental variable "I_MPI_MIC"
+```bash
$ export I_MPI_MIC=1
+```
Now the MPI program can be executed as:
- $ mpirun -genv LD_LIBRARY_PATH $MIC_LD_LIBRARY_PATH -host mic0 -n 4 ~/mpi-test-mic
+```bash
+ $ mpiexec.hydra -genv LD_LIBRARY_PATH /apps/intel/impi/4.1.1.036/mic/lib/ -host mic0 -n 4 ~/mpi-test-mic
+```
or using mpirun
- $ mpirun -genv LD_LIBRARY_PATH $MIC_LD_LIBRARY_PATH -host mic0 -n 4 ~/mpi-test-mic
+```bash
+ $ mpirun -genv LD_LIBRARY_PATH /apps/intel/impi/4.1.1.036/mic/lib/ -host mic0 -n 4 ~/mpi-test-mic
+```
-Please note:
-Â - the full path to the binary has to specified (here:
-"**>~/mpi-test-mic**")
-Â - the LD_LIBRARY_PATH has to match with Intel MPI module used to
-compile the MPI code
+>Please note:
+Â - the full path to the binary has to specified (here: "**>~/mpi-test-mic**")
+Â - the LD_LIBRARY_PATH has to match with Intel MPI module used to compile the MPI code
The output should be again similar to:
- Hello world from process 1 of 4 on host r38u31n1000-mic0
- Hello world from process 2 of 4 on host r38u31n1000-mic0
- Hello world from process 3 of 4 on host r38u31n1000-mic0
- Hello world from process 0 of 4 on host r38u31n1000-mic0
+```bash
+ Hello world from process 1 of 4 on host cn207-mic0
+ Hello world from process 2 of 4 on host cn207-mic0
+ Hello world from process 3 of 4 on host cn207-mic0
+ Hello world from process 0 of 4 on host cn207-mic0
+```
-Please note that the "mpiexec.hydra" requires a file
-"**>pmi_proxy**" from Intel MPI library to be copied to the
-MIC filesystem. If the file is missing please contact the system
-administrators. A simple test to see if the file is present is to
-execute:
+>>Please note that the **"mpiexec.hydra"** requires a file the MIC filesystem. If the file is missing please contact the system administrators. A simple test to see if the file is present is to execute:
+```bash
  $ ssh mic0 ls /bin/pmi_proxy
 /bin/pmi_proxy
+```
-Â **
+**Execution on host - MPI processes distributed over multiple accelerators on multiple nodes**
-**Execution on host - MPI processes distributed over multiple
-accelerators on multiple nodes**
+To get access to multiple nodes with MIC accelerator, user has to use PBS to allocate the resources. To start interactive session, that allocates 2 compute nodes = 2 MIC accelerators run qsub command with following parameters:
-To get access to multiple nodes with MIC accelerator, user has to
-use PBS to allocate the resources. To start interactive session, that
-allocates 2 compute nodes = 2 MIC accelerators run qsub command with
-following parameters:
+```bash
+ $ qsub -I -q qmic -A NONE-0-0 -l select=2:ncpus=16
- $ qsub -I -q qprod -l select=2:ncpus=24:accelerator=True:naccelerators=2:accelerator_model=phi7120 -A NONE-0-0
+ $ module load intel/13.5.192 impi/4.1.1.036
+```
- $ module load intel impi
-
-This command connects user through ssh to one of the nodes
-immediately. To see the other nodes that have been allocated use:
+This command connects user through ssh to one of the nodes immediately. To see the other nodes that have been allocated use:
+```bash
$ cat $PBS_NODEFILE
+```
+
+For example:
+
+```bash
+ cn204.bullx
+ cn205.bullx
+```
+
+This output means that the PBS allocated nodes cn204 and cn205, which means that user has direct access to "**cn204-mic0**" and "**cn-205-mic0**" accelerators.
-For example:
-
- r38u31n1000.bullx
- r38u32n1001.bullx
-
-This output means that the PBS allocated nodes r38u31n1000 and
-r38u32n1001, which means that user has direct access to
-"**r38u31n1000-mic0**" and "**>r38u32n1001-mic0**"
-accelerators.
-
-Please note: At this point user can connect to any of the
-allocated nodes or any of the allocated MIC accelerators using ssh:
-- to connect to the second node : ** $
-ssh >r38u32n1001**
-- to connect to the accelerator on the first node from the first
-node: **$ ssh
-r38u31n1000-mic0** or **
-$ ssh mic0**
--** to connect to the accelerator on the second node from the first
-node: **$ ssh
-r38u32n1001-mic0**
-
-At this point we expect that correct modules are loaded and binary
-is compiled. For parallel execution the mpiexec.hydra is used.
-Again the first step is to tell mpiexec that the MPI can be executed on
-MIC accelerators by setting up the environmental variable "I_MPI_MIC",
-don't forget to have correct FABRIC and PROVIDER defined.
+>Please note: At this point user can connect to any of the allocated nodes or any of the allocated MIC accelerators using ssh:
+- to connect to the second node : ** $ ssh cn205**
+- to connect to the accelerator on the first node from the first node: **$ ssh cn204-mic0** or **$ ssh mic0**
+- to connect to the accelerator on the second node from the first node: **$ ssh cn205-mic0**
+At this point we expect that correct modules are loaded and binary is compiled. For parallel execution the mpiexec.hydra is used. Again the first step is to tell mpiexec that the MPI can be executed on MIC accelerators by setting up the environmental variable "I_MPI_MIC"
+
+```bash
$ export I_MPI_MIC=1
- $ export I_MPI_FABRICS=shm:dapl
- $ export I_MPI_DAPL_PROVIDER_LIST=ofa-v2-mlx4_0-1u,ofa-v2-scif0,ofa-v2-mcm-1
+```
The launch the MPI program use:
- $ mpirun -genv LD_LIBRARY_PATH $MIC_LD_LIBRARY_PATH
- -host r38u31n1000-mic0 -n 4 ~/mpi-test-mic
- : -host r38u32n1001-mic0 -n 6 ~/mpi-test-mic
-
+```bash
+ $ mpiexec.hydra -genv LD_LIBRARY_PATH /apps/intel/impi/4.1.1.036/mic/lib/
+ -genv I_MPI_FABRICS_LIST tcp
+ Â -genv I_MPI_FABRICS shm:tcp
+ Â -genv I_MPI_TCP_NETMASK=10.1.0.0/16
+ -host cn204-mic0 -n 4 ~/mpi-test-mic
+ : -host cn205-mic0 -n 6 ~/mpi-test-mic
+```
or using mpirun:
- $ mpirun -genv LD_LIBRARY_PATH
- -host r38u31n1000-mic0 -n 4 ~/mpi-test-mic
- : -host r38u32n1001-mic0 -n 6 ~/mpi-test-mic
-
-In this case four MPI processes are executed on accelerator
-r38u31n1000-mic and six processes are executed on accelerator
-r38u32n1001-mic0. The sample output (sorted after execution) is:
-
- Hello world from process 0 of 10 on host r38u31n1000-mic0
- Hello world from process 1 of 10 on host r38u31n1000-mic0
- Hello world from process 2 of 10 on host r38u31n1000-mic0
- Hello world from process 3 of 10 on host r38u31n1000-mic0
- Hello world from process 4 of 10 on host r38u32n1001-mic0
- Hello world from process 5 of 10 on host r38u32n1001-mic0
- Hello world from process 6 of 10 on host r38u32n1001-mic0
- Hello world from process 7 of 10 on host r38u32n1001-mic0
- Hello world from process 8 of 10 on host r38u32n1001-mic0
- Hello world from process 9 of 10 on host r38u32n1001-mic0
-
-The same way MPI program can be executed on multiple hosts:Â
-
- $ mpirun -genv LD_LIBRARY_PATH $MIC_LD_LIBRARY_PATH
- -host r38u31n1000 -n 4 ~/mpi-test
- : -host r38u32n1001 -n 6 ~/mpi-test
-
-###Symmetric model
-
-In a symmetric mode MPI programs are executed on both host
-computer(s) and MIC accelerator(s). Since MIC has a different
-architecture and requires different binary file produced by the Intel
-compiler two different files has to be compiled before MPI program is
-executed.
-
-In the previous section we have compiled two binary files, one for
-hosts "**mpi-test**" and one for MIC accelerators "**mpi-test-mic**".
-These two binaries can be executed at once using mpiexec.hydra:
-
- $ mpirun
- -genv $MIC_LD_LIBRARY_PATH
- -host r38u32n1001 -n 2 ~/mpi-test
- : -host r38u32n1001-mic0 -n 2 ~/mpi-test-mic
-
-In this example the first two parameters (line 2 and 3) sets up required
-environment variables for execution. The third line specifies binary
-that is executed on host (here r38u32n1001) and the last line specifies
-the binary that is execute on the accelerator (here r38u32n1001-mic0).
-
-The output of the program is:
-
- Hello world from process 0 of 4 on host r38u32n1001
- Hello world from process 1 of 4 on host r38u32n1001
- Hello world from process 2 of 4 on host r38u32n1001-mic0
- Hello world from process 3 of 4 on host r38u32n1001-mic0
-
-The execution procedure can be simplified by using the mpirun
-command with the machine file a a parameter. Machine file contains list
-of all nodes and accelerators that should used to execute MPI processes.
-
-An example of a machine file that uses 2 >hosts (r38u32n1001
-and r38u33n1002) and 2 accelerators **(r38u32n1001-mic0** and
-r38u33n1002-mic0**) to run 2 MPI processes
-on each of them:
-
+```bash
+ $ mpirun -genv LD_LIBRARY_PATH /apps/intel/impi/4.1.1.036/mic/lib/
+ -genv I_MPI_FABRICS_LIST tcp
+ Â -genv I_MPI_FABRICS shm:tcp
+ Â -genv I_MPI_TCP_NETMASK=10.1.0.0/16
+ -host cn204-mic0 -n 4 ~/mpi-test-mic
+ : -host cn205-mic0 -n 6 ~/mpi-test-mic
+```
+
+In this case four MPI processes are executed on accelerator cn204-mic and six processes are executed on accelerator cn205-mic0. The sample output (sorted after execution) is:
+
+```bash
+ Hello world from process 0 of 10 on host cn204-mic0
+ Hello world from process 1 of 10 on host cn204-mic0
+ Hello world from process 2 of 10 on host cn204-mic0
+ Hello world from process 3 of 10 on host cn204-mic0
+ Hello world from process 4 of 10 on host cn205-mic0
+ Hello world from process 5 of 10 on host cn205-mic0
+ Hello world from process 6 of 10 on host cn205-mic0
+ Hello world from process 7 of 10 on host cn205-mic0
+ Hello world from process 8 of 10 on host cn205-mic0
+ Hello world from process 9 of 10 on host cn205-mic0
+```
+
+The same way MPI program can be executed on multiple hosts:
+
+```bash
+ $ mpiexec.hydra -genv LD_LIBRARY_PATH /apps/intel/impi/4.1.1.036/mic/lib/
+ -genv I_MPI_FABRICS_LIST tcp
+ Â -genv I_MPI_FABRICS shm:tcp
+ Â -genv I_MPI_TCP_NETMASK=10.1.0.0/16
+ -host cn204 -n 4 ~/mpi-test
+ : -host cn205 -n 6 ~/mpi-test
+```
+
+###Symmetric model
+
+In a symmetric mode MPI programs are executed on both host computer(s) and MIC accelerator(s). Since MIC has a different
+architecture and requires different binary file produced by the Intel compiler two different files has to be compiled before MPI program is executed.
+
+In the previous section we have compiled two binary files, one for hosts "**mpi-test**" and one for MIC accelerators "**mpi-test-mic**". These two binaries can be executed at once using mpiexec.hydra:
+
+```bash
+ $ mpiexec.hydra
+ -genv I_MPI_FABRICS_LIST tcp
+ -genv I_MPI_FABRICS shm:tcp
+ Â -genv I_MPI_TCP_NETMASK=10.1.0.0/16
+ -genv LD_LIBRARY_PATH /apps/intel/impi/4.1.1.036/mic/lib/
+ -host cn205 -n 2 ~/mpi-test
+ : -host cn205-mic0 -n 2 ~/mpi-test-mic
+```
+
+In this example the first two parameters (line 2 and 3) sets up required environment variables for execution. The third line specifies binary that is executed on host (here cn205) and the last line specifies the binary that is execute on the accelerator (here cn205-mic0).
+
+The output of the program is:
+
+```bash
+ Hello world from process 0 of 4 on host cn205
+ Hello world from process 1 of 4 on host cn205
+ Hello world from process 2 of 4 on host cn205-mic0
+ Hello world from process 3 of 4 on host cn205-mic0
+```
+
+The execution procedure can be simplified by using the mpirun command with the machine file a a parameter. Machine file contains list of all nodes and accelerators that should used to execute MPI processes.
+
+An example of a machine file that uses 2 >hosts (**cn205** and **cn206**) and 2 accelerators **(cn205-mic0** and **cn206-mic0**) to run 2 MPI processes on each of them:
+
+```bash
$ cat hosts_file_mix
- r38u32n1001:2
- r38u32n1001-mic0:2
- r38u33n1002:2
- r38u33n1002-mic0:2
-
-In addition if a naming convention is set in a way that the name
-of the binary for host is **"bin_name"**Â and the name of the binary
-for the accelerator is **"bin_name-mic"** then by setting up the
-environment variable **I_MPI_MIC_POSTFIX** to **"-mic"** user do not
-have to specify the names of booth binaries. In this case mpirun needs
-just the name of the host binary file (i.e. "mpi-test") and uses the
-suffix to get a name of the binary for accelerator (i..e.
-"mpi-test-mic").
+ cn205:2
+ cn205-mic0:2
+ cn206:2
+ cn206-mic0:2
+```
- $ export I_MPI_MIC_POSTFIX=-mic
-
-Â >To run the MPI code using mpirun and the machine file
-"hosts_file_mix" use:
+In addition if a naming convention is set in a way that the name of the binary for host is **"bin_name"** and the name of the binary for the accelerator is **"bin_name-mic"** then by setting up the environment variable **I_MPI_MIC_POSTFIX** to **"-mic"** user do not have to specify the names of booth binaries. In this case mpirun needs just the name of the host binary file (i.e. "mpi-test") and uses the suffix to get a name of the binary for accelerator (i..e. "mpi-test-mic").
- $ mpirun
- -genv LD_LIBRARY_PATH $MIC_LD_LIBRARY_PATH
- -machinefile hosts_file_mix
+```bash
+ $ export I_MPI_MIC_POSTFIX=-mic
+```
+
+To run the MPI code using mpirun and the machine file "hosts_file_mix" use:
+
+```bash
+ $ mpirun
+ -genv I_MPI_FABRICS shm:tcp
+ -genv LD_LIBRARY_PATH /apps/intel/impi/4.1.1.036/mic/lib/
+ -genv I_MPI_FABRICS_LIST tcp
+ Â -genv I_MPI_FABRICS shm:tcp
+ Â -genv I_MPI_TCP_NETMASK=10.1.0.0/16
+ -machinefile hosts_file_mix
~/mpi-test
+```
-A possible output of the MPI "hello-world" example executed on two
-hosts and two accelerators is:
+A possible output of the MPI "hello-world" example executed on two hosts and two accelerators is:
- Hello world from process 0 of 8 on host r38u31n1000
- Hello world from process 1 of 8 on host r38u31n1000
- Hello world from process 2 of 8 on host r38u31n1000-mic0
- Hello world from process 3 of 8 on host r38u31n1000-mic0
- Hello world from process 4 of 8 on host r38u32n1001
- Hello world from process 5 of 8 on host r38u32n1001
- Hello world from process 6 of 8 on host r38u32n1001-mic0
- Hello world from process 7 of 8 on host r38u32n1001-mic0
+```bash
+ Hello world from process 0 of 8 on host cn204
+ Hello world from process 1 of 8 on host cn204
+ Hello world from process 2 of 8 on host cn204-mic0
+ Hello world from process 3 of 8 on host cn204-mic0
+ Hello world from process 4 of 8 on host cn205
+ Hello world from process 5 of 8 on host cn205
+ Hello world from process 6 of 8 on host cn205-mic0
+ Hello world from process 7 of 8 on host cn205-mic0
+```
-Using the PBS automatically generated node-files
+>Please note: At this point the MPI communication between MIC accelerators on different nodes uses 1Gb Ethernet only.
-PBS also generates a set of node-files that can be used instead of
-manually creating a new one every time. Three node-files are genereated:
+**Using the PBS automatically generated node-files**
-**Host only node-file:**
-Â - /lscratch/${PBS_JOBID}/nodefile-cn
-MIC only node-file:
-Â - /lscratch/${PBS_JOBID}/nodefile-mic
-Host and MIC node-file:
+PBS also generates a set of node-files that can be used instead of manually creating a new one every time. Three node-files are genereated:
+
+>**Host only node-file:**
+Â - /lscratch/${PBS_JOBID}/nodefile-cn MIC only node-file:
+Â - /lscratch/${PBS_JOBID}/nodefile-mic Host and MIC node-file:
 - /lscratch/${PBS_JOBID}/nodefile-mix
-Please note each host or accelerator is listed only per files. User has
-to specify how many jobs should be executed per node using "-n"
-parameter of the mpirun command.
+Please note each host or accelerator is listed only per files. User has to specify how many jobs should be executed per node using "-n" parameter of the mpirun command.
Optimization
------------
-
-For more details about optimization techniques please read Intel
-document [Optimization and Performance Tuning for Intel® Xeon Phi™
-Coprocessors](http://software.intel.com/en-us/articles/optimization-and-performance-tuning-for-intel-xeon-phi-coprocessors-part-1-optimization "http://software.intel.com/en-us/articles/optimization-and-performance-tuning-for-intel-xeon-phi-coprocessors-part-1-optimization")
-
+For more details about optimization techniques please read Intel document [Optimization and Performance Tuning for Intel® Xeon Phi™ Coprocessors](http://software.intel.com/en-us/articles/optimization-and-performance-tuning-for-intel-xeon-phi-coprocessors-part-1-optimization "http://software.intel.com/en-us/articles/optimization-and-performance-tuning-for-intel-xeon-phi-coprocessors-part-1-optimization")
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/java.md b/docs.it4i/salomon/software/java.md
index ced88c9ef..aa3bbd30d 100644
--- a/docs.it4i/salomon/software/java.md
+++ b/docs.it4i/salomon/software/java.md
@@ -1,33 +1,26 @@
-Java
+Java
====
-Java on the cluster
+**Java on the cluster**
-
-
-Java is available on the cluster. Activate java by loading the Java
-module
+Java is available on the cluster. Activate java by loading the Java module
+```bash
$ module load Java
+```
-Note that the Java module must be loaded on the compute nodes as well,
-in order to run java on compute nodes.
+Note that the Java module must be loaded on the compute nodes as well, in order to run java on compute nodes.
Check for java version and path
+```bash
$ java -version
$ which java
+```
-With the module loaded, not only the runtime environment (JRE), but also
-the development environment (JDK) with the compiler is available.
+With the module loaded, not only the runtime environment (JRE), but also the development environment (JDK) with the compiler is available.
$ javac -version
$ which javac
-Java applications may use MPI for interprocess communication, in
-conjunction with OpenMPI. Read more
-on <http://www.open-mpi.org/faq/?category=java>.
-This functionality is currently not supported on Anselm cluster. In case
-you require the java interface to MPI, please contact [cluster
-support](https://support.it4i.cz/rt/).
-
+Java applications may use MPI for interprocess communication, in conjunction with OpenMPI. Read more on <http://www.open-mpi.org/faq/?category=java>. This functionality is currently not supported on Anselm cluster. In case you require the java interface to MPI, please contact [cluster support](https://support.it4i.cz/rt/).
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/mpi-1/mpi.md b/docs.it4i/salomon/software/mpi-1/mpi.md
deleted file mode 100644
index 13c3f1509..000000000
--- a/docs.it4i/salomon/software/mpi-1/mpi.md
+++ /dev/null
@@ -1,185 +0,0 @@
-MPI
-===
-
-
-
-Setting up MPI Environment
---------------------------
-
-The Salomon cluster provides several implementations of the MPI library:
-
- -------------------------------------------------------------------------
- MPI Library Thread support
- ------ |---|---|---
- **Intel MPI 4.1** Full thread support up to
- MPI_THREAD_MULTIPLE
-
- **Intel MPI 5.0** Full thread support up to
- MPI_THREAD_MULTIPLE
-
- OpenMPI 1.8.6 Full thread support up to
- MPI_THREAD_MULTIPLE, MPI-3.0
- support
-
- SGI MPT 2.12
- -------------------------------------------------------------------------
-
-MPI libraries are activated via the environment modules.
-
-Look up section modulefiles/mpi in module avail
-
- $ module avail
- ------------------------------ /apps/modules/mpi -------------------------------
- impi/4.1.1.036-iccifort-2013.5.192
- impi/4.1.1.036-iccifort-2013.5.192-GCC-4.8.3
- impi/5.0.3.048-iccifort-2015.3.187
- impi/5.0.3.048-iccifort-2015.3.187-GNU-5.1.0-2.25
- MPT/2.12
- OpenMPI/1.8.6-GNU-5.1.0-2.25
-
-There are default compilers associated with any particular MPI
-implementation. The defaults may be changed, the MPI libraries may be
-used in conjunction with any compiler.
-The defaults are selected via the modules in following way
-
- --------------------------------------------------------------------------
- Module MPI Compiler suite
- ------------------ |---|---|-------------- ------------------------
- impi-5.0.3.048-iccifort- Intel MPI 5.0.3
- 2015.3.187
-
- OpenMP-1.8.6-GNU-5.1.0-2 OpenMPI 1.8.6
- .25
- --------------------------------------------------------------------------
-
-Examples:
-
- $ module load gompi/2015b
-
-In this example, we activate the latest OpenMPI with latest GNU
-compilers (OpenMPI 1.8.6 and GCC 5.1). Please see more information about
-toolchains in section [Environment and
-Modules](../../environment-and-modules.html)Â .
-
-To use OpenMPI with the intel compiler suite, use
-
- $ module load iompi/2015.03
-
-In this example, the openmpi 1.8.6 using intel compilers is activated.
-It's used "iompi" toolchain.
-
-Compiling MPI Programs
-----------------------
-
-After setting up your MPI environment, compile your program using one of
-the mpi wrappers
-
- $ mpicc -v
- $ mpif77 -v
- $ mpif90 -v
-
-When using Intel MPI, use the following MPI wrappers:
-
- $ mpicc
- $ mpiifortÂ
-
-Wrappers mpif90, mpif77 that are provided by Intel MPI are designed for
-gcc and gfortran. You might be able to compile MPI code by them even
-with Intel compilers, but you might run into problems (for example,
-native MIC compilation with -mmic does not work with mpif90).
-
-Example program:
-
- // helloworld_mpi.c
- #include <stdio.h>
-
- #include<mpi.h>
-
- int main(int argc, char **argv) {
-
- int len;
- int rank, size;
- char node[MPI_MAX_PROCESSOR_NAME];
-
- // Initiate MPI
- MPI_Init(&argc, &argv);
- MPI_Comm_rank(MPI_COMM_WORLD,&rank);
- MPI_Comm_size(MPI_COMM_WORLD,&size);
-
- // Get hostame and print
- MPI_Get_processor_name(node,&len);
- printf("Hello world! from rank %d of %d on host %sn",rank,size,node);
-
- // Finalize and exit
- MPI_Finalize();
-
- return 0;
- }
-
-Compile the above example with
-
- $ mpicc helloworld_mpi.c -o helloworld_mpi.x
-
-Running MPI Programs
---------------------
-
-The MPI program executable must be compatible with the loaded MPI
-module.
-Always compile and execute using the very same MPI module.
-
-It is strongly discouraged to mix mpi implementations. Linking an
-application with one MPI implementation and running mpirun/mpiexec form
-other implementation may result in unexpected errors.
-
-The MPI program executable must be available within the same path on all
-nodes. This is automatically fulfilled on the /home and /scratch
-filesystem. You need to preload the executable, if running on the local
-scratch /lscratch filesystem.
-
-### Ways to run MPI programs
-
-Optimal way to run an MPI program depends on its memory requirements,
-memory access pattern and communication pattern.
-
-Consider these ways to run an MPI program:
-1. One MPI process per node, 24 threads per process
-2. Two MPI processes per node, 12 threads per process
-3. 24 MPI processes per node, 1 thread per process.
-
-One MPI** process per node, using 24 threads, is most useful for
-memory demanding applications, that make good use of processor cache
-memory and are not memory bound. This is also a preferred way for
-communication intensive applications as one process per node enjoys full
-bandwidth access to the network interface.Â
-
-Two MPI** processes per node, using 12 threads each, bound to
-processor socket is most useful for memory bandwidth bound applications
-such as BLAS1 or FFT, with scalable memory demand. However, note that
-the two processes will share access to the network interface. The 12
-threads and socket binding should ensure maximum memory access bandwidth
-and minimize communication, migration and numa effect overheads.
-
-Important! Bind every OpenMP thread to a core!
-
-In the previous two cases with one or two MPI processes per node, the
-operating system might still migrate OpenMP threads between cores. You
-want to avoid this by setting the KMP_AFFINITY or GOMP_CPU_AFFINITY
-environment variables.
-
-**24 MPI** processes per node, using 1 thread each bound to processor
-core is most suitable for highly scalable applications with low
-communication demand.
-
-### Running OpenMPI
-
-The [**OpenMPI 1.8.6**](http://www.open-mpi.org/) is
-based on OpenMPI. Read more on [how to run
-OpenMPI](Running_OpenMPI.html) based MPI.
-
-Â
-
-The Intel MPI may run on the[Intel Xeon
-Ph](../intel-xeon-phi.html)i accelerators as well. Read
-more on [how to run Intel MPI on
-accelerators](../intel-xeon-phi.html).
-
diff --git a/docs.it4i/salomon/software/mpi-1/Running_OpenMPI.md b/docs.it4i/salomon/software/mpi/Running_OpenMPI.md
similarity index 63%
rename from docs.it4i/salomon/software/mpi-1/Running_OpenMPI.md
rename to docs.it4i/salomon/software/mpi/Running_OpenMPI.md
index 856a3e95e..37867fc39 100644
--- a/docs.it4i/salomon/software/mpi-1/Running_OpenMPI.md
+++ b/docs.it4i/salomon/software/mpi/Running_OpenMPI.md
@@ -1,14 +1,9 @@
-Running OpenMPI
+Running OpenMPI
===============
-
-
OpenMPI program execution
-------------------------
-
-The OpenMPI programs may be executed only via the PBS Workload manager,
-by entering an appropriate queue. On the cluster, the **OpenMPI 1.8.6**
-is OpenMPI based MPI implementation.
+The OpenMPI programs may be executed only via the PBS Workload manager, by entering an appropriate queue. On the cluster, the **OpenMPI 1.8.6** is OpenMPI based MPI implementation.
### Basic usage
@@ -16,6 +11,7 @@ Use the mpiexec to run the OpenMPI code.
Example:
+```bash
$ qsub -q qexp -l select=4:ncpus=24 -I
qsub: waiting for job 15210.isrv5 to start
qsub: job 15210.isrv5 ready
@@ -29,24 +25,16 @@ Example:
Hello world! from rank 1 of 4 on host r1i0n5
Hello world! from rank 2 of 4 on host r1i0n6
Hello world! from rank 3 of 4 on host r1i0n7
+```
-Please be aware, that in this example, the directive **-pernode** is
-used to run only **one task per node**, which is normally an unwanted
-behaviour (unless you want to run hybrid code with just one MPI and 24
-OpenMP tasks per node). In normal MPI programs **omit the -pernode
-directive** to run up to 24 MPI tasks per each node.
+Please be aware, that in this example, the directive **-pernode** is used to run only **one task per node**, which is normally an unwanted behaviour (unless you want to run hybrid code with just one MPI and 24 OpenMP tasks per node). In normal MPI programs **omit the -pernode directive** to run up to 24 MPI tasks per each node.
-In this example, we allocate 4 nodes via the express queue
-interactively. We set up the openmpi environment and interactively run
-the helloworld_mpi.x program.
-Note that the executable
-helloworld_mpi.x must be available within the
-same path on all nodes. This is automatically fulfilled on the /home and
-/scratch filesystem.
+In this example, we allocate 4 nodes via the express queue interactively. We set up the openmpi environment and interactively run the helloworld_mpi.x program.
+Note that the executable helloworld_mpi.x must be available within the same path on all nodes. This is automatically fulfilled on the /home and /scratch filesystem.
-You need to preload the executable, if running on the local ramdisk /tmp
-filesystem
+You need to preload the executable, if running on the local ramdisk /tmp filesystem
+```bash
$ pwd
/tmp/pbs.15210.isrv5
@@ -55,146 +43,134 @@ filesystem
Hello world! from rank 1 of 4 on host r1i0n5
Hello world! from rank 2 of 4 on host r1i0n6
Hello world! from rank 3 of 4 on host r1i0n7
+```
-In this example, we assume the executable
-helloworld_mpi.x is present on compute node
-r1i0n17 on ramdisk. We call the mpiexec whith the **--preload-binary**
-argument (valid for openmpi). The mpiexec will copy the executable from
-r1i0n17 to the /tmp/pbs.15210.isrv5
-directory on r1i0n5, r1i0n6 and r1i0n7 and execute the program.
+In this example, we assume the executable helloworld_mpi.x is present on compute node r1i0n17 on ramdisk. We call the mpiexec whith the **--preload-binary** argument (valid for openmpi). The mpiexec will copy the executable from r1i0n17 to the /tmp/pbs.15210.isrv5 directory on r1i0n5, r1i0n6 and r1i0n7 and execute the program.
MPI process mapping may be controlled by PBS parameters.
-The mpiprocs and ompthreads parameters allow for selection of number of
-running MPI processes per node as well as number of OpenMP threads per
-MPI process.
+The mpiprocs and ompthreads parameters allow for selection of number of running MPI processes per node as well as number of OpenMP threads per MPI process.
### One MPI process per node
-Follow this example to run one MPI process per node, 24 threads per
-process.Â
+Follow this example to run one MPI process per node, 24 threads per process.Â
+```bash
$ qsub -q qexp -l select=4:ncpus=24:mpiprocs=1:ompthreads=24 -I
$ module load OpenMPI
$ mpiexec --bind-to-none ./helloworld_mpi.x
+```
-In this example, we demonstrate recommended way to run an MPI
-application, using 1 MPI processes per node and 24 threads per socket,
-on 4 nodes.
+In this example, we demonstrate recommended way to run an MPI application, using 1 MPI processes per node and 24 threads per socket, on 4 nodes.
### Two MPI processes per node
-Follow this example to run two MPI processes per node, 8 threads per
-process. Note the options to mpiexec.
+Follow this example to run two MPI processes per node, 8 threads per process. Note the options to mpiexec.
+```bash
$ qsub -q qexp -l select=4:ncpus=24:mpiprocs=2:ompthreads=12 -I
$ module load OpenMPI
$ mpiexec -bysocket -bind-to-socket ./helloworld_mpi.x
+```
-In this example, we demonstrate recommended way to run an MPI
-application, using 2 MPI processes per node and 12 threads per socket,
-each process and its threads bound to a separate processor socket of the
-node, on 4 nodes
+In this example, we demonstrate recommended way to run an MPI application, using 2 MPI processes per node and 12 threads per socket, each process and its threads bound to a separate processor socket of the node, on 4 nodes
### 24 MPI processes per node
-Follow this example to run 24 MPI processes per node, 1 thread per
-process. Note the options to mpiexec.
+Follow this example to run 24 MPI processes per node, 1 thread per process. Note the options to mpiexec.
+```bash
$ qsub -q qexp -l select=4:ncpus=24:mpiprocs=24:ompthreads=1 -I
$ module load OpenMPI
$ mpiexec -bycore -bind-to-core ./helloworld_mpi.x
+```
-In this example, we demonstrate recommended way to run an MPI
-application, using 24 MPI processes per node, single threaded. Each
-process is bound to separate processor core, on 4 nodes.
+In this example, we demonstrate recommended way to run an MPI application, using 24 MPI processes per node, single threaded. Each process is bound to separate processor core, on 4 nodes.
### OpenMP thread affinity
-Important! Bind every OpenMP thread to a core!
+>Important! Bind every OpenMP thread to a core!
-In the previous two examples with one or two MPI processes per node, the
-operating system might still migrate OpenMP threads between cores. You
-might want to avoid this by setting these environment variable for GCC
-OpenMP:
+In the previous two examples with one or two MPI processes per node, the operating system might still migrate OpenMP threads between cores. You might want to avoid this by setting these environment variable for GCC OpenMP:
+```bash
$ export GOMP_CPU_AFFINITY="0-23"
+```
or this one for Intel OpenMP:
+```bash
$ export KMP_AFFINITY=granularity=fine,compact,1,0
+```
-As of OpenMP 4.0 (supported by GCC 4.9 and later and Intel 14.0 and
-later) the following variables may be used for Intel or GCC:
+As of OpenMP 4.0 (supported by GCC 4.9 and later and Intel 14.0 and later) the following variables may be used for Intel or GCC:
+```bash
$ export OMP_PROC_BIND=true
- $ export OMP_PLACES=coresÂ
+ $ export OMP_PLACES=cores
+```
OpenMPI Process Mapping and Binding
------------------------------------------------
+The mpiexec allows for precise selection of how the MPI processes will be mapped to the computational nodes and how these processes will bind to particular processor sockets and cores.
-The mpiexec allows for precise selection of how the MPI processes will
-be mapped to the computational nodes and how these processes will bind
-to particular processor sockets and cores.
-
-MPI process mapping may be specified by a hostfile or rankfile input to
-the mpiexec program. Altough all implementations of MPI provide means
-for process mapping and binding, following examples are valid for the
-openmpi only.
+MPI process mapping may be specified by a hostfile or rankfile input to the mpiexec program. Altough all implementations of MPI provide means for process mapping and binding, following examples are valid for the openmpi only.
### Hostfile
Example hostfile
+```bash
r1i0n17.smc.salomon.it4i.cz
r1i0n5.smc.salomon.it4i.cz
r1i0n6.smc.salomon.it4i.cz
r1i0n7.smc.salomon.it4i.cz
+```
Use the hostfile to control process placement
+```bash
$ mpiexec -hostfile hostfile ./helloworld_mpi.x
Hello world! from rank 0 of 4 on host r1i0n17
Hello world! from rank 1 of 4 on host r1i0n5
Hello world! from rank 2 of 4 on host r1i0n6
Hello world! from rank 3 of 4 on host r1i0n7
+```
-In this example, we see that ranks have been mapped on nodes according
-to the order in which nodes show in the hostfile
+In this example, we see that ranks have been mapped on nodes according to the order in which nodes show in the hostfile
### Rankfile
-Exact control of MPI process placement and resource binding is provided
-by specifying a rankfile
+Exact control of MPI process placement and resource binding is provided by specifying a rankfile
Appropriate binding may boost performance of your application.
Example rankfile
+```bash
rank 0=r1i0n7.smc.salomon.it4i.cz slot=1:0,1
rank 1=r1i0n6.smc.salomon.it4i.cz slot=0:*
rank 2=r1i0n5.smc.salomon.it4i.cz slot=1:1-2
rank 3=r1i0n17.smc.salomon slot=0:1,1:0-2
rank 4=r1i0n6.smc.salomon.it4i.cz slot=0:*,1:*
+```
-This rankfile assumes 5 ranks will be running on 4 nodes and provides
-exact mapping and binding of the processes to the processor sockets and
-cores
+This rankfile assumes 5 ranks will be running on 4 nodes and provides exact mapping and binding of the processes to the processor sockets and cores
Explanation:
rank 0 will be bounded to r1i0n7, socket1 core0 and core1
rank 1 will be bounded to r1i0n6, socket0, all cores
rank 2 will be bounded to r1i0n5, socket1, core1 and core2
-rank 3 will be bounded to r1i0n17, socket0 core1, socket1 core0, core1,
-core2
+rank 3 will be bounded to r1i0n17, socket0 core1, socket1 core0, core1, core2
rank 4 will be bounded to r1i0n6, all cores on both sockets
+```bash
$ mpiexec -n 5 -rf rankfile --report-bindings ./helloworld_mpi.x
[r1i0n17:11180] MCW rank 3 bound to socket 0[core 1] socket 1[core 0-2]: [. B . . . . . . . . . .][B B B . . . . . . . . .] (slot list 0:1,1:0-2)
[r1i0n7:09928] MCW rank 0 bound to socket 1[core 0-1]: [. . . . . . . . . . . .][B B . . . . . . . . . .] (slot list 1:0,1)
@@ -206,28 +182,24 @@ rank 4 will be bounded to r1i0n6, all cores on both sockets
Hello world! from rank 0 of 5 on host r1i0n7
Hello world! from rank 4 of 5 on host r1i0n6
Hello world! from rank 2 of 5 on host r1i0n5
+```
-In this example we run 5 MPI processes (5 ranks) on four nodes. The
-rankfile defines how the processes will be mapped on the nodes, sockets
-and cores. The **--report-bindings** option was used to print out the
-actual process location and bindings. Note that ranks 1 and 4 run on the
-same node and their core binding overlaps.
+In this example we run 5 MPI processes (5 ranks) on four nodes. The rankfile defines how the processes will be mapped on the nodes, sockets and cores. The **--report-bindings** option was used to print out the actual process location and bindings. Note that ranks 1 and 4 run on the same node and their core binding overlaps.
-It is users responsibility to provide correct number of ranks, sockets
-and cores.
+It is users responsibility to provide correct number of ranks, sockets and cores.
### Bindings verification
-In all cases, binding and threading may be verified by executing for
-example:
+In all cases, binding and threading may be verified by executing for example:
+```bash
$ mpiexec -bysocket -bind-to-socket --report-bindings echo
$ mpiexec -bysocket -bind-to-socket numactl --show
$ mpiexec -bysocket -bind-to-socket echo $OMP_NUM_THREADS
+```
Changes in OpenMPI 1.8
----------------------
-
Some options have changed in OpenMPI version 1.8.
|version 1.6.5 |version 1.8.1 |
@@ -237,5 +209,4 @@ Some options have changed in OpenMPI version 1.8.
|--bind-to-socket |--bind-to socket |
|-bysocket |--map-by socket |
|-bycore |--map-by core |
- |-pernode |--map-by ppr:1:node\ |
-
+ |-pernode |--map-by ppr:1:node |
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/mpi/mpi.md b/docs.it4i/salomon/software/mpi/mpi.md
new file mode 100644
index 000000000..61ee83071
--- /dev/null
+++ b/docs.it4i/salomon/software/mpi/mpi.md
@@ -0,0 +1,142 @@
+MPI
+===
+
+Setting up MPI Environment
+--------------------------
+
+The Salomon cluster provides several implementations of the MPI library:
+
+|MPI Library|Thread support|
+|---|---|---
+|**Intel MPI 4.1**|Full thread support up to, MPI_THREAD_MULTIPLE|
+|**Intel MPI 5.0**|Full thread support up to, MPI_THREAD_MULTIPLE|
+|OpenMPI 1.8.6|Full thread support up to, MPI_THREAD_MULTIPLE, MPI-3.0, support|
+|SGI MPT 2.12 ||
+
+MPI libraries are activated via the environment modules.
+
+Look up section modulefiles/mpi in module avail
+
+```bash
+ $ module avail
+ ------------------------------ /apps/modules/mpi -------------------------------
+ impi/4.1.1.036-iccifort-2013.5.192
+ impi/4.1.1.036-iccifort-2013.5.192-GCC-4.8.3
+ impi/5.0.3.048-iccifort-2015.3.187
+ impi/5.0.3.048-iccifort-2015.3.187-GNU-5.1.0-2.25
+ MPT/2.12
+ OpenMPI/1.8.6-GNU-5.1.0-2.25
+```
+
+There are default compilers associated with any particular MPI implementation. The defaults may be changed, the MPI libraries may be used in conjunction with any compiler. The defaults are selected via the modules in following way
+
+
+|Module|MPI|Compiler suite|
+|---|---|
+|impi-5.0.3.048-iccifort- Intel MPI 5.0.3| 2015.3.187||
+| OpenMP-1.8.6-GNU-5.1.0-2 OpenMPI 1.8.6| .25||
+
+Examples:
+
+```bash
+ $ module load gompi/2015b
+```
+
+In this example, we activate the latest OpenMPI with latest GNU compilers (OpenMPI 1.8.6 and GCC 5.1). Please see more information about toolchains in section [Environment and Modules](../../environment-and-modules.md) .
+
+To use OpenMPI with the intel compiler suite, use
+
+```bash
+ $ module load iompi/2015.03
+```
+
+In this example, the openmpi 1.8.6 using intel compilers is activated. It's used "iompi" toolchain.
+
+Compiling MPI Programs
+----------------------
+After setting up your MPI environment, compile your program using one of the mpi wrappers
+
+```bash
+ $ mpicc -v
+ $ mpif77 -v
+ $ mpif90 -v
+```
+
+When using Intel MPI, use the following MPI wrappers:
+
+```bash
+ $ mpicc
+ $ mpiifort
+```
+
+Wrappers mpif90, mpif77 that are provided by Intel MPI are designed for gcc and gfortran. You might be able to compile MPI code by them even with Intel compilers, but you might run into problems (for example, native MIC compilation with -mmic does not work with mpif90).
+
+Example program:
+
+```cpp
+ // helloworld_mpi.c
+ #include <stdio.h>
+
+ #include<mpi.h>
+
+ int main(int argc, char **argv) {
+
+ int len;
+ int rank, size;
+ char node[MPI_MAX_PROCESSOR_NAME];
+
+ // Initiate MPI
+ MPI_Init(&argc, &argv);
+ MPI_Comm_rank(MPI_COMM_WORLD,&rank);
+ MPI_Comm_size(MPI_COMM_WORLD,&size);
+
+ // Get hostame and print
+ MPI_Get_processor_name(node,&len);
+ printf("Hello world! from rank %d of %d on host %sn",rank,size,node);
+
+ // Finalize and exit
+ MPI_Finalize();
+
+ return 0;
+ }
+```
+
+Compile the above example with
+
+```bash
+ $ mpicc helloworld_mpi.c -o helloworld_mpi.x
+```
+
+Running MPI Programs
+--------------------
+The MPI program executable must be compatible with the loaded MPI module.
+Always compile and execute using the very same MPI module.
+
+It is strongly discouraged to mix mpi implementations. Linking an application with one MPI implementation and running mpirun/mpiexec form other implementation may result in unexpected errors.
+
+The MPI program executable must be available within the same path on all nodes. This is automatically fulfilled on the /home and /scratch filesystem. You need to preload the executable, if running on the local scratch /lscratch filesystem.
+
+### Ways to run MPI programs
+
+Optimal way to run an MPI program depends on its memory requirements, memory access pattern and communication pattern.
+
+Consider these ways to run an MPI program:
+1. One MPI process per node, 24 threads per process
+2. Two MPI processes per node, 12 threads per process
+3. 24 MPI processes per node, 1 thread per process.
+
+**One MPI** process per node, using 24 threads, is most useful for memory demanding applications, that make good use of processor cache memory and are not memory bound. This is also a preferred way for communication intensive applications as one process per node enjoys full bandwidth access to the network interface.
+
+**Two MPI** processes per node, using 12 threads each, bound to processor socket is most useful for memory bandwidth bound applications such as BLAS1 or FFT, with scalable memory demand. However, note that the two processes will share access to the network interface. The 12 threads and socket binding should ensure maximum memory access bandwidth and minimize communication, migration and numa effect overheads.
+
+>Important! Bind every OpenMP thread to a core!
+
+In the previous two cases with one or two MPI processes per node, the operating system might still migrate OpenMP threads between cores. You want to avoid this by setting the KMP_AFFINITY or GOMP_CPU_AFFINITY environment variables.
+
+**24 MPI** processes per node, using 1 thread each bound to processor core is most suitable for highly scalable applications with low communication demand.
+
+### Running OpenMPI
+
+The [**OpenMPI 1.8.6**](http://www.open-mpi.org/) is based on OpenMPI. Read more on [how to run OpenMPI](Running_OpenMPI.md) based MPI.
+
+The Intel MPI may run on the[Intel Xeon Ph](../intel-xeon-phi.md)i accelerators as well. Read more on [how to run Intel MPI on accelerators](../intel-xeon-phi.md).
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/mpi-1/mpi4py-mpi-for-python.md b/docs.it4i/salomon/software/mpi/mpi4py-mpi-for-python.md
similarity index 62%
rename from docs.it4i/salomon/software/mpi-1/mpi4py-mpi-for-python.md
rename to docs.it4i/salomon/software/mpi/mpi4py-mpi-for-python.md
index 00577440a..ab593f3d6 100644
--- a/docs.it4i/salomon/software/mpi-1/mpi4py-mpi-for-python.md
+++ b/docs.it4i/salomon/software/mpi/mpi4py-mpi-for-python.md
@@ -1,59 +1,50 @@
-MPI4Py (MPI for Python)
+MPI4Py (MPI for Python)
=======================
OpenMPI interface to Python
-
-
Introduction
------------
+MPI for Python provides bindings of the Message Passing Interface (MPI) standard for the Python programming language, allowing any Python program to exploit multiple processors.
-MPI for Python provides bindings of the Message Passing Interface (MPI)
-standard for the Python programming language, allowing any Python
-program to exploit multiple processors.
-
-This package is constructed on top of the MPI-1/2 specifications and
-provides an object oriented interface which closely follows MPI-2 C++
-bindings. It supports point-to-point (sends, receives) and collective
-(broadcasts, scatters, gathers) communications of any picklable Python
-object, as well as optimized communications of Python object exposing
-the single-segment buffer interface (NumPy arrays, builtin
-bytes/string/array objects).
+This package is constructed on top of the MPI-1/2 specifications and provides an object oriented interface which closely follows MPI-2 C++ bindings. It supports point-to-point (sends, receives) and collective (broadcasts, scatters, gathers) communications of any picklable Python object, as well as optimized communications of Python object exposing the single-segment buffer interface (NumPy arrays, builtin bytes/string/array objects).
On Anselm MPI4Py is available in standard Python modules.
Modules
-------
+MPI4Py is build for OpenMPI. Before you start with MPI4Py you need to load Python and OpenMPI modules. You can use toolchain, that loads Python and OpenMPI at once.
-MPI4Py is build for OpenMPI. Before you start with MPI4Py you need to
-load Python and OpenMPI modules. You can use toolchain, that loads
-Python and OpenMPI at once.
-
+```bash
$ module load Python/2.7.9-foss-2015g
+```
Execution
---------
+You need to import MPI to your python program. Include the following line to the python script:
-You need to import MPI to your python program. Include the following
-line to the python script:
-
+```bash
from mpi4py import MPI
+```
-The MPI4Py enabled python programs [execute as any other
-OpenMPI](Running_OpenMPI.html) code.The simpliest way is
-to run
+The MPI4Py enabled python programs [execute as any other OpenMPI](Running_OpenMPI.md) code.The simpliest way is to run
+```bash
$ mpiexec python <script>.py
+```
For example
+```bash
$ mpiexec python hello_world.py
+```
Examples
--------
### Hello world!
+```cpp
from mpi4py import MPI
comm = MPI.COMM_WORLD
@@ -61,9 +52,11 @@ Examples
print "Hello! I'm rank %d from %d running in total..." % (comm.rank, comm.size)
comm.Barrier() Â # wait for everybody to synchronize
+```
###Collective Communication with NumPy arrays
+```cpp
from __future__ import division
from mpi4py import MPI
import numpy as np
@@ -88,18 +81,16 @@ Examples
# Everybody should now have the same...
print "[%02d] %s" % (comm.rank, A)
+```
Execute the above code as:
+```bash
$ qsub -q qexp -l select=4:ncpus=24:mpiprocs=24:ompthreads=1 -I
$ module load Python/2.7.9-foss-2015g
$ mpiexec --map-by core --bind-to core python hello_world.py
+```
-In this example, we run MPI4Py enabled code on 4 nodes, 24 cores per
-node (total of 96 processes), each python process is bound to a
-different core.
-More examples and documentation can be found on [MPI for Python
-webpage](https://pythonhosted.org/mpi4py/usrman/index.html).
-
+In this example, we run MPI4Py enabled code on 4 nodes, 24 cores per node (total of 96 processes), each python process is bound to a different core. More examples and documentation can be found on [MPI for Python webpage](https://pythonhosted.org/mpi4py/usrman/index.md).
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/numerical-languages/introduction.md b/docs.it4i/salomon/software/numerical-languages/introduction.md
index 6416c41fd..1bc6e2628 100644
--- a/docs.it4i/salomon/software/numerical-languages/introduction.md
+++ b/docs.it4i/salomon/software/numerical-languages/introduction.md
@@ -1,48 +1,42 @@
-Numerical languages
+Numerical languages
===================
Interpreted languages for numerical computations and analysis
-
-
Introduction
------------
-
-This section contains a collection of high-level interpreted languages,
-primarily intended for numerical computations.
+This section contains a collection of high-level interpreted languages, primarily intended for numerical computations.
Matlab
------
+MATLAB®^ is a high-level language and interactive environment for numerical computation, visualization, and programming.
-MATLAB®^ is a high-level language and interactive environment for
-numerical computation, visualization, and programming.
-
+```bash
$ module load MATLAB
$ matlab
+```
-Read more at the [Matlab
-page](matlab.html).
+Read more at the [Matlab page](matlab.md).
Octave
------
+GNU Octave is a high-level interpreted language, primarily intended for numerical computations. The Octave language is quite similar to Matlab so that most programs are easily portable.
-GNU Octave is a high-level interpreted language, primarily intended for
-numerical computations. The Octave language is quite similar to Matlab
-so that most programs are easily portable.
-
+```bash
$ module load Octave
$ octave
+```
-Read more at the [Octave page](octave.html).
+Read more at the [Octave page](octave.md).
R
--
+---
-The R is an interpreted language and environment for statistical
-computing and graphics.
+The R is an interpreted language and environment for statistical computing and graphics.
+```bash
$ module load R
$ R
+```
-Read more at the [R page](r.html).
-
+Read more at the [R page](r.md).
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/numerical-languages/matlab.md b/docs.it4i/salomon/software/numerical-languages/matlab.md
index c713cd7c9..d78bdce8c 100644
--- a/docs.it4i/salomon/software/numerical-languages/matlab.md
+++ b/docs.it4i/salomon/software/numerical-languages/matlab.md
@@ -1,121 +1,95 @@
-Matlab
+Matlab
======
-
-
Introduction
------------
+Matlab is available in versions R2015a and R2015b. There are always two variants of the release:
-Matlab is available in versions R2015a and R2015b. There are always two
-variants of the release:
-
-- Non commercial or so called EDU variant, which can be used for
- common research and educational purposes.
-- Commercial or so called COM variant, which can used also for
- commercial activities. The licenses for commercial variant are much
- more expensive, so usually the commercial variant has only subset of
- features compared to the EDU available.
-
-Â
+- Non commercial or so called EDU variant, which can be used for common research and educational purposes.
+- Commercial or so called COM variant, which can used also for commercial activities. The licenses for commercial variant are much more expensive, so usually the commercial variant has only subset of features compared to the EDU available.
To load the latest version of Matlab load the module
+```bash
$ module load MATLAB
+```
-By default the EDU variant is marked as default. If you need other
-version or variant, load the particular version. To obtain the list of
-available versions use
+By default the EDU variant is marked as default. If you need other version or variant, load the particular version. To obtain the list of available versions use
+```bash
$ module avail MATLAB
+```
-If you need to use the Matlab GUI to prepare your Matlab programs, you
-can use Matlab directly on the login nodes. But for all computations use
-Matlab on the compute nodes via PBS Pro scheduler.
+If you need to use the Matlab GUI to prepare your Matlab programs, you can use Matlab directly on the login nodes. But for all computations use Matlab on the compute nodes via PBS Pro scheduler.
-If you require the Matlab GUI, please follow the general informations
-about [running graphical
-applications](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-system/x-window-and-vnc.html).
+If you require the Matlab GUI, please follow the general informations about [running graphical applications](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-and-vnc.md).
-Matlab GUI is quite slow using the X forwarding built in the PBS (qsub
--X), so using X11 display redirection either via SSH or directly by
-xauth (please see the "GUI Applications on Compute Nodes over VNC" part
-[here](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-system/x-window-and-vnc.html))
-is recommended.
+Matlab GUI is quite slow using the X forwarding built in the PBS (qsub -X), so using X11 display redirection either via SSH or directly by xauth (please see the "GUI Applications on Compute Nodes over VNC" part [here](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-and-vnc.md)) is recommended.
To run Matlab with GUI, use
+```bash
$ matlab
+```
-To run Matlab in text mode, without the Matlab Desktop GUI environment,
-use
+To run Matlab in text mode, without the Matlab Desktop GUI environment, use
+```bash
$ matlab -nodesktop -nosplash
+```
plots, images, etc... will be still available.
Running parallel Matlab using Distributed Computing Toolbox / Engine
------------------------------------------------------------------------
-
Distributed toolbox is available only for the EDU variant
-The MPIEXEC mode available in previous versions is no longer available
-in MATLAB 2015. Also, the programming interface has changed. Refer
-to [Release
-Notes](http://www.mathworks.com/help/distcomp/release-notes.html#buanp9e-1).
+The MPIEXEC mode available in previous versions is no longer available in MATLAB 2015. Also, the programming interface has changed. Refer to [Release Notes](http://www.mathworks.com/help/distcomp/release-notes.html#buanp9e-1).
-Delete previously used file mpiLibConf.m, we have observed crashes when
-using Intel MPI.
+Delete previously used file mpiLibConf.m, we have observed crashes when using Intel MPI.
-To use Distributed Computing, you first need to setup a parallel
-profile. We have provided the profile for you, you can either import it
-in MATLAB command line:
+To use Distributed Computing, you first need to setup a parallel profile. We have provided the profile for you, you can either import it in MATLAB command line:
+```bash
> parallel.importProfile('/apps/all/MATLAB/2015b-EDU/SalomonPBSPro.settings')
- ans =
+ ans =
- SalomonPBSProÂ
+ SalomonPBSPro
+```
-Or in the GUI, go to tab HOME -> Parallel -> Manage Cluster
-Profiles..., click Import and navigate to :
+Or in the GUI, go to tab HOME -> Parallel -> Manage Cluster Profiles..., click Import and navigate to :
/apps/all/MATLAB/2015b-EDU/SalomonPBSPro.settings
-With the new mode, MATLAB itself launches the workers via PBS, so you
-can either use interactive mode or a batch mode on one node, but the
-actual parallel processing will be done in a separate job started by
-MATLAB itself. Alternatively, you can use "local" mode to run parallel
-code on just a single node.
+With the new mode, MATLAB itself launches the workers via PBS, so you can either use interactive mode or a batch mode on one node, but the actual parallel processing will be done in a separate job started by MATLAB itself. Alternatively, you can use "local" mode to run parallel code on just a single node.
### Parallel Matlab interactive session
-Following example shows how to start interactive session with support
-for Matlab GUI. For more information about GUI based applications on
-Anselm see [this
-page](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-system/x-window-and-vnc.html).
+Following example shows how to start interactive session with support for Matlab GUI. For more information about GUI based applications on Anselm see [this page](../../../get-started-with-it4innovations/accessing-the-clusters/graphical-user-interface/x-window-and-vnc.md).
+```bash
$ xhost +
- $ qsub -I -v DISPLAY=$(uname -n):$(echo $DISPLAY | cut -d ':' -f 2) -A NONE-0-0 -q qexp -l select=1 -l walltime=00:30:00
- -l feature__matlab__MATLAB=1
+ $ qsub -I -v DISPLAY=$(uname -n):$(echo $DISPLAY | cut -d ':' -f 2) -A NONE-0-0 -q qexp -l select=1 -l walltime=00:30:00
+ -l feature__matlab__MATLAB=1
+```
This qsub command example shows how to run Matlab on a single node.
-The second part of the command shows how to request all necessary
-licenses. In this case 1 Matlab-EDU license and 48 Distributed Computing
-Engines licenses.
+The second part of the command shows how to request all necessary licenses. In this case 1 Matlab-EDU license and 48 Distributed Computing Engines licenses.
-Once the access to compute nodes is granted by PBS, user can load
-following modules and start Matlab:Â
+Once the access to compute nodes is granted by PBS, user can load following modules and start Matlab:
+```bash
r1i0n17$ module load MATLAB/2015a-EDU
r1i0n17$ matlab &
+```
### Parallel Matlab batch job in Local mode
-To run matlab in batch mode, write an matlab script, then write a bash
-jobscript and execute via the qsub command. By default, matlab will
-execute one matlab worker instance per allocated core.
+To run matlab in batch mode, write an matlab script, then write a bash jobscript and execute via the qsub command. By default, matlab will execute one matlab worker instance per allocated core.
+```bash
#!/bin/bash
#PBS -A PROJECT ID
#PBS -q qprod
@@ -125,7 +99,7 @@ execute one matlab worker instance per allocated core.
SCR=/scratch/work/user/$USER/$PBS_JOBID
mkdir -p $SCR ; cd $SCR || exit
- # copy input file to scratch
+ # copy input file to scratch
cp $PBS_O_WORKDIR/matlabcode.m .
# load modules
@@ -136,46 +110,43 @@ execute one matlab worker instance per allocated core.
# copy output file to home
cp output.out $PBS_O_WORKDIR/.
+```
-This script may be submitted directly to the PBS workload manager via
-the qsub command. The inputs and matlab script are in matlabcode.m
-file, outputs in output.out file. Note the missing .m extension in the
-matlab -r matlabcodefile call, **the .m must not be included**. Note
-that the **shared /scratch must be used**. Further, it is **important to
-include quit** statement at the end of the matlabcode.m script.
+This script may be submitted directly to the PBS workload manager via the qsub command. The inputs and matlab script are in matlabcode.m file, outputs in output.out file. Note the missing .m extension in the matlab -r matlabcodefile call, **the .m must not be included**. Note that the **shared /scratch must be used**. Further, it is **important to include quit** statement at the end of the matlabcode.m script.
Submit the jobscript using qsub
+```bash
$ qsub ./jobscript
+```
### Parallel Matlab Local mode program example
-The last part of the configuration is done directly in the user Matlab
-script before Distributed Computing Toolbox is started.
+The last part of the configuration is done directly in the user Matlab script before Distributed Computing Toolbox is started.
+```bash
cluster = parcluster('local')
+```
-This script creates scheduler object "cluster" of type "local" that
-starts workers locally.Â
+This script creates scheduler object "cluster" of type "local" that starts workers locally.
-Please note: Every Matlab script that needs to initialize/use matlabpool
-has to contain these three lines prior to calling parpool(sched, ...)
-function.Â
+Please note: Every Matlab script that needs to initialize/use matlabpool has to contain these three lines prior to calling parpool(sched, ...) function.
-The last step is to start matlabpool with "cluster" object and correct
-number of workers. We have 24 cores per node, so we start 24 workers.
+The last step is to start matlabpool with "cluster" object and correct number of workers. We have 24 cores per node, so we start 24 workers.
+```bash
parpool(cluster,24);
-
-
+
+
... parallel code ...
-
-
+
+
parpool close
+```
-The complete example showing how to use Distributed Computing Toolbox in
-local mode is shown here.Â
+The complete example showing how to use Distributed Computing Toolbox in local mode is shown here.
+```bash
cluster = parcluster('local');
cluster
@@ -189,7 +160,7 @@ local mode is shown here.Â
x = distributed(x);
spmd
[~, name] = system('hostname')
- Â Â Â
+
   T = W*x; % Calculation performed on labs, in parallel.
            % T and W are both codistributed arrays here.
end
@@ -198,20 +169,17 @@ local mode is shown here.Â
parpool close
quit
+```
-You can copy and paste the example in a .m file and execute. Note that
-the parpool size should correspond to **total number of cores**
-available on allocated nodes.
+You can copy and paste the example in a .m file and execute. Note that the parpool size should correspond to **total number of cores** available on allocated nodes.
### Parallel Matlab Batch job using PBS mode (workers spawned in a separate job)
-This mode uses PBS scheduler to launch the parallel pool. It uses the
-SalomonPBSPro profile that needs to be imported to Cluster Manager, as
-mentioned before. This methodod uses MATLAB's PBS Scheduler interface -
-it spawns the workers in a separate job submitted by MATLAB using qsub.
+This mode uses PBS scheduler to launch the parallel pool. It uses the SalomonPBSPro profile that needs to be imported to Cluster Manager, as mentioned before. This methodod uses MATLAB's PBS Scheduler interface - it spawns the workers in a separate job submitted by MATLAB using qsub.
This is an example of m-script using PBS mode:
+```bash
cluster = parcluster('SalomonPBSPro');
set(cluster, 'SubmitArguments', '-A OPEN-0-0');
set(cluster, 'ResourceTemplate', '-q qprod -l select=10:ncpus=24');
@@ -235,33 +203,23 @@ This is an example of m-script using PBS mode:
% shut down parallel pool
delete(pool)
+```
-Note that we first construct a cluster object using the imported
-profile, then set some important options, namely : SubmitArguments,
-where you need to specify accounting id, and ResourceTemplate, where you
-need to specify number of nodes to run the job.Â
+Note that we first construct a cluster object using the imported profile, then set some important options, namely : SubmitArguments, where you need to specify accounting id, and ResourceTemplate, where you need to specify number of nodes to run the job.
-You can start this script using batch mode the same way as in Local mode
-example.
+You can start this script using batch mode the same way as in Local mode example.
### Parallel Matlab Batch with direct launch (workers spawned within the existing job)
-This method is a "hack" invented by us to emulate the mpiexec
-functionality found in previous MATLAB versions. We leverage the MATLAB
-Generic Scheduler interface, but instead of submitting the workers to
-PBS, we launch the workers directly within the running job, thus we
-avoid the issues with master script and workers running in separate jobs
-(issues with license not available, waiting for the worker's job to
-spawn etc.)
+This method is a "hack" invented by us to emulate the mpiexec functionality found in previous MATLAB versions. We leverage the MATLAB Generic Scheduler interface, but instead of submitting the workers to PBS, we launch the workers directly within the running job, thus we avoid the issues with master script and workers running in separate jobs (issues with license not available, waiting for the worker's job to spawn etc.)
Please note that this method is experimental.
-For this method, you need to use SalomonDirect profile, import it
-using [the same way as
-SalomonPBSPro](matlab.html#running-parallel-matlab-using-distributed-computing-toolbox---engine)Â
+For this method, you need to use SalomonDirect profile, import it using [the same way as SalomonPBSPro](matlab.md#running-parallel-matlab-using-distributed-computing-toolbox---engine)
This is an example of m-script using direct mode:
+```bash
parallel.importProfile('/apps/all/MATLAB/2015b-EDU/SalomonDirect.settings')
cluster = parcluster('SalomonDirect');
set(cluster, 'NumWorkers', 48);
@@ -284,27 +242,19 @@ This is an example of m-script using direct mode:
% shut down parallel pool
delete(pool)
+```
### Non-interactive Session and Licenses
-If you want to run batch jobs with Matlab, be sure to request
-appropriate license features with the PBS Pro scheduler, at least the "
--l __feature__matlab__MATLAB=1" for EDU variant of Matlab. More
-information about how to check the license features states and how to
-request them with PBS Pro, please [look
-here](../../../anselm-cluster-documentation/software/isv_licenses.html).
+If you want to run batch jobs with Matlab, be sure to request appropriate license features with the PBS Pro scheduler, at least the " -l __feature__matlab__MATLAB=1" for EDU variant of Matlab. More information about how to check the license features states and how to request them with PBS Pro, please [look here](../../../anselm-cluster-documentation/software/isv_licenses.md).
The licensing feature of PBS is currently disabled.
-In case of non-interactive session please read the [following
-information](../../../anselm-cluster-documentation/software/isv_licenses.html)
-on how to modify the qsub command to test for available licenses prior
-getting the resource allocation.
+In case of non-interactive session please read the [following information](../../../anselm-cluster-documentation/software/isv_licenses.md) on how to modify the qsub command to test for available licenses prior getting the resource allocation.
### Matlab Distributed Computing Engines start up time
-Starting Matlab workers is an expensive process that requires certain
-amount of time. For your information please see the following table:
+Starting Matlab workers is an expensive process that requires certain amount of time. For your information please see the following table:
|compute nodes|number of workers|start-up time[s]|
|---|---|---|
@@ -313,33 +263,16 @@ amount of time. For your information please see the following table:
|4|96|483|
|2|48|16|
-MATLAB on UV2000Â
+MATLAB on UV2000
-----------------
+UV2000 machine available in queue "qfat" can be used for MATLAB computations. This is a SMP NUMA machine with large amount of RAM, which can be beneficial for certain types of MATLAB jobs. CPU cores are allocated in chunks of 8 for this machine.
-UV2000 machine available in queue "qfat" can be used for MATLAB
-computations. This is a SMP NUMA machine with large amount of RAM, which
-can be beneficial for certain types of MATLAB jobs. CPU cores are
-allocated in chunks of 8 for this machine.
-
-You can use MATLAB on UV2000 in two parallel modes :
+You can use MATLAB on UV2000 in two parallel modes:
### Threaded mode
-Since this is a SMP machine, you can completely avoid using Parallel
-Toolbox and use only MATLAB's threading. MATLAB will automatically
-detect the number of cores you have allocated and will setÂ
-maxNumCompThreads accordingly and certain
-operations, such as fft, , eig, svd,
-etc. will be automatically run in threads. The advantage of this mode is
-that you don't need to modify your existing sequential codes.
+Since this is a SMP machine, you can completely avoid using Parallel Toolbox and use only MATLAB's threading. MATLAB will automatically detect the number of cores you have allocated and will set maxNumCompThreads accordingly and certain operations, such as fft, , eig, svd, etc. will be automatically run in threads. The advantage of this mode is that you don't need to modify your existing sequential codes.
### Local cluster mode
-You can also use Parallel Toolbox on UV2000. Use l[ocal cluster
-mode](matlab.html#parallel-matlab-batch-job-in-local-mode),
-"SalomonPBSPro" profile will not work.
-
-Â
-
-Â
-
+You can also use Parallel Toolbox on UV2000. Use l[ocal cluster mode](matlab.md#parallel-matlab-batch-job-in-local-mode), "SalomonPBSPro" profile will not work.
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/numerical-languages/octave.md b/docs.it4i/salomon/software/numerical-languages/octave.md
index 6bb5f2d84..3c598b37c 100644
--- a/docs.it4i/salomon/software/numerical-languages/octave.md
+++ b/docs.it4i/salomon/software/numerical-languages/octave.md
@@ -1,44 +1,29 @@
-Octave
+Octave
======
-
-
-GNU Octave is a high-level interpreted language, primarily intended for
-numerical computations. It provides capabilities for the numerical
-solution of linear and nonlinear problems, and for performing other
-numerical experiments. It also provides extensive graphics capabilities
-for data visualization and manipulation. Octave is normally used through
-its interactive command line interface, but it can also be used to write
-non-interactive programs. The Octave language is quite similar to Matlab
-so that most programs are easily portable. Read more on
-<http://www.gnu.org/software/octave/>***
+GNU Octave is a high-level interpreted language, primarily intended for numerical computations. It provides capabilities for the numerical solution of linear and nonlinear problems, and for performing other numerical experiments. It also provides extensive graphics capabilities for data visualization and manipulation. Octave is normally used through its interactive command line interface, but it can also be used to write non-interactive programs. The Octave language is quite similar to Matlab so that most programs are easily portable. Read more on <http://www.gnu.org/software/octave/>
Two versions of octave are available on the cluster, via module
- Status Version module
- ------ |---|---|---- --------
- **Stable** Octave 3.8.2 Octave
-
-Â
+ |Status | Version | module|
+|---|---|
+ |**Stable** | Octave 3.8.2 | Octave|
- $ module load Octave
+```bash
+ $ module load Octave
+```
-The octave on the cluster is linked to highly optimized MKL mathematical
-library. This provides threaded parallelization to many octave kernels,
-notably the linear algebra subroutines. Octave runs these heavy
-calculation kernels without any penalty. By default, octave would
-parallelize to 24 threads. You may control the threads by setting the
-OMP_NUM_THREADS environment variable.
+The octave on the cluster is linked to highly optimized MKL mathematical library. This provides threaded parallelization to many octave kernels, notably the linear algebra subroutines. Octave runs these heavy calculation kernels without any penalty. By default, octave would parallelize to 24 threads. You may control the threads by setting the OMP_NUM_THREADS environment variable.
-To run octave interactively, log in with ssh -X parameter for X11
-forwarding. Run octave:
+To run octave interactively, log in with ssh -X parameter for X11 forwarding. Run octave:
+```bash
$ octave
+```
-To run octave in batch mode, write an octave script, then write a bash
-jobscript and execute via the qsub command. By default, octave will use
-16 threads when running MKL kernels.
+To run octave in batch mode, write an octave script, then write a bash jobscript and execute via the qsub command. By default, octave will use 16 threads when running MKL kernels.
+```bash
#!/bin/bash
# change to local scratch directory
@@ -59,21 +44,14 @@ jobscript and execute via the qsub command. By default, octave will use
#exit
exit
+```
-This script may be submitted directly to the PBS workload manager via
-the qsub command. The inputs are in octcode.m file, outputs in
-output.out file. See the single node jobscript example in the [Job
-execution
-section](../../resource-allocation-and-job-execution.html).
-
-The octave c compiler mkoctfile calls the GNU gcc 4.8.1 for compiling
-native c code. This is very useful for running native c subroutines in
-octave environment.
+This script may be submitted directly to the PBS workload manager via the qsub command. The inputs are in octcode.m file, outputs in output.out file. See the single node jobscript example in the [Job execution section](../../resource-allocation-and-job-execution.md).
-$ mkoctfile -v
+The octave c compiler mkoctfile calls the GNU gcc 4.8.1 for compiling native c code. This is very useful for running native c subroutines in octave environment.
-Octave may use MPI for interprocess communication
-This functionality is currently not supported on the cluster cluster. In
-case you require the octave interface to MPI, please contact our
-[cluster support](https://support.it4i.cz/rt/).
+```bash
+ $ mkoctfile -v
+```
+Octave may use MPI for interprocess communication This functionality is currently not supported on the cluster cluster. In case you require the octave interface to MPI, please contact our [cluster support](https://support.it4i.cz/rt/).
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/numerical-languages/r.md b/docs.it4i/salomon/software/numerical-languages/r.md
index 85e34b691..5617640dd 100644
--- a/docs.it4i/salomon/software/numerical-languages/r.md
+++ b/docs.it4i/salomon/software/numerical-languages/r.md
@@ -1,76 +1,58 @@
-R
-=
+R
+===
-
-
-Introduction
+Introduction
------------
+The R is a language and environment for statistical computing and graphics. R provides a wide variety of statistical (linear and nonlinear modelling, classical statistical tests, time-series analysis, classification, clustering, ...) and graphical techniques, and is highly extensible.
-The R is a language and environment for statistical computing and
-graphics. R provides a wide variety of statistical (linear and
-nonlinear modelling, classical statistical tests, time-series analysis,
-classification, clustering, ...) and graphical techniques, and is highly
-extensible.
-
-One of R's strengths is the ease with which well-designed
-publication-quality plots can be produced, including mathematical
-symbols and formulae where needed. Great care has been taken over the
-defaults for the minor design choices in graphics, but the user retains
-full control.
+One of R's strengths is the ease with which well-designed publication-quality plots can be produced, including mathematical symbols and formulae where needed. Great care has been taken over the defaults for the minor design choices in graphics, but the user retains full control.
-Another convenience is the ease with which the C code or third party
-libraries may be integrated within R.
+Another convenience is the ease with which the C code or third party libraries may be integrated within R.
Extensive support for parallel computing is available within R.
-Read more on <http://www.r-project.org/>,
-<http://cran.r-project.org/doc/manuals/r-release/R-lang.html>
+Read more on <http://www.r-project.org/>, <http://cran.r-project.org/doc/manuals/r-release/R-lang.html>
Modules
-------
+**The R version 3.1.1 is available on the cluster, along with GUI interface Rstudio**
-**The R version 3.1.1 is available on the cluster, along with GUI
-interface Rstudio**
-
- |Application|Version|module|
- ------- |---|---|---- ---------------------
- |**R**|R 3.1.1|R/3.1.1-intel-2015b|
- |**Rstudio**|Rstudio 0.97|Rstudio|
+|Application|Version|module|
+|---|---|
+|**R**|R 3.1.1|R/3.1.1-intel-2015b|
+|**Rstudio**|Rstudio 0.97|Rstudio|
+```bash
$ module load R
+```
Execution
---------
-The R on Anselm is linked to highly optimized MKL mathematical
-library. This provides threaded parallelization to many R kernels,
-notably the linear algebra subroutines. The R runs these heavy
-calculation kernels without any penalty. By default, the R would
-parallelize to 24 threads. You may control the threads by setting the
-OMP_NUM_THREADS environment variable.
+The R on Anselm is linked to highly optimized MKL mathematical library. This provides threaded parallelization to many R kernels, notably the linear algebra subroutines. The R runs these heavy calculation kernels without any penalty. By default, the R would parallelize to 24 threads. You may control the threads by setting the OMP_NUM_THREADS environment variable.
### Interactive execution
-To run R interactively, using Rstudio GUI, log in with ssh -X parameter
-for X11 forwarding. Run rstudio:
+To run R interactively, using Rstudio GUI, log in with ssh -X parameter for X11 forwarding. Run rstudio:
+```bash
$ module load Rstudio
$ rstudio
+```
### Batch execution
-To run R in batch mode, write an R script, then write a bash jobscript
-and execute via the qsub command. By default, R will use 24 threads when
-running MKL kernels.
+To run R in batch mode, write an R script, then write a bash jobscript and execute via the qsub command. By default, R will use 24 threads when running MKL kernels.
Example jobscript:
+```bash
#!/bin/bash
# change to local scratch directory
cd /lscratch/$PBS_JOBID || exit
- # copy input file to scratch
+ # copy input file to scratch
cp $PBS_O_WORKDIR/rscript.R .
# load R module
@@ -84,57 +66,45 @@ Example jobscript:
#exit
exit
+```
-This script may be submitted directly to the PBS workload manager via
-the qsub command. The inputs are in rscript.R file, outputs in
-routput.out file. See the single node jobscript example in the [Job
-execution
-section](../../resource-allocation-and-job-execution/job-submission-and-execution.html).
+This script may be submitted directly to the PBS workload manager via the qsub command. The inputs are in rscript.R file, outputs in routput.out file. See the single node jobscript example in the [Job execution section](../../resource-allocation-and-job-execution/job-submission-and-execution.md).
Parallel R
----------
-
-Parallel execution of R may be achieved in many ways. One approach is
-the implied parallelization due to linked libraries or specially enabled
-functions, as [described
-above](r.html#interactive-execution). In the following
-sections, we focus on explicit parallelization, where parallel
-constructs are directly stated within the R script.
+Parallel execution of R may be achieved in many ways. One approach is the implied parallelization due to linked libraries or specially enabled functions, as [described above](r.md#interactive-execution). In the following sections, we focus on explicit parallelization, where parallel constructs are directly stated within the R script.
Package parallel
--------------------
-
-The package parallel provides support for parallel computation,
-including by forking (taken from package multicore), by sockets (taken
-from package snow) and random-number generation.
+The package parallel provides support for parallel computation, including by forking (taken from package multicore), by sockets (taken from package snow) and random-number generation.
The package is activated this way:
+```bash
$ R
> library(parallel)
+```
-More information and examples may be obtained directly by reading the
-documentation available in R
+More information and examples may be obtained directly by reading the documentation available in R
+```bash
> ?parallel
> library(help = "parallel")
> vignette("parallel")
+```
-Download the package
-[parallell](package-parallel-vignette) vignette.
+Download the package [parallell](package-parallel-vignette) vignette.
-The forking is the most simple to use. Forking family of functions
-provide parallelized, drop in replacement for the serial apply() family
-of functions.
+The forking is the most simple to use. Forking family of functions provide parallelized, drop in replacement for the serial apply() family of functions.
-Forking via package parallel provides functionality similar to OpenMP
-construct
-#omp parallel for
+>Forking via package parallel provides functionality similar to OpenMP construct
+omp parallel for
-Only cores of single node can be utilized this way!
+>Only cores of single node can be utilized this way!
Forking example:
+```cpp
library(parallel)
#integrand function
@@ -164,47 +134,40 @@ Forking example:
#print results
cat(sprintf("Value of PI %16.14f, diff= %16.14fn",pi3,pi3-pi))
}
+```
-The above example is the classic parallel example for calculating the
-number π. Note the **detectCores()** and **mclapply()** functions.
-Execute the example as:
+The above example is the classic parallel example for calculating the number π. Note the **detectCores()** and **mclapply()** functions. Execute the example as:
+```bash
$ R --slave --no-save --no-restore -f pi3p.R
+```
-Every evaluation of the integrad function runs in parallel on different
-process.
+Every evaluation of the integrad function runs in parallel on different process.
Package Rmpi
------------
-
package Rmpi provides an interface (wrapper) to MPI APIs.
-It also provides interactive R slave environment. On the cluster, Rmpi
-provides interface to the
-[OpenMPI](../mpi-1/Running_OpenMPI.html).
+It also provides interactive R slave environment. On the cluster, Rmpi provides interface to the [OpenMPI](../mpi/Running_OpenMPI.md).
-Read more on Rmpi at <http://cran.r-project.org/web/packages/Rmpi/>,
-reference manual is available at
-<http://cran.r-project.org/web/packages/Rmpi/Rmpi.pdf>
+Read more on Rmpi at <http://cran.r-project.org/web/packages/Rmpi/>, reference manual is available at <http://cran.r-project.org/web/packages/Rmpi/Rmpi.pdf>
When using package Rmpi, both openmpi and R modules must be loaded
+```bash
$ module load OpenMPI
$ module load R
+```
-Rmpi may be used in three basic ways. The static approach is identical
-to executing any other MPI programm. In addition, there is Rslaves
-dynamic MPI approach and the mpi.apply approach. In the following
-section, we will use the number π integration example, to illustrate all
-these concepts.
+Rmpi may be used in three basic ways. The static approach is identical to executing any other MPI programm. In addition, there is Rslaves dynamic MPI approach and the mpi.apply approach. In the following section, we will use the number π integration example, to illustrate all these concepts.
### static Rmpi
-Static Rmpi programs are executed via mpiexec, as any other MPI
-programs. Number of processes is static - given at the launch time.
+Static Rmpi programs are executed via mpiexec, as any other MPI programs. Number of processes is static - given at the launch time.
Static Rmpi example:
+```cpp
library(Rmpi)
#integrand function
@@ -246,21 +209,21 @@ Static Rmpi example:
}
mpi.quit()
+```
-The above is the static MPI example for calculating the number π. Note
-the **library(Rmpi)** and **mpi.comm.dup()** function calls.
-Execute the example as:
+The above is the static MPI example for calculating the number π. Note the **library(Rmpi)** and **mpi.comm.dup()** function calls. Execute the example as:
+```bash
$ mpirun R --slave --no-save --no-restore -f pi3.R
+```
### dynamic Rmpi
-Dynamic Rmpi programs are executed by calling the R directly. OpenMPI
-module must be still loaded. The R slave processes will be spawned by a
-function call within the Rmpi program.
+Dynamic Rmpi programs are executed by calling the R directly. OpenMPI module must be still loaded. The R slave processes will be spawned by a function call within the Rmpi program.
Dynamic Rmpi example:
+```cpp
#integrand function
f <- function(i,h) {
x <- h*(i-0.5)
@@ -316,32 +279,29 @@ Dynamic Rmpi example:
workerpi()
mpi.quit()
+```
+
+The above example is the dynamic MPI example for calculating the number π. Both master and slave processes carry out the calculation. Note the mpi.spawn.Rslaves(), mpi.bcast.Robj2slave()** and the mpi.bcast.cmd()** function calls.
-The above example is the dynamic MPI example for calculating the number
-Ď€. Both master and slave processes carry out the calculation. Note the
-mpi.spawn.Rslaves(), mpi.bcast.Robj2slave()** and the
-mpi.bcast.cmd()** function calls.
Execute the example as:
+```bash
$ mpirun -np 1 R --slave --no-save --no-restore -f pi3Rslaves.R
+```
+
+Note that this method uses MPI_Comm_spawn (Dynamic process feature of MPI-2) to start the slave processes - the master process needs to be launched with MPI. In general, Dynamic processes are not well supported among MPI implementations, some issues might arise. Also, environment variables are not propagated to spawned processes, so they will not see paths from modules.
-Note that this method uses MPI_Comm_spawn (Dynamic process feature of
-MPI-2) to start the slave processes - the master process needs to be
-launched with MPI. In general, Dynamic processes are not well supported
-among MPI implementations, some issues might arise. Also, environment
-variables are not propagated to spawned processes, so they will not see
-paths from modules.
### mpi.apply Rmpi
mpi.apply is a specific way of executing Dynamic Rmpi programs.
-mpi.apply() family of functions provide MPI parallelized, drop in
-replacement for the serial apply() family of functions.
+mpi.apply() family of functions provide MPI parallelized, drop in replacement for the serial apply() family of functions.
Execution is identical to other dynamic Rmpi programs.
mpi.apply Rmpi example:
+```cpp
#integrand function
f <- function(i,h) {
x <- h*(i-0.5)
@@ -387,34 +347,27 @@ mpi.apply Rmpi example:
}
mpi.quit()
+```
-The above is the mpi.apply MPI example for calculating the number π.
-Only the slave processes carry out the calculation. Note the
-mpi.parSapply(), ** function call. The package
-parallel
-[example](r.html#package-parallel)[above](r.html#package-parallel){.anchor
-may be trivially adapted (for much better performance) to this structure
-using the mclapply() in place of mpi.parSapply().
+The above is the mpi.apply MPI example for calculating the number π. Only the slave processes carry out the calculation. Note the **mpi.parSapply(), ** function call. The package parallel [example](r.md#package-parallel)[above](r.md#package-parallel) may be trivially adapted (for much better performance) to this structure using the mclapply() in place of mpi.parSapply().
Execute the example as:
+```bash
$ mpirun -np 1 R --slave --no-save --no-restore -f pi3parSapply.R
+```
Combining parallel and Rmpi
---------------------------
-
Currently, the two packages can not be combined for hybrid calculations.
Parallel execution
------------------
+The R parallel jobs are executed via the PBS queue system exactly as any other parallel jobs. User must create an appropriate jobscript and submit via the **qsub**
-The R parallel jobs are executed via the PBS queue system exactly as any
-other parallel jobs. User must create an appropriate jobscript and
-submit via the **qsub**
-
-Example jobscript for [static Rmpi](r.html#static-rmpi)
-parallel R execution, running 1 process per core:
+Example jobscript for [static Rmpi](r.md#static-rmpi) parallel R execution, running 1 process per core:
+```bash
#!/bin/bash
#PBS -q qprod
#PBS -N Rjob
@@ -424,7 +377,7 @@ parallel R execution, running 1 process per core:
SCRDIR=/scratch/work/user/$USER/myjob
cd $SCRDIR || exit
- # copy input file to scratch
+ # copy input file to scratch
cp $PBS_O_WORKDIR/rscript.R .
# load R and openmpi module
@@ -439,22 +392,16 @@ parallel R execution, running 1 process per core:
#exit
exit
+```
-For more information about jobscripts and MPI execution refer to the
-[Job
-submission](../../resource-allocation-and-job-execution/job-submission-and-execution.html)
-and general [MPI](../mpi-1.html) sections.
+For more information about jobscripts and MPI execution refer to the [Job submission](../../resource-allocation-and-job-execution/job-submission-and-execution.md) and general [MPI](../mpi.md) sections.
Xeon Phi Offload
----------------
+By leveraging MKL, R can accelerate certain computations, most notably linear algebra operations on the Xeon Phi accelerator by using Automated Offload. To use MKL Automated Offload, you need to first set this environment variable before R execution:
-By leveraging MKL, R can accelerate certain computations, most notably
-linear algebra operations on the Xeon Phi accelerator by using Automated
-Offload. To use MKL Automated Offload, you need to first set this
-environment variable before R execution :
-
+```bash
$ export MKL_MIC_ENABLE=1
+```
-[Read more about automatic
-offload](../intel-xeon-phi.html)
-
+[Read more about automatic offload](../intel-xeon-phi.md)
\ No newline at end of file
diff --git a/docs.it4i/salomon/software/operating-system.md b/docs.it4i/salomon/software/operating-system.md
index 9cadb41ff..b692a3f73 100644
--- a/docs.it4i/salomon/software/operating-system.md
+++ b/docs.it4i/salomon/software/operating-system.md
@@ -1,13 +1,8 @@
-Operating System
+Operating System
================
-The operating system, deployed on Salomon cluster
-
-
+**The operating system, deployed on Salomon cluster**
The operating system on Salomon is Linux - CentOS 6.6.
-The CentOS Linux distribution is a stable, predictable, manageable
-and reproducible platform derived from the sources of Red Hat Enterprise
-Linux (RHEL).
-
+The CentOS Linux distribution is a stable, predictable, manageable and reproducible platform derived from the sources of Red Hat Enterprise Linux (RHEL).
\ No newline at end of file
diff --git a/docs.it4i/salomon/storage/cesnet-data-storage.md b/docs.it4i/salomon/storage/cesnet-data-storage.md
index 22b880420..8185c1bdd 100644
--- a/docs.it4i/salomon/storage/cesnet-data-storage.md
+++ b/docs.it4i/salomon/storage/cesnet-data-storage.md
@@ -1,128 +1,99 @@
-CESNET Data Storage
+CESNET Data Storage
===================
-
-
Introduction
------------
+Do not use shared filesystems at IT4Innovations as a backup for large amount of data or long-term archiving purposes.
-Do not use shared filesystems at IT4Innovations as a backup for large
-amount of data or long-term archiving purposes.
-
-The IT4Innovations does not provide storage capacity for data archiving.
-Academic staff and students of research institutions in the Czech
-Republic can use [CESNET Storage
-service](https://du.cesnet.cz/).
+>The IT4Innovations does not provide storage capacity for data archiving. Academic staff and students of research institutions in the Czech Republic can use [CESNET Storage service](https://du.cesnet.cz/).
-The CESNET Storage service can be used for research purposes, mainly by
-academic staff and students of research institutions in the Czech
-Republic.
+The CESNET Storage service can be used for research purposes, mainly by academic staff and students of research institutions in the Czech Republic.
-User of data storage CESNET (DU) association can become organizations or
-an individual person who is either in the current employment
-relationship (employees) or the current study relationship (students) to
-a legal entity (organization) that meets the “Principles for access to
-CESNET Large infrastructure (Access Policy)”.
+User of data storage CESNET (DU) association can become organizations or an individual person who is either in the current employment relationship (employees) or the current study relationship (students) to a legal entity (organization) that meets the “Principles for access to CESNET Large infrastructure (Access Policy)”.
-User may only use data storage CESNET for data transfer and storage
-which are associated with activities in science, research, development,
-the spread of education, culture and prosperity. In detail see
-“Acceptable Use Policy CESNET Large Infrastructure (Acceptable Use
-Policy, AUP)”.
+User may only use data storage CESNET for data transfer and storage which are associated with activities in science, research, development, the spread of education, culture and prosperity. In detail see “Acceptable Use Policy CESNET Large Infrastructure (Acceptable Use Policy, AUP)”.
-The service is documented at
-<https://du.cesnet.cz/wiki/doku.php/en/start>. For special requirements
-please contact directly CESNET Storage Department via e-mail
-[du-support(at)cesnet.cz](mailto:du-support@cesnet.cz).
+The service is documented at <https://du.cesnet.cz/wiki/doku.php/en/start>. For special requirements please contact directly CESNET Storage Department via e-mail [du-support(at)cesnet.cz](mailto:du-support@cesnet.cz).
The procedure to obtain the CESNET access is quick and trouble-free.
-(source
-[https://du.cesnet.cz/](https://du.cesnet.cz/wiki/doku.php/en/start "CESNET Data Storage"))
+(source [https://du.cesnet.cz/](https://du.cesnet.cz/wiki/doku.php/en/start "CESNET Data Storage"))
CESNET storage access
---------------------
### Understanding Cesnet storage
-It is very important to understand the Cesnet storage before uploading
-data. Please read
-<https://du.cesnet.cz/en/navody/home-migrace-plzen/start> first.
+>It is very important to understand the Cesnet storage before uploading data. Please read <https://du.cesnet.cz/en/navody/home-migrace-plzen/start> first.
-Once registered for CESNET Storage, you may [access the
-storage](https://du.cesnet.cz/en/navody/faq/start) in
-number of ways. We recommend the SSHFS and RSYNC methods.
+Once registered for CESNET Storage, you may [access the storage](https://du.cesnet.cz/en/navody/faq/start) in number of ways. We recommend the SSHFS and RSYNC methods.
### SSHFS Access
-SSHFS: The storage will be mounted like a local hard drive
+>SSHFS: The storage will be mounted like a local hard drive
-The SSHFSÂ provides a very convenient way to access the CESNET Storage.
-The storage will be mounted onto a local directory, exposing the vast
-CESNET Storage as if it was a local removable harddrive. Files can be
-than copied in and out in a usual fashion.
+The SSHFSÂ provides a very convenient way to access the CESNET Storage. The storage will be mounted onto a local directory, exposing the vast CESNET Storage as if it was a local removable harddrive. Files can be than copied in and out in a usual fashion.
First, create the mountpoint
+```bash
$ mkdir cesnet
+```
-Mount the storage. Note that you can choose among the ssh.du1.cesnet.cz
-(Plzen), ssh.du2.cesnet.cz (Jihlava), ssh.du3.cesnet.cz (Brno)
-Mount tier1_home **(only 5120M !)**:
+Mount the storage. Note that you can choose among the ssh.du1.cesnet.cz (Plzen), ssh.du2.cesnet.cz (Jihlava), ssh.du3.cesnet.cz (Brno) Mount tier1_home **(only 5120M !)**:
+```bash
$ sshfs username@ssh.du1.cesnet.cz:. cesnet/
+```
For easy future access from Anselm, install your public key
+```bash
$ cp .ssh/id_rsa.pub cesnet/.ssh/authorized_keys
+```
Mount tier1_cache_tape for the Storage VO:
+```bash
$ sshfs username@ssh.du1.cesnet.cz:/cache_tape/VO_storage/home/username cesnet/
+```
View the archive, copy the files and directories in and out
- $ ls cesnet/
+```bash
+ $ ls cesnet/
$ cp -a mydir cesnet/.
$ cp cesnet/myfile .
+```
Once done, please remember to unmount the storage
+```bash
$ fusermount -u cesnet
+```
### Rsync access
-Rsync provides delta transfer for best performance, can resume
-interrupted transfers
+>Rsync provides delta transfer for best performance, can resume interrupted transfers
-Rsync is a fast and extraordinarily versatile file copying tool. It is
-famous for its delta-transfer algorithm, which reduces the amount of
-data sent over the network by sending only the differences between the
-source files and the existing files in the destination. Rsync is widely
-used for backups and mirroring and as an improved copy command for
-everyday use.
+Rsync is a fast and extraordinarily versatile file copying tool. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
-Rsync finds files that need to be transferred using a "quick check"
-algorithm (by default) that looks for files that have changed in size or
-in last-modified time. Any changes in the other preserved attributes
-(as requested by options) are made on the destination file directly when
-the quick check indicates that the file's data does not need to be
-updated.
+Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated.
-More about Rsync at
-<https://du.cesnet.cz/en/navody/rsync/start#pro_bezne_uzivatele>
+More about Rsync at <https://du.cesnet.cz/en/navody/rsync/start#pro_bezne_uzivatele>
-Transfer large files to/from Cesnet storage, assuming membership in the
-Storage VO
+Transfer large files to/from Cesnet storage, assuming membership in the Storage VO
+```bash
$ rsync --progress datafile username@ssh.du1.cesnet.cz:VO_storage-cache_tape/.
$ rsync --progress username@ssh.du1.cesnet.cz:VO_storage-cache_tape/datafile .
+```
-Transfer large directories to/from Cesnet storage, assuming membership
-in the Storage VO
+Transfer large directories to/from Cesnet storage, assuming membership in the Storage VO
+```bash
$ rsync --progress -av datafolder username@ssh.du1.cesnet.cz:VO_storage-cache_tape/.
$ rsync --progress -av username@ssh.du1.cesnet.cz:VO_storage-cache_tape/datafolder .
+```
-Transfer rates of about 28MB/s can be expected.
-
+Transfer rates of about 28MB/s can be expected.
\ No newline at end of file
diff --git a/docs.it4i/salomon/storage/storage.md b/docs.it4i/salomon/storage/storage.md
index b170dd2ff..3352868da 100644
--- a/docs.it4i/salomon/storage/storage.md
+++ b/docs.it4i/salomon/storage/storage.md
@@ -1,150 +1,76 @@
-Storage
+Storage
=======
-
-
Introduction
------------
+There are two main shared file systems on Salomon cluster, the [HOME](storage.md#home)and [SCRATCH](storage.md#shared-filesystems).
-There are two main shared file systems on Salomon cluster, the [HOME](storage.html#home)and [SCRATCH](storage.html#shared-filesystems).
-
-All login and compute nodes may access same data on shared filesystems.
-Compute nodes are also equipped with local (non-shared) scratch, ramdisk
-and tmp filesystems.
+All login and compute nodes may access same data on shared filesystems. Compute nodes are also equipped with local (non-shared) scratch, ramdisk and tmp filesystems.
Policy (in a nutshell)
----------------------
+>Use [ for your most valuable data and programs.
+Use [WORK](storage.md#work) for your large project files.
+Use [TEMP](storage.md#temp) for large scratch data.
-Use [ for your most valuable data
-and programs.
-Use [WORK](storage.html#work) for your large project
-files
-Use [TEMP](storage.html#temp) for large scratch data.
-
-Do not use for [archiving](storage.html#archiving)!
+>Do not use for [archiving](storage.md#archiving)!
Archiving
-------------
-
-Please don't use shared filesystems as a backup for large amount of data
-or long-term archiving mean. The academic staff and students of research
-institutions in the Czech Republic can use [CESNET storage
-service](../../anselm-cluster-documentation/storage-1/cesnet-data-storage.html),
-which is available via SSHFS.
+Please don't use shared filesystems as a backup for large amount of data or long-term archiving mean. The academic staff and students of research institutions in the Czech Republic can use [CESNET storage service](../../anselm-cluster-documentation/storage/cesnet-data-storage.md), which is available via SSHFS.
Shared Filesystems
----------------------
-
-Salomon computer provides two main shared filesystems, the [
-HOME
-filesystem](storage.html#home-filesystem) and the
-[SCRATCH filesystem](storage.html#scratch-filesystem). The
-SCRATCH filesystem is partitioned to [WORK and TEMP
-workspaces](storage.html#shared-workspaces). The HOME
-filesystem is realized as a tiered NFSÂ disk storage. The SCRATCH
-filesystem is realized as a parallel Lustre filesystem. Both shared file
-systems are accessible via the Infiniband network. Extended ACLs are
-provided on both HOME/SCRATCH filesystems for the purpose of sharing
-data with other users using fine-grained control.
+Salomon computer provides two main shared filesystems, the [ HOME filesystem](storage.md#home-filesystem) and the [SCRATCH filesystem](storage.md#scratch-filesystem). The SCRATCH filesystem is partitioned to [WORK and TEMP workspaces](storage.md#shared-workspaces). The HOME filesystem is realized as a tiered NFSÂ disk storage. The SCRATCH filesystem is realized as a parallel Lustre filesystem. Both shared file systems are accessible via the Infiniband network. Extended ACLs are provided on both HOME/SCRATCH filesystems for the purpose of sharing data with other users using fine-grained control.
###HOME filesystem
-The HOME filesystem is realized as a Tiered filesystem, exported via
-NFS. The first tier has capacity 100TB, second tier has capacity 400TB.
-The filesystem is available on all login and computational nodes. The
-Home filesystem hosts the [HOME
-workspace](storage.html#home).
+The HOME filesystem is realized as a Tiered filesystem, exported via NFS. The first tier has capacity 100TB, second tier has capacity 400TB. The filesystem is available on all login and computational nodes. The Home filesystem hosts the [HOME workspace](storage.md#home).
###SCRATCH filesystem
-The architecture of Lustre on Salomon is composed of two metadata
-servers (MDS) and six data/object storage servers (OSS). Accessible
-capacity is 1.69 PB, shared among all users. The SCRATCH filesystem
-hosts the [WORK and TEMP
-workspaces](storage.html#shared-workspaces).
+The architecture of Lustre on Salomon is composed of two metadata servers (MDS) and six data/object storage servers (OSS). Accessible capacity is 1.69 PB, shared among all users. The SCRATCH filesystem hosts the [WORK and TEMP workspaces](storage.md#shared-workspaces).
- Configuration of the SCRATCH Lustre storage
+Configuration of the SCRATCH Lustre storage
-
- SCRATCH Lustre object storage
-
-
- Disk array SFA12KX
- 540 4TB SAS 7.2krpm disks
- 54 OSTs of 10 disks in RAID6 (8+2)
- 15 hot-spare disks
- 4x 400GB SSD cache
-
-
-
- SCRATCH Lustre metadata storage
-
-
- Disk array EF3015
- 12 600GB SAS 15krpm disks
-
-
### Understanding the Lustre Filesystems
(source <http://www.nas.nasa.gov>)
-A user file on the Lustre filesystem can be divided into multiple chunks
-(stripes) and stored across a subset of the object storage targets
-(OSTs) (disks). The stripes are distributed among the OSTs in a
-round-robin fashion to ensure load balancing.
-
-When a client (a compute
-node from your job) needs to create
-or access a file, the client queries the metadata server (
-MDS) and the metadata target (
-MDT) for the layout and location of the
-[file's
-stripes](http://www.nas.nasa.gov/hecc/support/kb/Lustre_Basics_224.html#striping).
-Once the file is opened and the client obtains the striping information,
-the MDS is no longer involved in the
-file I/O process. The client interacts directly with the object storage
-servers (OSSes) and OSTs to perform I/O operations such as locking, disk
-allocation, storage, and retrieval.
-
-If multiple clients try to read and write the same part of a file at the
-same time, the Lustre distributed lock manager enforces coherency so
-that all clients see consistent results.
-
-There is default stripe configuration for Salomon Lustre filesystems.
-However, users can set the following stripe parameters for their own
-directories or files to get optimum I/O performance:
-
-1. stripe_size: the size of the chunk in bytes; specify with k, m, or
- g to use units of KB, MB, or GB, respectively; the size must be an
- even multiple of 65,536 bytes; default is 1MB for all Salomon Lustre
- filesystems
-2. stripe_count the number of OSTs to stripe across; default is 1 for
- Salomon Lustre filesystems one can specify -1 to use all OSTs in
- the filesystem.
-3. stripe_offset The index of the
- OST where the first stripe is to be
- placed; default is -1 which results in random selection; using a
- non-default value is NOT recommended.
-
-Â
-
-Setting stripe size and stripe count correctly for your needs may
-significantly impact the I/O performance you experience.
-
-Use the lfs getstripe for getting the stripe parameters. Use the lfs
-setstripe command for setting the stripe parameters to get optimal I/O
-performance The correct stripe setting depends on your needs and file
-access patterns.Â
-
-`
-$ lfs getstripe dir|filename
+A user file on the Lustre filesystem can be divided into multiple chunks (stripes) and stored across a subset of the object storage targets (OSTs) (disks). The stripes are distributed among the OSTs in a round-robin fashion to ensure load balancing.
+
+When a client (a compute node from your job) needs to create or access a file, the client queries the metadata server ( MDS) and the metadata target ( MDT) for the layout and location of the [file's stripes](http://www.nas.nasa.gov/hecc/support/kb/Lustre_Basics_224.html#striping). Once the file is opened and the client obtains the striping information, the MDS is no longer involved in the file I/O process. The client interacts directly with the object storage servers (OSSes) and OSTs to perform I/O operations such as locking, disk allocation, storage, and retrieval.
+
+If multiple clients try to read and write the same part of a file at the same time, the Lustre distributed lock manager enforces coherency so that all clients see consistent results.
+
+There is default stripe configuration for Salomon Lustre filesystems. However, users can set the following stripe parameters for their own directories or files to get optimum I/O performance:
+
+1. stripe_size: the size of the chunk in bytes; specify with k, m, or g to use units of KB, MB, or GB, respectively; the size must be an even multiple of 65,536 bytes; default is 1MB for all Salomon Lustre filesystems
+2. stripe_count the number of OSTs to stripe across; default is 1 for Salomon Lustre filesystems one can specify -1 to use all OSTs in the filesystem.
+3. stripe_offset The index of the OST where the first stripe is to be placed; default is -1 which results in random selection; using a non-default value is NOT recommended.
+
+>Setting stripe size and stripe count correctly for your needs may significantly impact the I/O performance you experience.
+
+Use the lfs getstripe for getting the stripe parameters. Use the lfs setstripe command for setting the stripe parameters to get optimal I/O performance The correct stripe setting depends on your needs and file access patterns.
+
+```bash
+$ lfs getstripe dir|filename
$ lfs setstripe -s stripe_size -c stripe_count -o stripe_offset dir|filename
-`
+```
Example:
-`
+```bash
$ lfs getstripe /scratch/work/user/username
/scratch/work/user/username
stripe_count: 1 stripe_size: 1048576 stripe_offset: -1
@@ -153,65 +79,42 @@ $ lfs setstripe -c -1 /scratch/work/user/username/
$ lfs getstripe /scratch/work/user/username/
/scratch/work/user/username/
stripe_count: -1 stripe_size: 1048576 stripe_offset: -1
-`
+```
-In this example, we view current stripe setting of the
-/scratch/username/ directory. The stripe count is changed to all OSTs,
-and verified. All files written to this directory will be striped over
-all (54) OSTs
+In this example, we view current stripe setting of the /scratch/username/ directory. The stripe count is changed to all OSTs, and verified. All files written to this directory will be striped over all (54) OSTs
-Use lfs check OSTs to see the number and status of active OSTs for each
-filesystem on Salomon. Learn more by reading the man page
+Use lfs check OSTs to see the number and status of active OSTs for each filesystem on Salomon. Learn more by reading the man page
-`
+```bash
$ lfs check osts
$ man lfs
-`
+```
### Hints on Lustre Stripping
-Increase the stripe_count for parallel I/O to the same file.
+>Increase the stripe_count for parallel I/O to the same file.
-When multiple processes are writing blocks of data to the same file in
-parallel, the I/O performance for large files will improve when the
-stripe_count is set to a larger value. The stripe count sets the number
-of OSTs the file will be written to. By default, the stripe count is set
-to 1. While this default setting provides for efficient access of
-metadata (for example to support the ls -l command), large files should
-use stripe counts of greater than 1. This will increase the aggregate
-I/O bandwidth by using multiple OSTs in parallel instead of just one. A
-rule of thumb is to use a stripe count approximately equal to the number
-of gigabytes in the file.
+When multiple processes are writing blocks of data to the same file in parallel, the I/O performance for large files will improve when the stripe_count is set to a larger value. The stripe count sets the number of OSTs the file will be written to. By default, the stripe count is set to 1. While this default setting provides for efficient access of metadata (for example to support the ls -l command), large files should use stripe counts of greater than 1. This will increase the aggregate I/O bandwidth by using multiple OSTs in parallel instead of just one. A rule of thumb is to use a stripe count approximately equal to the number of gigabytes in the file.
-Another good practice is to make the stripe count be an integral factor
-of the number of processes performing the write in parallel, so that you
-achieve load balance among the OSTs. For example, set the stripe count
-to 16 instead of 15 when you have 64 processes performing the writes.
+Another good practice is to make the stripe count be an integral factor of the number of processes performing the write in parallel, so that you achieve load balance among the OSTs. For example, set the stripe count to 16 instead of 15 when you have 64 processes performing the writes.
-Using a large stripe size can improve performance when accessing very
-large files
+>Using a large stripe size can improve performance when accessing very large files
-Large stripe size allows each client to have exclusive access to its own
-part of a file. However, it can be counterproductive in some cases if it
-does not match your I/O pattern. The choice of stripe size has no effect
-on a single-stripe file.
+Large stripe size allows each client to have exclusive access to its own part of a file. However, it can be counterproductive in some cases if it does not match your I/O pattern. The choice of stripe size has no effect on a single-stripe file.
-Read more on
-<http://wiki.lustre.org/manual/LustreManual20_HTML/ManagingStripingFreeSpace.html>
+Read more on <http://wiki.lustre.org/manual/LustreManual20_HTML/ManagingStripingFreeSpace.html>
Disk usage and quota commands
------------------------------------------
+User quotas on the Lustre file systems (SCRATCH) can be checked and reviewed using following command:
-User quotas on the Lustre file systems (SCRATCH) can be checked
-and reviewed using following command:
-
-`
+```bash
$ lfs quota dir
-`
+```
Example for Lustre SCRATCH directory:
-`
+```bash
$ lfs quota /scratch
Disk quotas for user user001 (uid 1234):
Filesystem kbytes quota limit grace files quota limit grace
@@ -219,34 +122,35 @@ Disk quotas for user user001 (uid 1234):
Disk quotas for group user001 (gid 1234):
Filesystem kbytes quota limit grace files quota limit grace
/scratch    8    0    0    -    3    0    0    -
-`
+```
-In this example, we view current quota size limit of 100TB and 8KB
-currently used by user001.
+In this example, we view current quota size limit of 100TB and 8KB currently used by user001.
-HOME directory is mounted via NFS, so a different command must be used
-to obtain quota information:
+HOME directory is mounted via NFS, so a different command must be used to obtain quota information:
+```bash
 $ quota
+```
Example output:
+```bash
$ quota
Disk quotas for user vop999 (uid 1025):
Filesystem blocks quota limit grace files quota limit grace
home-nfs-ib.salomon.it4i.cz:/home
28 0 250000000 10 0 500000
+```
-To have a better understanding of where the space is exactly used, you
-can use following command to find out.
+To have a better understanding of where the space is exactly used, you can use following command to find out.
-`
+```bash
$ du -hs dir
-`
+```
Example for your HOME directory:
-`
+```bash
$ cd /home
$ du -hs * .[a-zA-z0-9]* | grep -E "[0-9]*G|[0-9]*M" | sort -hr
258M cuda-samples
@@ -254,38 +158,27 @@ $ du -hs * .[a-zA-z0-9]* | grep -E "[0-9]*G|[0-9]*M" | sort -hr
13M .mozilla
5,5M .eclipse
2,7M .idb_13.0_linux_intel64_app
-`
+```
-This will list all directories which are having MegaBytes or GigaBytes
-of consumed space in your actual (in this example HOME) directory. List
-is sorted in descending order from largest to smallest
-files/directories.
+This will list all directories which are having MegaBytes or GigaBytes of consumed space in your actual (in this example HOME) directory. List is sorted in descending order from largest to smallest files/directories.
-To have a better understanding of previous commands, you can read
-manpages.
+To have a better understanding of previous commands, you can read manpages.
-`
+```bash
$ man lfs
-`
+```
-`
-$ man du
-`
+```bash
+$ man du
+```
Extended Access Control List (ACL)
----------------------------------
+Extended ACLs provide another security mechanism beside the standard POSIX ACLs which are defined by three entries (for owner/group/others). Extended ACLs have more than the three basic entries. In addition, they also contain a mask entry and may contain any number of named user and named group entries.
-Extended ACLs provide another security mechanism beside the standard
-POSIX ACLs which are defined by three entries (for
-owner/group/others). Extended ACLs have more than the three basic
-entries. In addition, they also contain a mask entry and may contain any
-number of named user and named group entries.
-
-ACLs on a Lustre file system work exactly like ACLs on any Linux file
-system. They are manipulated with the standard tools in the standard
-manner. Below, we create a directory and allow a specific user access.
+ACLs on a Lustre file system work exactly like ACLs on any Linux file system. They are manipulated with the standard tools in the standard manner. Below, we create a directory and allow a specific user access.
-`
+```bash
[vop999@login1.salomon ~]$ umask 027
[vop999@login1.salomon ~]$ mkdir test
[vop999@login1.salomon ~]$ ls -ld test
@@ -310,13 +203,9 @@ user:johnsm:rwx
group::r-x
mask::rwx
other::---
-`
+```
-Default ACL mechanism can be used to replace setuid/setgid permissions
-on directories. Setting a default ACL on a directory (-d flag to
-setfacl) will cause the ACL permissions to be inherited by any newly
-created file or subdirectory within the directory. Refer to this page
-for more information on Linux ACL:
+Default ACL mechanism can be used to replace setuid/setgid permissions on directories. Setting a default ACL on a directory (-d flag to setfacl) will cause the ACL permissions to be inherited by any newly created file or subdirectory within the directory. Refer to this page for more information on Linux ACL:
[http://www.vanemery.com/Linux/ACL/POSIX_ACL_on_Linux.html ](http://www.vanemery.com/Linux/ACL/POSIX_ACL_on_Linux.html)
@@ -325,189 +214,99 @@ Shared Workspaces
###HOME
-Users home directories /home/username reside on HOME filesystem.
-Accessible capacity is 0.5PB, shared among all users. Individual users
-are restricted by filesystem usage quotas, set to 250GB per user.
-If 250GB should prove as insufficient for particular user, please
-contact [support](https://support.it4i.cz/rt),
-the quota may be lifted upon request.
-
-The HOME filesystem is intended for preparation, evaluation, processing
-and storage of data generated by active Projects.
-
-The HOMEÂ should not be used to archive data of past Projects or other
-unrelated data.
-
-The files on HOME will not be deleted until end of the [users
-lifecycle](../../get-started-with-it4innovations/obtaining-login-credentials/obtaining-login-credentials.html).
-
-The workspace is backed up, such that it can be restored in case ofÂ
-catasthropic failure resulting in significant data loss. This backup
-however is not intended to restore old versions of user data or to
-restore (accidentaly) deleted files.
-
-HOME workspace
-Accesspoint
-/home/username
-Capacity
-0.5PB
-Throughput
-6GB/s
-User quota
-250GB
-Protocol
-NFS, 2-Tier
+Users home directories /home/username reside on HOME filesystem. Accessible capacity is 0.5PB, shared among all users. Individual users are restricted by filesystem usage quotas, set to 250GB per user. If 250GB should prove as insufficient for particular user, please contact [support](https://support.it4i.cz/rt), the quota may be lifted upon request.
+
+>The HOME filesystem is intended for preparation, evaluation, processing and storage of data generated by active Projects.
+
+The HOMEÂ should not be used to archive data of past Projects or other unrelated data.
+
+The files on HOME will not be deleted until end of the [users lifecycle](../../get-started-with-it4innovations/obtaining-login-credentials/obtaining-login-credentials.md).
+
+The workspace is backed up, such that it can be restored in case of catasthropic failure resulting in significant data loss. This backup however is not intended to restore old versions of user data or to restore (accidentaly) deleted files.
+
+|HOME workspace||
+|---|---|
+|Accesspoint|/home/username|
+|Capacity|0.5PB|
+|Throughput|6GB/s|
+|User quota|250GB|
+|Protocol|NFS, 2-Tier|
+
### WORK
-The WORK workspace resides on SCRATCH filesystem. Users may create
-subdirectories and files in directories **/scratch/work/user/username**
-and **/scratch/work/project/projectid. **The /scratch/work/user/username
-is private to user, much like the home directory. The
-/scratch/work/project/projectid is accessible to all users involved in
-project projectid. >
+The WORK workspace resides on SCRATCH filesystem. Users may create subdirectories and files in directories **/scratch/work/user/username** and **/scratch/work/project/projectid. **The /scratch/work/user/username is private to user, much like the home directory. The /scratch/work/project/projectid is accessible to all users involved in project projectid.
-The WORK workspace is intended to store users project data as well as
-for high performance access to input and output files. All project data
-should be removed once the project is finished. The data on the WORK
-workspace are not backed up.
+>The WORK workspace is intended to store users project data as well as for high performance access to input and output files. All project data should be removed once the project is finished. The data on the WORK workspace are not backed up.
-Files on the WORK filesystem are **persistent** (not automatically
-deleted) throughout duration of the project.
+>Files on the WORK filesystem are **persistent** (not automatically deleted) throughout duration of the project.
-The WORK workspace is hosted on SCRATCH filesystem. The SCRATCH is
-realized as Lustre parallel filesystem and is available from all login
-and computational nodes. Default stripe size is 1MB, stripe count is 1.
-There are 54 OSTs dedicated for the SCRATCH filesystem.
+The WORK workspace is hosted on SCRATCH filesystem. The SCRATCH is realized as Lustre parallel filesystem and is available from all login and computational nodes. Default stripe size is 1MB, stripe count is 1. There are 54 OSTs dedicated for the SCRATCH filesystem.
-Setting stripe size and stripe count correctly for your needs may
-significantly impact the I/O performance you experience.
+>Setting stripe size and stripe count correctly for your needs may significantly impact the I/O performance you experience.
+
+|WORK workspace||
+|---|---|
+|Accesspoints|/scratch/work/user/username, /scratch/work/user/projectid|
+|Capacity |1.6P|
+|Throughput|30GB/s|
+|User quota|100TB|
+|Default stripe size|1MB|
+|Default stripe count|1|
+|Number of OSTs|54|
+|Protocol|Lustre|
-WORK workspace
-Accesspoints
-/scratch/work/user/username
-/scratch/work/user/projectid
-Capacity
-1.6P
-Throughput
-30GB/s
-User quota
-100TB
-Default stripe size
-1MB
-Default stripe count
-1
-Number of OSTs
-54
-Protocol
-Lustre
### TEMP
-The TEMP workspace resides on SCRATCH filesystem. The TEMP workspace
-accesspoint is /scratch/temp. Users may freely create subdirectories
-and files on the workspace. Accessible capacity is 1.6P, shared among
-all users on TEMP and WORK. Individual users are restricted by
-filesystem usage quotas, set to 100TB per user. The purpose of this
-quota is to prevent runaway programs from filling the entire filesystem
-and deny service to other users. >If 100TB should prove as
-insufficient for particular user, please contact
-[support](https://support.it4i.cz/rt), the quota may be
-lifted upon request.Â
-
-The TEMP workspace is intended for temporary scratch data generated
-during the calculation as well as for high performance access to input
-and output files. All I/O intensive jobs must use the TEMP workspace as
-their working directory.
-
-Users are advised to save the necessary data from the TEMP workspace to
-HOME or WORK after the calculations and clean up the scratch files.
-
-Files on the TEMP filesystem that are **not accessed for more than 90
-days** will be automatically **deleted**.
-
-The TEMP workspace is hosted on SCRATCH filesystem. The SCRATCH is
-realized as Lustre parallel filesystem and is available from all login
-and computational nodes. Default stripe size is 1MB, stripe count is 1.
-There are 54 OSTs dedicated for the SCRATCH filesystem.
-
-Setting stripe size and stripe count correctly for your needs may
-significantly impact the I/O performance you experience.
-
-TEMP workspace
-Accesspoint
-/scratch/temp
-Capacity
-1.6P
-Throughput
-30GB/s
-User quota
-100TB
-Default stripe size
-1MB
-Default stripe count
-1
-Number of OSTs
-54
-Protocol
-Lustre
-Â
+The TEMP workspace resides on SCRATCH filesystem. The TEMP workspace accesspoint is /scratch/temp. Users may freely create subdirectories and files on the workspace. Accessible capacity is 1.6P, shared among all users on TEMP and WORK. Individual users are restricted by filesystem usage quotas, set to 100TB per user. The purpose of this quota is to prevent runaway programs from filling the entire filesystem and deny service to other users. >If 100TB should prove as insufficient for particular user, please contact [support](https://support.it4i.cz/rt), the quota may be lifted upon request.
-RAM disk
---------
+>The TEMP workspace is intended for temporary scratch data generated during the calculation as well as for high performance access to input and output files. All I/O intensive jobs must use the TEMP workspace as their working directory.
-Every computational node is equipped with filesystem realized in memory,
-so called RAM disk.
+>Users are advised to save the necessary data from the TEMP workspace to HOME or WORK after the calculations and clean up the scratch files.
-Use RAM disk in case you need really fast access to your data of limited
-size during your calculation.
-Be very careful, use of RAM disk filesystem is at the expense of
-operational memory.
+>Files on the TEMP filesystem that are **not accessed for more than 90 days** will be automatically **deleted**.
-The local RAM disk is mounted as /ramdisk and is accessible to user
-at /ramdisk/$PBS_JOBID directory.
+The TEMP workspace is hosted on SCRATCH filesystem. The SCRATCH is realized as Lustre parallel filesystem and is available from all login and computational nodes. Default stripe size is 1MB, stripe count is 1. There are 54 OSTs dedicated for the SCRATCH filesystem.
-The local RAM disk filesystem is intended for temporary scratch data
-generated during the calculation as well as for high performance access
-to input and output files. Size of RAM disk filesystem is limited. Be
-very careful, use of RAM disk filesystem is at the expense of
-operational memory. It is not recommended to allocate large amount of
-memory and use large amount of data in RAM disk filesystem at the same
-time.
+>Setting stripe size and stripe count correctly for your needs may significantly impact the I/O performance you experience.
-The local RAM disk directory /ramdisk/$PBS_JOBID will be deleted
-immediately after the calculation end. Users should take care to save
-the output data from within the jobscript.
+|TEMP workspace||
+|---|---|
+|Accesspoint|/scratch/temp|
+|Capacity|1.6P|
+|Throughput|30GB/s|
+|User quota|100TB|
+|Default stripe size|1MB|
+|Default stripe count|1|
+|Number of OSTs|54|
+|Protocol|Lustre|
RAM disk
-Mountpoint
- /ramdisk
-Accesspoint
- /ramdisk/$PBS_JOBID
-Capacity
-120 GB
-Throughput
-over 1.5 GB/s write, over 5 GB/s read, single thread
-over 10 GB/s write, over 50 GB/s read, 16 threads
-
-User quota
-none
-Â
-
-Summary
+--------
+Every computational node is equipped with filesystem realized in memory, so called RAM disk.
-----------
+>Use RAM disk in case you need really fast access to your data of limited size during your calculation. Be very careful, use of RAM disk filesystem is at the expense of operational memory.
- --------------------------------------------------
- |Mountpoint|Usage|Protocol|Net|Capacity|Throughput|Limitations|Access|
- ---------------------------------------- |---|---|---------------------- ------------- -------------- ------------ ------------- ---------- |**Version**|**Module**|------
- | /home|home directory|NFS, 2-Tier|0.5 PB|6 GB/s|Quota 250GB|Compute and login nodes|backed up|
+The local RAM disk is mounted as /ramdisk and is accessible to user at /ramdisk/$PBS_JOBID directory.
- |/scratch/work|large project files|Lustre|1.69 PB|30 GB/s|Quota|Compute and login nodes|none|
-
+The local RAM disk filesystem is intended for temporary scratch data generated during the calculation as well as for high performance access to input and output files. Size of RAM disk filesystem is limited. Be very careful, use of RAM disk filesystem is at the expense of operational memory. It is not recommended to allocate large amount of memory and use large amount of data in RAM disk filesystem at the same time.
- |/scratch/temp|job temporary data|Lustre|1.69 PB|30 GB/s|Quota 100TB|Compute and login nodes|files older 90 days removed|
+>The local RAM disk directory /ramdisk/$PBS_JOBID will be deleted immediately after the calculation end. Users should take care to save the output data from within the jobscript.
- |/ramdisk|job temporary data, node local|local|120GB|90 GB/s|none|Compute nodes|purged after job ends|
- --------------------------------------------------
+|RAM disk||
+|---|---|
+|Mountpoint| /ramdisk|
+|Accesspoint| /ramdisk/$PBS_JOBID|
+|Capacity|120 GB|
+|Throughput|over 1.5 GB/s write, over 5 GB/s read, single thread, over 10 GB/s write, over 50 GB/s read, 16 threads|
+|User quota|none|
-Â
+Summary
+-------
+
+|Mountpoint|Usage|Protocol|Net|Capacity|Throughput|Limitations|Access|
+|---|---|
+| /home|home directory|NFS, 2-Tier|0.5 PB|6 GB/s|Quota 250GB|Compute and login nodes|backed up|
+|/scratch/work|large project files|Lustre|1.69 PB|30 GB/s|Quota|Compute and login nodes|none|
+|/scratch/temp|job temporary data|Lustre|1.69 PB|30 GB/s|Quota 100TB|Compute and login nodes|files older 90 days removed|
+|/ramdisk|job temporary data, node local|local|120GB|90 GB/s|none|Compute nodes|purged after job ends|
\ No newline at end of file
--
GitLab