Skip to content
Snippets Groups Projects
Commit 2f2efc63 authored by David Hrbáč's avatar David Hrbáč
Browse files

Merge branch 'clean_unicode' into 'master'

Clean UNICODE hard space

See merge request !42
parents e61facfb 2d119ecf
No related branches found
Tags
5 merge requests!368Update prace.md to document the change from qprace to qprod as the default...,!367Update prace.md to document the change from qprace to qprod as the default...,!366Update prace.md to document the change from qprace to qprod as the default...,!323extended-acls-storage-section,!42Clean UNICODE hard space
Pipeline #
Showing
with 142 additions and 142 deletions
...@@ -3,7 +3,7 @@ Capacity computing ...@@ -3,7 +3,7 @@ Capacity computing
Introduction Introduction
------------ ------------
In many cases, it is useful to submit huge (>100+) number of computational jobs into the PBS queue system. Huge number of (small) jobs is one of the most effective ways to execute embarrassingly parallel calculations, achieving best runtime, throughput and computer utilization. In many cases, it is useful to submit huge (>100+) number of computational jobs into the PBS queue system. Huge number of (small) jobs is one of the most effective ways to execute embarrassingly parallel calculations, achieving best runtime, throughput and computer utilization.
However, executing huge number of jobs via the PBS queue may strain the system. This strain may result in slow response to commands, inefficient scheduling and overall degradation of performance and user experience, for all users. For this reason, the number of jobs is **limited to 100 per user, 1000 per job array** However, executing huge number of jobs via the PBS queue may strain the system. This strain may result in slow response to commands, inefficient scheduling and overall degradation of performance and user experience, for all users. For this reason, the number of jobs is **limited to 100 per user, 1000 per job array**
...@@ -12,7 +12,7 @@ However, executing huge number of jobs via the PBS queue may strain the system. ...@@ -12,7 +12,7 @@ However, executing huge number of jobs via the PBS queue may strain the system.
- Use [Job arrays](capacity-computing/#job-arrays) when running huge number of [multithread](capacity-computing/#shared-jobscript-on-one-node) (bound to one node only) or multinode (multithread across several nodes) jobs - Use [Job arrays](capacity-computing/#job-arrays) when running huge number of [multithread](capacity-computing/#shared-jobscript-on-one-node) (bound to one node only) or multinode (multithread across several nodes) jobs
- Use [GNU parallel](capacity-computing/#gnu-parallel) when running single core jobs - Use [GNU parallel](capacity-computing/#gnu-parallel) when running single core jobs
- Combine [GNU parallel with Job arrays](capacity-computing/#job-arrays-and-gnu-parallel) when running huge number of single core jobs - Combine [GNU parallel with Job arrays](capacity-computing/#job-arrays-and-gnu-parallel) when running huge number of single core jobs
Policy Policy
------ ------
...@@ -73,7 +73,7 @@ cp $PBS_O_WORKDIR/$TASK input ; cp $PBS_O_WORKDIR/myprog.x . ...@@ -73,7 +73,7 @@ cp $PBS_O_WORKDIR/$TASK input ; cp $PBS_O_WORKDIR/myprog.x .
cp output $PBS_O_WORKDIR/$TASK.out cp output $PBS_O_WORKDIR/$TASK.out
``` ```
In this example, the submit directory holds the 900 input files, executable myprog.x and the jobscript file. As input for each run, we take the filename of input file from created tasklist file. We copy the input file to local scratch /lscratch/$PBS_JOBID, execute the myprog.x and copy the output file back to >the submit directory, under the $TASK.out name. The myprog.x runs on one node only and must use threads to run in parallel. Be aware, that if the myprog.x **is not multithreaded**, then all the **jobs are run as single thread programs in sequential** manner. Due to allocation of the whole node, the accounted time is equal to the usage of whole node**, while using only 1/16 of the node! In this example, the submit directory holds the 900 input files, executable myprog.x and the jobscript file. As input for each run, we take the filename of input file from created tasklist file. We copy the input file to local scratch /lscratch/$PBS_JOBID, execute the myprog.x and copy the output file back to >the submit directory, under the $TASK.out name. The myprog.x runs on one node only and must use threads to run in parallel. Be aware, that if the myprog.x **is not multithreaded**, then all the **jobs are run as single thread programs in sequential** manner. Due to allocation of the whole node, the accounted time is equal to the usage of whole node**, while using only 1/16 of the node!
If huge number of parallel multicore (in means of multinode multithread, e. g. MPI enabled) jobs is needed to run, then a job array approach should also be used. The main difference compared to previous example using one node is that the local scratch should not be used (as it's not shared between nodes) and MPI or other technique for parallel multinode run has to be used properly. If huge number of parallel multicore (in means of multinode multithread, e. g. MPI enabled) jobs is needed to run, then a job array approach should also be used. The main difference compared to previous example using one node is that the local scratch should not be used (as it's not shared between nodes) and MPI or other technique for parallel multinode run has to be used properly.
...@@ -156,7 +156,7 @@ GNU parallel ...@@ -156,7 +156,7 @@ GNU parallel
!!! Note "Note" !!! Note "Note"
Use GNU parallel to run many single core tasks on one node. Use GNU parallel to run many single core tasks on one node.
GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. GNU parallel is most useful in running single core jobs via the queue system on Anselm. GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. GNU parallel is most useful in running single core jobs via the queue system on Anselm.
For more information and examples see the parallel man page: For more information and examples see the parallel man page:
...@@ -171,7 +171,7 @@ The GNU parallel shell executes multiple instances of the jobscript using all co ...@@ -171,7 +171,7 @@ The GNU parallel shell executes multiple instances of the jobscript using all co
Example: Example:
Assume we have 101 input files with name beginning with "file" (e. g. file001, ..., file101). Assume we would like to use each of these input files with program executable myprog.x, each as a separate single core job. We call these single core jobs tasks. Assume we have 101 input files with name beginning with "file" (e. g. file001, ..., file101). Assume we would like to use each of these input files with program executable myprog.x, each as a separate single core job. We call these single core jobs tasks.
First, we create a tasklist file, listing all tasks - all input files in our example: First, we create a tasklist file, listing all tasks - all input files in our example:
...@@ -201,13 +201,13 @@ TASK=$1 ...@@ -201,13 +201,13 @@ TASK=$1
cp $PBS_O_WORKDIR/$TASK input cp $PBS_O_WORKDIR/$TASK input
# execute the calculation # execute the calculation
cat input > output cat input > output
# copy output file to submit directory # copy output file to submit directory
cp output $PBS_O_WORKDIR/$TASK.out cp output $PBS_O_WORKDIR/$TASK.out
``` ```
In this example, tasks from tasklist are executed via the GNU parallel. The jobscript executes multiple instances of itself in parallel, on all cores of the node. Once an instace of jobscript is finished, new instance starts until all entries in tasklist are processed. Currently processed entry of the joblist may be retrieved via $1 variable. Variable $TASK expands to one of the input filenames from tasklist. We copy the input file to local scratch, execute the myprog.x and copy the output file back to the submit directory, under the $TASK.out name. In this example, tasks from tasklist are executed via the GNU parallel. The jobscript executes multiple instances of itself in parallel, on all cores of the node. Once an instace of jobscript is finished, new instance starts until all entries in tasklist are processed. Currently processed entry of the joblist may be retrieved via $1 variable. Variable $TASK expands to one of the input filenames from tasklist. We copy the input file to local scratch, execute the myprog.x and copy the output file back to the submit directory, under the $TASK.out name.
### Submit the job ### Submit the job
...@@ -218,7 +218,7 @@ $ qsub -N JOBNAME jobscript ...@@ -218,7 +218,7 @@ $ qsub -N JOBNAME jobscript
12345.dm2 12345.dm2
``` ```
In this example, we submit a job of 101 tasks. 16 input files will be processed in parallel. The 101 tasks on 16 cores are assumed to complete in less than 2 hours. In this example, we submit a job of 101 tasks. 16 input files will be processed in parallel. The 101 tasks on 16 cores are assumed to complete in less than 2 hours.
Please note the #PBS directives in the beginning of the jobscript file, dont' forget to set your valid PROJECT_ID and desired queue. Please note the #PBS directives in the beginning of the jobscript file, dont' forget to set your valid PROJECT_ID and desired queue.
...@@ -239,7 +239,7 @@ Combined approach, very similar to job arrays, can be taken. Job array is submit ...@@ -239,7 +239,7 @@ Combined approach, very similar to job arrays, can be taken. Job array is submit
Example: Example:
Assume we have 992 input files with name beginning with "file" (e. g. file001, ..., file992). Assume we would like to use each of these input files with program executable myprog.x, each as a separate single core job. We call these single core jobs tasks. Assume we have 992 input files with name beginning with "file" (e. g. file001, ..., file992). Assume we would like to use each of these input files with program executable myprog.x, each as a separate single core job. We call these single core jobs tasks.
First, we create a tasklist file, listing all tasks - all input files in our example: First, we create a tasklist file, listing all tasks - all input files in our example:
...@@ -283,14 +283,14 @@ cat input > output ...@@ -283,14 +283,14 @@ cat input > output
cp output $PBS_O_WORKDIR/$TASK.out cp output $PBS_O_WORKDIR/$TASK.out
``` ```
In this example, the jobscript executes in multiple instances in parallel, on all cores of a computing node. Variable $TASK expands to one of the input filenames from tasklist. We copy the input file to local scratch, execute the myprog.x and copy the output file back to the submit directory, under the $TASK.out name. The numtasks file controls how many tasks will be run per subjob. Once an task is finished, new task starts, until the number of tasks in numtasks file is reached. In this example, the jobscript executes in multiple instances in parallel, on all cores of a computing node. Variable $TASK expands to one of the input filenames from tasklist. We copy the input file to local scratch, execute the myprog.x and copy the output file back to the submit directory, under the $TASK.out name. The numtasks file controls how many tasks will be run per subjob. Once an task is finished, new task starts, until the number of tasks in numtasks file is reached.
!!! Note "Note" !!! Note "Note"
Select subjob walltime and number of tasks per subjob carefully Select subjob walltime and number of tasks per subjob carefully
When deciding this values, think about following guiding rules: When deciding this values, think about following guiding rules:
1. Let n=N/16. Inequality (n+1) * T < W should hold. The N is number of tasks per subjob, T is expected single task walltime and W is subjob walltime. Short subjob walltime improves scheduling and job throughput. 1. Let n=N/16. Inequality (n+1) * T < W should hold. The N is number of tasks per subjob, T is expected single task walltime and W is subjob walltime. Short subjob walltime improves scheduling and job throughput.
2. Number of tasks should be modulo 16. 2. Number of tasks should be modulo 16.
3. These rules are valid only when all tasks have similar task walltimes T. 3. These rules are valid only when all tasks have similar task walltimes T.
...@@ -303,14 +303,14 @@ $ qsub -N JOBNAME -J 1-992:32 jobscript ...@@ -303,14 +303,14 @@ $ qsub -N JOBNAME -J 1-992:32 jobscript
12345[].dm2 12345[].dm2
``` ```
In this example, we submit a job array of 31 subjobs. Note the -J 1-992:**32**, this must be the same as the number sent to numtasks file. Each subjob will run on full node and process 16 input files in parallel, 32 in total per subjob. Every subjob is assumed to complete in less than 2 hours. In this example, we submit a job array of 31 subjobs. Note the -J 1-992:**32**, this must be the same as the number sent to numtasks file. Each subjob will run on full node and process 16 input files in parallel, 32 in total per subjob. Every subjob is assumed to complete in less than 2 hours.
Please note the #PBS directives in the beginning of the jobscript file, dont' forget to set your valid PROJECT_ID and desired queue. Please note the #PBS directives in the beginning of the jobscript file, dont' forget to set your valid PROJECT_ID and desired queue.
Examples Examples
-------- --------
Download the examples in [capacity.zip](capacity.zip), illustrating the above listed ways to run huge number of jobs. We recommend to try out the examples, before using this for running production jobs. Download the examples in [capacity.zip](capacity.zip), illustrating the above listed ways to run huge number of jobs. We recommend to try out the examples, before using this for running production jobs.
Unzip the archive in an empty directory on Anselm and follow the instructions in the README file Unzip the archive in an empty directory on Anselm and follow the instructions in the README file
......
...@@ -95,7 +95,7 @@ $ qsub -A OPEN-0-0 -q qprod -l select=4:ncpus=16:cpu_freq=24 -I ...@@ -95,7 +95,7 @@ $ qsub -A OPEN-0-0 -q qprod -l select=4:ncpus=16:cpu_freq=24 -I
In this example, we allocate 4 nodes, 16 cores at 2.4GHhz per node. In this example, we allocate 4 nodes, 16 cores at 2.4GHhz per node.
Intel Turbo Boost Technology is used by default, you can disable it for all nodes of job by using resource attribute cpu_turbo_boost. Intel Turbo Boost Technology is used by default, you can disable it for all nodes of job by using resource attribute cpu_turbo_boost.
```bash ```bash
$ qsub -A OPEN-0-0 -q qprod -l select=4:ncpus=16 -l cpu_turbo_boost=0 -I $ qsub -A OPEN-0-0 -q qprod -l select=4:ncpus=16 -l cpu_turbo_boost=0 -I
......
...@@ -29,12 +29,12 @@ fi ...@@ -29,12 +29,12 @@ fi
### Application Modules ### Application Modules
In order to configure your shell for running particular application on Anselm we use Module package interface. In order to configure your shell for running particular application on Anselm we use Module package interface.
!!! Note "Note" !!! Note "Note"
The modules set up the application paths, library paths and environment variables for running particular application. The modules set up the application paths, library paths and environment variables for running particular application.
We have also second modules repository. This modules repository is created using tool called EasyBuild. On Salomon cluster, all modules will be build by this tool. If you want to use software from this modules repository, please follow instructions in section [Application Modules Path Expansion](environment-and-modules/#EasyBuild). We have also second modules repository. This modules repository is created using tool called EasyBuild. On Salomon cluster, all modules will be build by this tool. If you want to use software from this modules repository, please follow instructions in section [Application Modules Path Expansion](environment-and-modules/#EasyBuild).
The modules may be loaded, unloaded and switched, according to momentary needs. The modules may be loaded, unloaded and switched, according to momentary needs.
...@@ -44,7 +44,7 @@ To check available modules use ...@@ -44,7 +44,7 @@ To check available modules use
$ module avail $ module avail
``` ```
To load a module, for example the octave module use To load a module, for example the octave module use
```bash ```bash
$ module load octave $ module load octave
...@@ -58,7 +58,7 @@ To check loaded modules use ...@@ -58,7 +58,7 @@ To check loaded modules use
$ module list $ module list
``` ```
To unload a module, for example the octave module use To unload a module, for example the octave module use
```bash ```bash
$ module unload octave $ module unload octave
......
...@@ -15,17 +15,17 @@ There are four types of compute nodes: ...@@ -15,17 +15,17 @@ There are four types of compute nodes:
- 180 compute nodes without the accelerator - 180 compute nodes without the accelerator
- 23 compute nodes with GPU accelerator - equipped with NVIDIA Tesla Kepler K20 - 23 compute nodes with GPU accelerator - equipped with NVIDIA Tesla Kepler K20
- 4 compute nodes with MIC accelerator - equipped with Intel Xeon Phi 5110P - 4 compute nodes with MIC accelerator - equipped with Intel Xeon Phi 5110P
- 2 fat nodes - equipped with 512 GB RAM and two 100 GB SSD drives - 2 fat nodes - equipped with 512 GB RAM and two 100 GB SSD drives
[More about Compute nodes](compute-nodes/). [More about Compute nodes](compute-nodes/).
GPU and accelerated nodes are available upon request, see the [Resources Allocation Policy](resources-allocation-policy/). GPU and accelerated nodes are available upon request, see the [Resources Allocation Policy](resources-allocation-policy/).
All these nodes are interconnected by fast InfiniBand network and Ethernet network. [More about the Network](network/). All these nodes are interconnected by fast InfiniBand network and Ethernet network. [More about the Network](network/).
Every chassis provides InfiniBand switch, marked **isw**, connecting all nodes in the chassis, as well as connecting the chassis to the upper level switches. Every chassis provides InfiniBand switch, marked **isw**, connecting all nodes in the chassis, as well as connecting the chassis to the upper level switches.
All nodes share 360 TB /home disk storage to store user files. The 146 TB shared /scratch storage is available for the scratch data. These file systems are provided by Lustre parallel file system. There is also local disk storage available on all compute nodes in /lscratch. [More about Storage](storage/). All nodes share 360 TB /home disk storage to store user files. The 146 TB shared /scratch storage is available for the scratch data. These file systems are provided by Lustre parallel file system. There is also local disk storage available on all compute nodes in /lscratch. [More about Storage](storage/).
The user access to the Anselm cluster is provided by two login nodes login1, login2, and data mover node dm1. [More about accessing cluster.](shell-and-data-access/) The user access to the Anselm cluster is provided by two login nodes login1, login2, and data mover node dm1. [More about accessing cluster.](shell-and-data-access/)
...@@ -47,8 +47,8 @@ The parameters are summarized in the following tables: ...@@ -47,8 +47,8 @@ The parameters are summarized in the following tables:
|MIC accelerated|4, cn[204-207]| |MIC accelerated|4, cn[204-207]|
|Fat compute nodes|2, cn[208-209]| |Fat compute nodes|2, cn[208-209]|
|**In total**|| |**In total**||
|Total theoretical peak performance (Rpeak)|94 TFLOP/s| |Total theoretical peak performance (Rpeak)|94 TFLOP/s|
|Total max. LINPACK performance (Rmax)|73 TFLOP/s| |Total max. LINPACK performance (Rmax)|73 TFLOP/s|
|Total amount of RAM|15.136 TB| |Total amount of RAM|15.136 TB|
|Node|Processor|Memory|Accelerator| |Node|Processor|Memory|Accelerator|
......
Introduction Introduction
============ ============
Welcome to Anselm supercomputer cluster. The Anselm cluster consists of 209 compute nodes, totaling 3344 compute cores with 15 TB RAM and giving over 94 TFLOP/s theoretical peak performance. Each node is a powerful x86-64 computer, equipped with 16 cores, at least 64 GB RAM, and 500 GB hard disk drive. Nodes are interconnected by fully non-blocking fat-tree InfiniBand network and equipped with Intel Sandy Bridge processors. A few nodes are also equipped with NVIDIA Kepler GPU or Intel Xeon Phi MIC accelerators. Read more in [Hardware Overview](hardware-overview/). Welcome to Anselm supercomputer cluster. The Anselm cluster consists of 209 compute nodes, totaling 3344 compute cores with 15 TB RAM and giving over 94 TFLOP/s theoretical peak performance. Each node is a powerful x86-64 computer, equipped with 16 cores, at least 64 GB RAM, and 500 GB hard disk drive. Nodes are interconnected by fully non-blocking fat-tree InfiniBand network and equipped with Intel Sandy Bridge processors. A few nodes are also equipped with NVIDIA Kepler GPU or Intel Xeon Phi MIC accelerators. Read more in [Hardware Overview](hardware-overview/).
The cluster runs bullx Linux ([bull](http://www.bull.com/bullx-logiciels/systeme-exploitation.html)) [operating system](software/operating-system/), which is compatible with the RedHat [ Linux family.](http://upload.wikimedia.org/wikipedia/commons/1/1b/Linux_Distribution_Timeline.svg) We have installed a wide range of software packages targeted at different scientific domains. These packages are accessible via the [modules environment](environment-and-modules/). The cluster runs bullx Linux ([bull](http://www.bull.com/bullx-logiciels/systeme-exploitation.html)) [operating system](software/operating-system/), which is compatible with the RedHat [ Linux family.](http://upload.wikimedia.org/wikipedia/commons/1/1b/Linux_Distribution_Timeline.svg) We have installed a wide range of software packages targeted at different scientific domains. These packages are accessible via the [modules environment](environment-and-modules/).
......
Job scheduling Job scheduling
============== ==============
Job execution priority Job execution priority
---------------------- ----------------------
Scheduler gives each job an execution priority and then uses this job execution priority to select which job(s) to run. Scheduler gives each job an execution priority and then uses this job execution priority to select which job(s) to run.
Job execution priority on Anselm is determined by these job properties (in order of importance): Job execution priority on Anselm is determined by these job properties (in order of importance):
1. queue priority 1. queue priority
2. fair-share priority 2. fair-share priority
...@@ -16,7 +16,7 @@ Job execution priority on Anselm is determined by these job properties (in orde ...@@ -16,7 +16,7 @@ Job execution priority on Anselm is determined by these job properties (in orde
Queue priority is priority of queue where job is queued before execution. Queue priority is priority of queue where job is queued before execution.
Queue priority has the biggest impact on job execution priority. Execution priority of jobs in higher priority queues is always greater than execution priority of jobs in lower priority queues. Other properties of job used for determining job execution priority (fair-share priority, eligible time) cannot compete with queue priority. Queue priority has the biggest impact on job execution priority. Execution priority of jobs in higher priority queues is always greater than execution priority of jobs in lower priority queues. Other properties of job used for determining job execution priority (fair-share priority, eligible time) cannot compete with queue priority.
Queue priorities can be seen at <https://extranet.it4i.cz/anselm/queues> Queue priorities can be seen at <https://extranet.it4i.cz/anselm/queues>
...@@ -48,7 +48,7 @@ Eligible time is amount (in seconds) of eligible time job accrued while waiting ...@@ -48,7 +48,7 @@ Eligible time is amount (in seconds) of eligible time job accrued while waiting
Eligible time has the least impact on execution priority. Eligible time is used for sorting jobs with equal queue priority and fair-share priority. It is very, very difficult for eligible time to compete with fair-share priority. Eligible time has the least impact on execution priority. Eligible time is used for sorting jobs with equal queue priority and fair-share priority. It is very, very difficult for eligible time to compete with fair-share priority.
Eligible time can be seen as eligible_time attribute of job. Eligible time can be seen as eligible_time attribute of job.
### Formula ### Formula
......
...@@ -41,13 +41,13 @@ In this example, we allocate 4 nodes, 16 cores per node, for 1 hour. We allocate ...@@ -41,13 +41,13 @@ In this example, we allocate 4 nodes, 16 cores per node, for 1 hour. We allocate
$ qsub -A OPEN-0-0 -q qnvidia -l select=10:ncpus=16 ./myjob $ qsub -A OPEN-0-0 -q qnvidia -l select=10:ncpus=16 ./myjob
``` ```
In this example, we allocate 10 nvidia accelerated nodes, 16 cores per node, for 24 hours. We allocate these resources via the qnvidia queue. Jobscript myjob will be executed on the first node in the allocation. In this example, we allocate 10 nvidia accelerated nodes, 16 cores per node, for 24 hours. We allocate these resources via the qnvidia queue. Jobscript myjob will be executed on the first node in the allocation.
```bash ```bash
$ qsub -A OPEN-0-0 -q qfree -l select=10:ncpus=16 ./myjob $ qsub -A OPEN-0-0 -q qfree -l select=10:ncpus=16 ./myjob
``` ```
In this example, we allocate 10 nodes, 16 cores per node, for 12 hours. We allocate these resources via the qfree queue. It is not required that the project OPEN-0-0 has any available resources left. Consumed resources are still accounted for. Jobscript myjob will be executed on the first node in the allocation. In this example, we allocate 10 nodes, 16 cores per node, for 12 hours. We allocate these resources via the qfree queue. It is not required that the project OPEN-0-0 has any available resources left. Consumed resources are still accounted for. Jobscript myjob will be executed on the first node in the allocation.
All qsub options may be [saved directly into the jobscript](job-submission-and-execution/#PBSsaved). In such a case, no options to qsub are needed. All qsub options may be [saved directly into the jobscript](job-submission-and-execution/#PBSsaved). In such a case, no options to qsub are needed.
...@@ -72,11 +72,11 @@ Specific nodes may be allocated via the PBS ...@@ -72,11 +72,11 @@ Specific nodes may be allocated via the PBS
qsub -A OPEN-0-0 -q qprod -l select=1:ncpus=16:host=cn171+1:ncpus=16:host=cn172 -I qsub -A OPEN-0-0 -q qprod -l select=1:ncpus=16:host=cn171+1:ncpus=16:host=cn172 -I
``` ```
In this example, we allocate nodes cn171 and cn172, all 16 cores per node, for 24 hours. Consumed resources will be accounted to the Project identified by Project ID OPEN-0-0. The resources will be available interactively. In this example, we allocate nodes cn171 and cn172, all 16 cores per node, for 24 hours. Consumed resources will be accounted to the Project identified by Project ID OPEN-0-0. The resources will be available interactively.
### Placement by CPU type ### Placement by CPU type
Nodes equipped with Intel Xeon E5-2665 CPU have base clock frequency 2.4GHz, nodes equipped with Intel Xeon E5-2470 CPU have base frequency 2.3 GHz (see section Compute Nodes for details). Nodes may be selected via the PBS resource attribute cpu_freq . Nodes equipped with Intel Xeon E5-2665 CPU have base clock frequency 2.4GHz, nodes equipped with Intel Xeon E5-2470 CPU have base frequency 2.3 GHz (see section Compute Nodes for details). Nodes may be selected via the PBS resource attribute cpu_freq .
|CPU Type|base freq.|Nodes|cpu_freq attribute| |CPU Type|base freq.|Nodes|cpu_freq attribute|
|---|---|---|---| |---|---|---|---|
...@@ -91,7 +91,7 @@ In this example, we allocate 4 nodes, 16 cores, selecting only the nodes with In ...@@ -91,7 +91,7 @@ In this example, we allocate 4 nodes, 16 cores, selecting only the nodes with In
### Placement by IB switch ### Placement by IB switch
Groups of computational nodes are connected to chassis integrated Infiniband switches. These switches form the leaf switch layer of the [Infiniband network](../network/) fat tree topology. Nodes sharing the leaf switch can communicate most efficiently. Sharing the same switch prevents hops in the network and provides for unbiased, most efficient network communication. Groups of computational nodes are connected to chassis integrated Infiniband switches. These switches form the leaf switch layer of the [Infiniband network](../network/) fat tree topology. Nodes sharing the leaf switch can communicate most efficiently. Sharing the same switch prevents hops in the network and provides for unbiased, most efficient network communication.
Nodes sharing the same switch may be selected via the PBS resource attribute ibswitch. Values of this attribute are iswXX, where XX is the switch number. The node-switch mapping can be seen at [Hardware Overview](../hardware-overview/) section. Nodes sharing the same switch may be selected via the PBS resource attribute ibswitch. Values of this attribute are iswXX, where XX is the switch number. The node-switch mapping can be seen at [Hardware Overview](../hardware-overview/) section.
...@@ -129,7 +129,7 @@ In the following example, we select an allocation for benchmarking a very specia ...@@ -129,7 +129,7 @@ In the following example, we select an allocation for benchmarking a very specia
-N Benchmark ./mybenchmark -N Benchmark ./mybenchmark
``` ```
The MPI processes will be distributed differently on the nodes connected to the two switches. On the isw10 nodes, we will run 1 MPI process per node 16 threads per process, on isw20 nodes we will run 16 plain MPI processes. The MPI processes will be distributed differently on the nodes connected to the two switches. On the isw10 nodes, we will run 1 MPI process per node 16 threads per process, on isw20 nodes we will run 16 plain MPI processes.
Although this example is somewhat artificial, it demonstrates the flexibility of the qsub command options. Although this example is somewhat artificial, it demonstrates the flexibility of the qsub command options.
...@@ -207,8 +207,8 @@ $ check-pbs-jobs --jobid 35141.dm2 --print-job-out ...@@ -207,8 +207,8 @@ $ check-pbs-jobs --jobid 35141.dm2 --print-job-out
JOB 35141.dm2, session_id 71995, user user2, nodes cn164,cn165 JOB 35141.dm2, session_id 71995, user user2, nodes cn164,cn165
Print job standard output: Print job standard output:
======================== Job start ========================== ======================== Job start ==========================
Started at : Fri Aug 30 02:47:53 CEST 2013 Started at : Fri Aug 30 02:47:53 CEST 2013
Script name : script Script name : script
Run loop 1 Run loop 1
Run loop 2 Run loop 2
Run loop 3 Run loop 3
...@@ -278,7 +278,7 @@ $ pwd ...@@ -278,7 +278,7 @@ $ pwd
In this example, 4 nodes were allocated interactively for 1 hour via the qexp queue. The interactive shell is executed in the home directory. In this example, 4 nodes were allocated interactively for 1 hour via the qexp queue. The interactive shell is executed in the home directory.
!!! Note "Note" !!! Note "Note"
All nodes within the allocation may be accessed via ssh. Unallocated nodes are not accessible to user. All nodes within the allocation may be accessed via ssh. Unallocated nodes are not accessible to user.
The allocated nodes are accessible via ssh from login nodes. The nodes may access each other via ssh as well. The allocated nodes are accessible via ssh from login nodes. The nodes may access each other via ssh as well.
......
...@@ -243,10 +243,10 @@ Users who have undergone the full local registration procedure (including signin ...@@ -243,10 +243,10 @@ Users who have undergone the full local registration procedure (including signin
```bash ```bash
$ it4ifree $ it4ifree
Password: Password:
PID Total Used ...by me Free PID Total Used ...by me Free
-------- ------- ------ -------- ------- -------- ------- ------ -------- -------
OPEN-0-0 1500000 400644 225265 1099356 OPEN-0-0 1500000 400644 225265 1099356
DD-13-1 10000 2606 2606 7394 DD-13-1 10000 2606 2606 7394
``` ```
By default file system quota is applied. To check the current status of the quota use By default file system quota is applied. To check the current status of the quota use
......
...@@ -34,6 +34,6 @@ Capacity computing ...@@ -34,6 +34,6 @@ Capacity computing
Use GNU Parallel and/or Job arrays when running (many) single core jobs. Use GNU Parallel and/or Job arrays when running (many) single core jobs.
In many cases, it is useful to submit huge (100+) number of computational jobs into the PBS queue system. Huge number of (small) jobs is one of the most effective ways to execute embarrassingly parallel calculations, achieving best runtime, throughput and computer utilization. In this chapter, we discuss the the recommended way to run huge number of jobs, including **ways to run huge number of single core jobs**. In many cases, it is useful to submit huge (100+) number of computational jobs into the PBS queue system. Huge number of (small) jobs is one of the most effective ways to execute embarrassingly parallel calculations, achieving best runtime, throughput and computer utilization. In this chapter, we discuss the the recommended way to run huge number of jobs, including **ways to run huge number of single core jobs**.
Read more on [Capacity computing](capacity-computing/) page. Read more on [Capacity computing](capacity-computing/) page.
...@@ -29,7 +29,7 @@ The resources are allocated to the job in a fair-share fashion, subject to const ...@@ -29,7 +29,7 @@ The resources are allocated to the job in a fair-share fashion, subject to const
### Notes ### Notes
The job wall clock time defaults to **half the maximum time**, see table above. Longer wall time limits can be [set manually, see examples](job-submission-and-execution/). The job wall clock time defaults to **half the maximum time**, see table above. Longer wall time limits can be [set manually, see examples](job-submission-and-execution/).
Jobs that exceed the reserved wall clock time (Req'd Time) get killed automatically. Wall clock time limit can be changed for queuing jobs (state Q) using the qalter command, however can not be changed for a running job (state R). Jobs that exceed the reserved wall clock time (Req'd Time) get killed automatically. Wall clock time limit can be changed for queuing jobs (state Q) using the qalter command, however can not be changed for a running job (state R).
...@@ -54,55 +54,55 @@ $ rspbs ...@@ -54,55 +54,55 @@ $ rspbs
Usage: rspbs [options] Usage: rspbs [options]
Options: Options:
--version show program's version number and exit --version show program's version number and exit
-h, --help show this help message and exit -h, --help show this help message and exit
--get-node-ncpu-chart --get-node-ncpu-chart
Print chart of allocated ncpus per node Print chart of allocated ncpus per node
--summary Print summary --summary Print summary
--get-server-details Print server --get-server-details Print server
--get-queues Print queues --get-queues Print queues
--get-queues-details Print queues details --get-queues-details Print queues details
--get-reservations Print reservations --get-reservations Print reservations
--get-reservations-details --get-reservations-details
Print reservations details Print reservations details
--get-nodes Print nodes of PBS complex --get-nodes Print nodes of PBS complex
--get-nodeset Print nodeset of PBS complex --get-nodeset Print nodeset of PBS complex
--get-nodes-details Print nodes details --get-nodes-details Print nodes details
--get-jobs Print jobs --get-jobs Print jobs
--get-jobs-details Print jobs details --get-jobs-details Print jobs details
--get-jobs-check-params --get-jobs-check-params
Print jobid, job state, session_id, user, nodes Print jobid, job state, session_id, user, nodes
--get-users Print users of jobs --get-users Print users of jobs
--get-allocated-nodes --get-allocated-nodes
Print allocated nodes of jobs Print allocated nodes of jobs
--get-allocated-nodeset --get-allocated-nodeset
Print allocated nodeset of jobs Print allocated nodeset of jobs
--get-node-users Print node users --get-node-users Print node users
--get-node-jobs Print node jobs --get-node-jobs Print node jobs
--get-node-ncpus Print number of ncpus per node --get-node-ncpus Print number of ncpus per node
--get-node-allocated-ncpus --get-node-allocated-ncpus
Print number of allocated ncpus per node Print number of allocated ncpus per node
--get-node-qlist Print node qlist --get-node-qlist Print node qlist
--get-node-ibswitch Print node ibswitch --get-node-ibswitch Print node ibswitch
--get-user-nodes Print user nodes --get-user-nodes Print user nodes
--get-user-nodeset Print user nodeset --get-user-nodeset Print user nodeset
--get-user-jobs Print user jobs --get-user-jobs Print user jobs
--get-user-jobc Print number of jobs per user --get-user-jobc Print number of jobs per user
--get-user-nodec Print number of allocated nodes per user --get-user-nodec Print number of allocated nodes per user
--get-user-ncpus Print number of allocated ncpus per user --get-user-ncpus Print number of allocated ncpus per user
--get-qlist-nodes Print qlist nodes --get-qlist-nodes Print qlist nodes
--get-qlist-nodeset Print qlist nodeset --get-qlist-nodeset Print qlist nodeset
--get-ibswitch-nodes Print ibswitch nodes --get-ibswitch-nodes Print ibswitch nodes
--get-ibswitch-nodeset --get-ibswitch-nodeset
Print ibswitch nodeset Print ibswitch nodeset
--state=STATE Only for given job state --state=STATE Only for given job state
--jobid=JOBID Only for given job ID --jobid=JOBID Only for given job ID
--user=USER Only for given user --user=USER Only for given user
--node=NODE Only for given node --node=NODE Only for given node
--nodestate=NODESTATE --nodestate=NODESTATE
Only for given node state (affects only --get-node* Only for given node state (affects only --get-node*
--get-qlist-* --get-ibswitch-* actions) --get-qlist-* --get-ibswitch-* actions)
--incl-finished Include finished jobs --incl-finished Include finished jobs
``` ```
Resources Accounting Policy Resources Accounting Policy
...@@ -110,7 +110,7 @@ Resources Accounting Policy ...@@ -110,7 +110,7 @@ Resources Accounting Policy
### The Core-Hour ### The Core-Hour
The resources that are currently subject to accounting are the core-hours. The core-hours are accounted on the wall clock basis. The accounting runs whenever the computational cores are allocated or blocked via the PBS Pro workload manager (the qsub command), regardless of whether the cores are actually used for any calculation. 1 core-hour is defined as 1 processor core allocated for 1 hour of wall clock time. Allocating a full node (16 cores) for 1 hour accounts to 16 core-hours. See example in the [Job submission and execution](job-submission-and-execution/) section. The resources that are currently subject to accounting are the core-hours. The core-hours are accounted on the wall clock basis. The accounting runs whenever the computational cores are allocated or blocked via the PBS Pro workload manager (the qsub command), regardless of whether the cores are actually used for any calculation. 1 core-hour is defined as 1 processor core allocated for 1 hour of wall clock time. Allocating a full node (16 cores) for 1 hour accounts to 16 core-hours. See example in the [Job submission and execution](job-submission-and-execution/) section.
### Check consumed resources ### Check consumed resources
...@@ -122,8 +122,8 @@ User may check at any time, how many core-hours have been consumed by himself/he ...@@ -122,8 +122,8 @@ User may check at any time, how many core-hours have been consumed by himself/he
```bash ```bash
$ it4ifree $ it4ifree
Password: Password:
PID Total Used ...by me Free PID Total Used ...by me Free
-------- ------- ------ -------- ------- -------- ------- ------ -------- -------
OPEN-0-0 1500000 400644 225265 1099356 OPEN-0-0 1500000 400644 225265 1099356
DD-13-1 10000 2606 2606 7394 DD-13-1 10000 2606 2606 7394
``` ```
...@@ -16,8 +16,8 @@ The authentication is by the [private key](../get-started-with-it4innovations/ac ...@@ -16,8 +16,8 @@ The authentication is by the [private key](../get-started-with-it4innovations/ac
!!! Note "Note" !!! Note "Note"
Please verify SSH fingerprints during the first logon. They are identical on all login nodes: Please verify SSH fingerprints during the first logon. They are identical on all login nodes:
29:b3:f4:64:b0:73:f5:6f:a7:85:0f:e0:0d:be:76:bf (DSA) 29:b3:f4:64:b0:73:f5:6f:a7:85:0f:e0:0d:be:76:bf (DSA)
d4:6f:5c:18:f4:3f:70:ef:bc:fc:cc:2b:fd:13:36:b7 (RSA) d4:6f:5c:18:f4:3f:70:ef:bc:fc:cc:2b:fd:13:36:b7 (RSA)
Private key authentication: Private key authentication:
...@@ -59,7 +59,7 @@ Example to the cluster login: ...@@ -59,7 +59,7 @@ Example to the cluster login:
Data Transfer Data Transfer
------------- -------------
Data in and out of the system may be transferred by the [scp](http://en.wikipedia.org/wiki/Secure_copy) and sftp protocols. (Not available yet.) In case large volumes of data are transferred, use dedicated data mover node dm1.anselm.it4i.cz for increased performance. Data in and out of the system may be transferred by the [scp](http://en.wikipedia.org/wiki/Secure_copy) and sftp protocols. (Not available yet.) In case large volumes of data are transferred, use dedicated data mover node dm1.anselm.it4i.cz for increased performance.
|Address|Port|Protocol| |Address|Port|Protocol|
|---|---|---| |---|---|---|
...@@ -75,7 +75,7 @@ The authentication is by the [private key](../get-started-with-it4innovations/ac ...@@ -75,7 +75,7 @@ The authentication is by the [private key](../get-started-with-it4innovations/ac
1TB may be transferred in 1:50h. 1TB may be transferred in 1:50h.
To achieve 160MB/s transfer rates, the end user must be connected by 10G line all the way to IT4Innovations and use computer with fast processor for the transfer. Using Gigabit ethernet connection, up to 110MB/s may be expected. Fast cipher (aes128-ctr) should be used. To achieve 160MB/s transfer rates, the end user must be connected by 10G line all the way to IT4Innovations and use computer with fast processor for the transfer. Using Gigabit ethernet connection, up to 110MB/s may be expected. Fast cipher (aes128-ctr) should be used.
!!! Note "Note" !!! Note "Note"
If you experience degraded data transfer performance, consult your local network provider. If you experience degraded data transfer performance, consult your local network provider.
...@@ -143,13 +143,13 @@ Port forwarding ...@@ -143,13 +143,13 @@ Port forwarding
It works by tunneling the connection from Anselm back to users workstation and forwarding from the workstation to the remote host. It works by tunneling the connection from Anselm back to users workstation and forwarding from the workstation to the remote host.
Pick some unused port on Anselm login node (for example 6000) and establish the port forwarding: Pick some unused port on Anselm login node (for example 6000) and establish the port forwarding:
```bash ```bash
local $ ssh -R 6000:remote.host.com:1234 anselm.it4i.cz local $ ssh -R 6000:remote.host.com:1234 anselm.it4i.cz
``` ```
In this example, we establish port forwarding between port 6000 on Anselm and port 1234 on the remote.host.com. By accessing localhost:6000 on Anselm, an application will see response of remote.host.com:1234. The traffic will run via users local workstation. In this example, we establish port forwarding between port 6000 on Anselm and port 1234 on the remote.host.com. By accessing localhost:6000 on Anselm, an application will see response of remote.host.com:1234. The traffic will run via users local workstation.
Port forwarding may be done **using PuTTY** as well. On the PuTTY Configuration screen, load your Anselm configuration first. Then go to Connection-&gt;SSH-&gt;Tunnels to set up the port forwarding. Click Remote radio button. Insert 6000 to Source port textbox. Insert remote.host.com:1234. Click Add button, then Open. Port forwarding may be done **using PuTTY** as well. On the PuTTY Configuration screen, load your Anselm configuration first. Then go to Connection-&gt;SSH-&gt;Tunnels to set up the port forwarding. Click Remote radio button. Insert 6000 to Source port textbox. Insert remote.host.com:1234. Click Add button, then Open.
...@@ -170,7 +170,7 @@ First, establish the remote port forwarding form the login node, as [described a ...@@ -170,7 +170,7 @@ First, establish the remote port forwarding form the login node, as [described a
Second, invoke port forwarding from the compute node to the login node. Insert following line into your jobscript or interactive shell Second, invoke port forwarding from the compute node to the login node. Insert following line into your jobscript or interactive shell
```bash ```bash
$ ssh -TN -f -L 6000:localhost:6000 login1 $ ssh -TN -f -L 6000:localhost:6000 login1
``` ```
In this example, we assume that port forwarding from login1:6000 to remote.host.com:1234 has been established beforehand. By accessing localhost:6000, an application running on a compute node will see response of remote.host.com:1234 In this example, we assume that port forwarding from login1:6000 to remote.host.com:1234 has been established beforehand. By accessing localhost:6000, an application running on a compute node will see response of remote.host.com:1234
...@@ -196,7 +196,7 @@ Once the proxy server is running, establish ssh port forwarding from Anselm to t ...@@ -196,7 +196,7 @@ Once the proxy server is running, establish ssh port forwarding from Anselm to t
local $ ssh -R 6000:localhost:1080 anselm.it4i.cz local $ ssh -R 6000:localhost:1080 anselm.it4i.cz
``` ```
Now, configure the applications proxy settings to **localhost:6000**. Use port forwarding to access the [proxy server from compute nodes](#port-forwarding-from-compute-nodes) as well. Now, configure the applications proxy settings to **localhost:6000**. Use port forwarding to access the [proxy server from compute nodes](#port-forwarding-from-compute-nodes) as well.
Graphical User Interface Graphical User Interface
------------------------ ------------------------
......
...@@ -48,9 +48,9 @@ echo Machines: $hl ...@@ -48,9 +48,9 @@ echo Machines: $hl
/ansys_inc/v145/CFX/bin/cfx5solve -def input.def -size 4 -size-ni 4x -part-large -start-method "Platform MPI Distributed Parallel" -par-dist $hl -P aa_r /ansys_inc/v145/CFX/bin/cfx5solve -def input.def -size 4 -size-ni 4x -part-large -start-method "Platform MPI Distributed Parallel" -par-dist $hl -P aa_r
``` ```
Header of the PBS file (above) is common and description can be find on [this site](../../resource-allocation-and-job-execution/job-submission-and-execution/). SVS FEM recommends to utilize sources by keywords: nodes, ppn. These keywords allows to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources. Header of the PBS file (above) is common and description can be find on [this site](../../resource-allocation-and-job-execution/job-submission-and-execution/). SVS FEM recommends to utilize sources by keywords: nodes, ppn. These keywords allows to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources.
Working directory has to be created before sending PBS job into the queue. Input file should be in working directory or full path to input file has to be specified. >Input file has to be defined by common CFX def file which is attached to the cfx solver via parameter -def Working directory has to be created before sending PBS job into the queue. Input file should be in working directory or full path to input file has to be specified. >Input file has to be defined by common CFX def file which is attached to the cfx solver via parameter -def
**License** should be selected by parameter -P (Big letter **P**). Licensed products are the following: aa_r (ANSYS **Academic** Research), ane3fl (ANSYS Multiphysics)-**Commercial**. **License** should be selected by parameter -P (Big letter **P**). Licensed products are the following: aa_r (ANSYS **Academic** Research), ane3fl (ANSYS Multiphysics)-**Commercial**.
[More about licensing here](licensing/) [More about licensing here](licensing/)
...@@ -39,9 +39,9 @@ NCORES=`wc -l $PBS_NODEFILE |awk '{print $1}'` ...@@ -39,9 +39,9 @@ NCORES=`wc -l $PBS_NODEFILE |awk '{print $1}'`
/ansys_inc/v145/fluent/bin/fluent 3d -t$NCORES -cnf=$PBS_NODEFILE -g -i fluent.jou /ansys_inc/v145/fluent/bin/fluent 3d -t$NCORES -cnf=$PBS_NODEFILE -g -i fluent.jou
``` ```
Header of the pbs file (above) is common and description can be find on [this site](../../resources-allocation-policy/). [SVS FEM](http://www.svsfem.cz) recommends to utilize sources by keywords: nodes, ppn. These keywords allows to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources. Header of the pbs file (above) is common and description can be find on [this site](../../resources-allocation-policy/). [SVS FEM](http://www.svsfem.cz) recommends to utilize sources by keywords: nodes, ppn. These keywords allows to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources.
Working directory has to be created before sending pbs job into the queue. Input file should be in working directory or full path to input file has to be specified. Input file has to be defined by common Fluent journal file which is attached to the Fluent solver via parameter -i fluent.jou Working directory has to be created before sending pbs job into the queue. Input file should be in working directory or full path to input file has to be specified. Input file has to be defined by common Fluent journal file which is attached to the Fluent solver via parameter -i fluent.jou
Journal file with definition of the input geometry and boundary conditions and defined process of solution has e.g. the following structure: Journal file with definition of the input geometry and boundary conditions and defined process of solution has e.g. the following structure:
...@@ -64,11 +64,11 @@ The appropriate dimension of the problem has to be set by parameter (2d/3d). ...@@ -64,11 +64,11 @@ The appropriate dimension of the problem has to be set by parameter (2d/3d).
fluent solver_version [FLUENT_options] -i journal_file -pbs fluent solver_version [FLUENT_options] -i journal_file -pbs
``` ```
This syntax will start the ANSYS FLUENT job under PBS Professional using the qsub command in a batch manner. When resources are available, PBS Professional will start the job and return a job ID, usually in the form of *job_ID.hostname*. This job ID can then be used to query, control, or stop the job using standard PBS Professional commands, such as qstat or qdel. The job will be run out of the current working directory, and all output will be written to the file fluent.o *job_ID*. This syntax will start the ANSYS FLUENT job under PBS Professional using the qsub command in a batch manner. When resources are available, PBS Professional will start the job and return a job ID, usually in the form of *job_ID.hostname*. This job ID can then be used to query, control, or stop the job using standard PBS Professional commands, such as qstat or qdel. The job will be run out of the current working directory, and all output will be written to the file fluent.o *job_ID*.
3. Running Fluent via user's config file 3. Running Fluent via user's config file
---------------------------------------- ----------------------------------------
The sample script uses a configuration file called pbs_fluent.conf if no command line arguments are present. This configuration file should be present in the directory from which the jobs are submitted (which is also the directory in which the jobs are executed). The following is an example of what the content of pbs_fluent.conf can be: The sample script uses a configuration file called pbs_fluent.conf if no command line arguments are present. This configuration file should be present in the directory from which the jobs are submitted (which is also the directory in which the jobs are executed). The following is an example of what the content of pbs_fluent.conf can be:
```bash ```bash
input="example_small.flin" input="example_small.flin"
...@@ -82,9 +82,9 @@ The following is an explanation of the parameters: ...@@ -82,9 +82,9 @@ The following is an explanation of the parameters:
input is the name of the input file. input is the name of the input file.
case is the name of the .cas file that the input file will utilize. case is the name of the .cas file that the input file will utilize.
fluent_args are extra ANSYS FLUENT arguments. As shown in the previous example, you can specify the interconnect by using the -p interconnect command. The available interconnects include ethernet (the default), myrinet, infiniband, vendor, altix, and crayx. The MPI is selected automatically, based on the specified interconnect. fluent_args are extra ANSYS FLUENT arguments. As shown in the previous example, you can specify the interconnect by using the -p interconnect command. The available interconnects include ethernet (the default), myrinet, infiniband, vendor, altix, and crayx. The MPI is selected automatically, based on the specified interconnect.
outfile is the name of the file to which the standard output will be sent. outfile is the name of the file to which the standard output will be sent.
......
ANSYS LS-DYNA ANSYS LS-DYNA
============= =============
**[ANSYSLS-DYNA](http://www.ansys.com/Products/Simulation+Technology/Structural+Mechanics/Explicit+Dynamics/ANSYS+LS-DYNA)** software provides convenient and easy-to-use access to the technology-rich, time-tested explicit solver without the need to contend with the complex input requirements of this sophisticated program. Introduced in 1996, ANSYS LS-DYNA capabilities have helped customers in numerous industries to resolve highly intricate design issues. ANSYS Mechanical users have been able take advantage of complex explicit solutions for a long time utilizing the traditional ANSYS Parametric Design Language (APDL) environment. These explicit capabilities are available to ANSYS Workbench users as well. The Workbench platform is a powerful, comprehensive, easy-to-use environment for engineering simulation. CAD import from all sources, geometry cleanup, automatic meshing, solution, parametric optimization, result visualization and comprehensive report generation are all available within a single fully interactive modern graphical user environment. **[ANSYSLS-DYNA](http://www.ansys.com/Products/Simulation+Technology/Structural+Mechanics/Explicit+Dynamics/ANSYS+LS-DYNA)** software provides convenient and easy-to-use access to the technology-rich, time-tested explicit solver without the need to contend with the complex input requirements of this sophisticated program. Introduced in 1996, ANSYS LS-DYNA capabilities have helped customers in numerous industries to resolve highly intricate design issues. ANSYS Mechanical users have been able take advantage of complex explicit solutions for a long time utilizing the traditional ANSYS Parametric Design Language (APDL) environment. These explicit capabilities are available to ANSYS Workbench users as well. The Workbench platform is a powerful, comprehensive, easy-to-use environment for engineering simulation. CAD import from all sources, geometry cleanup, automatic meshing, solution, parametric optimization, result visualization and comprehensive report generation are all available within a single fully interactive modern graphical user environment.
To run ANSYS LS-DYNA in batch mode you can utilize/modify the default ansysdyna.pbs script and execute it via the qsub command. To run ANSYS LS-DYNA in batch mode you can utilize/modify the default ansysdyna.pbs script and execute it via the qsub command.
...@@ -51,6 +51,6 @@ echo Machines: $hl ...@@ -51,6 +51,6 @@ echo Machines: $hl
/ansys_inc/v145/ansys/bin/ansys145 -dis -lsdynampp i=input.k -machines $hl /ansys_inc/v145/ansys/bin/ansys145 -dis -lsdynampp i=input.k -machines $hl
``` ```
Header of the PBS file (above) is common and description can be find on [this site](../../resource-allocation-and-job-execution/job-submission-and-execution/). [SVS FEM](http://www.svsfem.cz) recommends to utilize sources by keywords: nodes, ppn. These keywords allows to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources. Header of the PBS file (above) is common and description can be find on [this site](../../resource-allocation-and-job-execution/job-submission-and-execution/). [SVS FEM](http://www.svsfem.cz) recommends to utilize sources by keywords: nodes, ppn. These keywords allows to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources.
Working directory has to be created before sending PBS job into the queue. Input file should be in working directory or full path to input file has to be specified. Input file has to be defined by common LS-DYNA .**k** file which is attached to the ANSYS solver via parameter i= Working directory has to be created before sending PBS job into the queue. Input file should be in working directory or full path to input file has to be specified. Input file has to be defined by common LS-DYNA .**k** file which is attached to the ANSYS solver via parameter i=
...@@ -2,7 +2,7 @@ ANSYS MAPDL ...@@ -2,7 +2,7 @@ ANSYS MAPDL
=========== ===========
**[ANSYS Multiphysics](http://www.ansys.com/Products/Simulation+Technology/Structural+Mechanics/ANSYS+Multiphysics)** **[ANSYS Multiphysics](http://www.ansys.com/Products/Simulation+Technology/Structural+Mechanics/ANSYS+Multiphysics)**
software offers a comprehensive product solution for both multiphysics and single-physics analysis. The product includes structural, thermal, fluid and both high- and low-frequency electromagnetic analysis. The product also contains solutions for both direct and sequentially coupled physics problems including direct coupled-field elements and the ANSYS multi-field solver. software offers a comprehensive product solution for both multiphysics and single-physics analysis. The product includes structural, thermal, fluid and both high- and low-frequency electromagnetic analysis. The product also contains solutions for both direct and sequentially coupled physics problems including direct coupled-field elements and the ANSYS multi-field solver.
To run ANSYS MAPDL in batch mode you can utilize/modify the default mapdl.pbs script and execute it via the qsub command. To run ANSYS MAPDL in batch mode you can utilize/modify the default mapdl.pbs script and execute it via the qsub command.
...@@ -52,7 +52,7 @@ echo Machines: $hl ...@@ -52,7 +52,7 @@ echo Machines: $hl
Header of the PBS file (above) is common and description can be found on [this site](../../resource-allocation-policy/). [SVS FEM](http://www.svsfem.cz) recommends to utilize sources by keywords: nodes, ppn. These keywords allow to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources. Header of the PBS file (above) is common and description can be found on [this site](../../resource-allocation-policy/). [SVS FEM](http://www.svsfem.cz) recommends to utilize sources by keywords: nodes, ppn. These keywords allow to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources.
Working directory has to be created before sending PBS job into the queue. Input file should be in working directory or full path to input file has to be specified. Input file has to be defined by common APDL file which is attached to the ANSYS solver via parameter -i Working directory has to be created before sending PBS job into the queue. Input file should be in working directory or full path to input file has to be specified. Input file has to be defined by common APDL file which is attached to the ANSYS solver via parameter -i
**License** should be selected by parameter -p. Licensed products are the following: aa_r (ANSYS **Academic** Research), ane3fl (ANSYS Multiphysics)-**Commercial**, aa_r_dy (ANSYS **Academic** AUTODYN) **License** should be selected by parameter -p. Licensed products are the following: aa_r (ANSYS **Academic** Research), ane3fl (ANSYS Multiphysics)-**Commercial**, aa_r_dy (ANSYS **Academic** AUTODYN)
[More about licensing here](licensing/) [More about licensing here](licensing/)
Overview of ANSYS Products Overview of ANSYS Products
========================== ==========================
**[SVS FEM](http://www.svsfem.cz/)** as **[ANSYS Channel partner](http://www.ansys.com/)** for Czech Republic provided all ANSYS licenses for ANSELM cluster and supports of all ANSYS Products (Multiphysics, Mechanical, MAPDL, CFX, Fluent, Maxwell, LS-DYNA...) to IT staff and ANSYS users. If you are challenging to problem of ANSYS functionality contact please [hotline@svsfem.cz](mailto:hotline@svsfem.cz?subject=Ostrava%20-%20ANSELM) **[SVS FEM](http://www.svsfem.cz/)** as **[ANSYS Channel partner](http://www.ansys.com/)** for Czech Republic provided all ANSYS licenses for ANSELM cluster and supports of all ANSYS Products (Multiphysics, Mechanical, MAPDL, CFX, Fluent, Maxwell, LS-DYNA...) to IT staff and ANSYS users. If you are challenging to problem of ANSYS functionality contact please [hotline@svsfem.cz](mailto:hotline@svsfem.cz?subject=Ostrava%20-%20ANSELM)
Anselm provides commercial as well as academic variants. Academic variants are distinguished by "**Academic...**" word in the name of license or by two letter preposition "**aa_**" in the license feature name. Change of license is realized on command line respectively directly in user's PBS file (see individual products). [ More about licensing here](ansys/licensing/) Anselm provides commercial as well as academic variants. Academic variants are distinguished by "**Academic...**" word in the name of license or by two letter preposition "**aa_**" in the license feature name. Change of license is realized on command line respectively directly in user's PBS file (see individual products). [ More about licensing here](ansys/licensing/)
To load the latest version of any ANSYS product (Mechanical, Fluent, CFX, MAPDL,...) load the module: To load the latest version of any ANSYS product (Mechanical, Fluent, CFX, MAPDL,...) load the module:
...@@ -13,5 +13,5 @@ To load the latest version of any ANSYS product (Mechanical, Fluent, CFX, MAPDL, ...@@ -13,5 +13,5 @@ To load the latest version of any ANSYS product (Mechanical, Fluent, CFX, MAPDL,
ANSYS supports interactive regime, but due to assumed solution of extremely difficult tasks it is not recommended. ANSYS supports interactive regime, but due to assumed solution of extremely difficult tasks it is not recommended.
If user needs to work in interactive regime we recommend to configure the RSM service on the client machine which allows to forward the solution to the Anselm directly from the client's Workbench project (see ANSYS RSM service). If user needs to work in interactive regime we recommend to configure the RSM service on the client machine which allows to forward the solution to the Anselm directly from the client's Workbench project (see ANSYS RSM service).
LS-DYNA LS-DYNA
======= =======
[LS-DYNA](http://www.lstc.com/) is a multi-purpose, explicit and implicit finite element program used to analyze the nonlinear dynamic response of structures. Its fully automated contact analysis capability, a wide range of constitutive models to simulate a whole range of engineering materials (steels, composites, foams, concrete, etc.), error-checking features and the high scalability have enabled users worldwide to solve successfully many complex problems. Additionally LS-DYNA is extensively used to simulate impacts on structures from drop tests, underwater shock, explosions or high-velocity impacts. Explosive forming, process engineering, accident reconstruction, vehicle dynamics, thermal brake disc analysis or nuclear safety are further areas in the broad range of possible applications. In leading-edge research LS-DYNA is used to investigate the behavior of materials like composites, ceramics, concrete, or wood. Moreover, it is used in biomechanics, human modeling, molecular structures, casting, forging, or virtual testing. [LS-DYNA](http://www.lstc.com/) is a multi-purpose, explicit and implicit finite element program used to analyze the nonlinear dynamic response of structures. Its fully automated contact analysis capability, a wide range of constitutive models to simulate a whole range of engineering materials (steels, composites, foams, concrete, etc.), error-checking features and the high scalability have enabled users worldwide to solve successfully many complex problems. Additionally LS-DYNA is extensively used to simulate impacts on structures from drop tests, underwater shock, explosions or high-velocity impacts. Explosive forming, process engineering, accident reconstruction, vehicle dynamics, thermal brake disc analysis or nuclear safety are further areas in the broad range of possible applications. In leading-edge research LS-DYNA is used to investigate the behavior of materials like composites, ceramics, concrete, or wood. Moreover, it is used in biomechanics, human modeling, molecular structures, casting, forging, or virtual testing.
Anselm provides **1 commercial license of LS-DYNA without HPC** support now. Anselm provides **1 commercial license of LS-DYNA without HPC** support now.
...@@ -31,6 +31,6 @@ module load lsdyna ...@@ -31,6 +31,6 @@ module load lsdyna
/apps/engineering/lsdyna/lsdyna700s i=input.k /apps/engineering/lsdyna/lsdyna700s i=input.k
``` ```
Header of the PBS file (above) is common and description can be find on [this site](../../resource-allocation-and-job-execution/job-submission-and-execution.html). [SVS FEM](http://www.svsfem.cz) recommends to utilize sources by keywords: nodes, ppn. These keywords allows to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources. Header of the PBS file (above) is common and description can be find on [this site](../../resource-allocation-and-job-execution/job-submission-and-execution.html). [SVS FEM](http://www.svsfem.cz) recommends to utilize sources by keywords: nodes, ppn. These keywords allows to address directly the number of nodes (computers) and cores (ppn) which will be utilized in the job. Also the rest of code assumes such structure of allocated resources.
Working directory has to be created before sending PBS job into the queue. Input file should be in working directory or full path to input file has to be specified. Input file has to be defined by common LS-DYNA **.k** file which is attached to the LS-DYNA solver via parameter i= Working directory has to be created before sending PBS job into the queue. Input file should be in working directory or full path to input file has to be specified. Input file has to be defined by common LS-DYNA **.k** file which is attached to the LS-DYNA solver via parameter i=
...@@ -9,7 +9,7 @@ Molpro is a software package used for accurate ab-initio quantum chemistry calcu ...@@ -9,7 +9,7 @@ Molpro is a software package used for accurate ab-initio quantum chemistry calcu
License License
------- -------
Molpro software package is available only to users that have a valid license. Please contact support to enable access to Molpro if you have a valid license appropriate for running on our cluster (eg. academic research group licence, parallel execution). Molpro software package is available only to users that have a valid license. Please contact support to enable access to Molpro if you have a valid license appropriate for running on our cluster (eg. academic research group licence, parallel execution).
To run Molpro, you need to have a valid license token present in " $HOME/.molpro/token". You can download the token from [Molpro website](https://www.molpro.net/licensee/?portal=licensee). To run Molpro, you need to have a valid license token present in " $HOME/.molpro/token". You can download the token from [Molpro website](https://www.molpro.net/licensee/?portal=licensee).
......
...@@ -39,7 +39,7 @@ NWChem is compiled for parallel MPI execution. Normal procedure for MPI jobs app ...@@ -39,7 +39,7 @@ NWChem is compiled for parallel MPI execution. Normal procedure for MPI jobs app
Options Options
-------------------- --------------------
Please refer to [the documentation](http://www.nwchem-sw.org/index.php/Release62:Top-level) and in the input file set the following directives : Please refer to [the documentation](http://www.nwchem-sw.org/index.php/Release62:Top-level) and in the input file set the following directives :
- MEMORY : controls the amount of memory NWChem will use - MEMORY : controls the amount of memory NWChem will use
- SCRATCH_DIR : set this to a directory in [SCRATCH file system](../../storage/storage/#scratch) (or run the calculation completely in a scratch directory). For certain calculations, it might be advisable to reduce I/O by forcing "direct" mode, e.g.. "scf direct" - SCRATCH_DIR : set this to a directory in [SCRATCH file system](../../storage/storage/#scratch) (or run the calculation completely in a scratch directory). For certain calculations, it might be advisable to reduce I/O by forcing "direct" mode, e.g.. "scf direct"
...@@ -19,7 +19,7 @@ For information about the usage of Intel Compilers and other Intel products, ple ...@@ -19,7 +19,7 @@ For information about the usage of Intel Compilers and other Intel products, ple
GNU C/C++ and Fortran Compilers GNU C/C++ and Fortran Compilers
------------------------------- -------------------------------
For compatibility reasons there are still available the original (old 4.4.6-4) versions of GNU compilers as part of the OS. These are accessible in the search path by default. For compatibility reasons there are still available the original (old 4.4.6-4) versions of GNU compilers as part of the OS. These are accessible in the search path by default.
It is strongly recommended to use the up to date version (4.8.1) which comes with the module gcc: It is strongly recommended to use the up to date version (4.8.1) which comes with the module gcc:
...@@ -69,12 +69,12 @@ Simple program to test the compiler ...@@ -69,12 +69,12 @@ Simple program to test the compiler
#include <stdio.h> #include <stdio.h>
int main() { int main() {
if (MYTHREAD == 0) { if (MYTHREAD == 0) {
printf("Welcome to GNU UPC!!!n"); printf("Welcome to GNU UPC!!!n");
} }
upc_barrier; upc_barrier;
printf(" - Hello from thread %in", MYTHREAD); printf(" - Hello from thread %in", MYTHREAD);
return 0; return 0;
} }
``` ```
...@@ -115,12 +115,12 @@ Example UPC code: ...@@ -115,12 +115,12 @@ Example UPC code:
#include <stdio.h> #include <stdio.h>
int main() { int main() {
if (MYTHREAD == 0) { if (MYTHREAD == 0) {
printf("Welcome to Berkeley UPC!!!n"); printf("Welcome to Berkeley UPC!!!n");
} }
upc_barrier; upc_barrier;
printf(" - Hello from thread %in", MYTHREAD); printf(" - Hello from thread %in", MYTHREAD);
return 0; return 0;
} }
``` ```
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment