Skip to content
Snippets Groups Projects
Commit b2bcfbcb authored by Roman Sliva's avatar Roman Sliva Committed by Lukáš Krupčík
Browse files

Job features update

parent b3ccb874
No related branches found
No related tags found
4 merge requests!368Update prace.md to document the change from qprace to qprod as the default...,!367Update prace.md to document the change from qprace to qprod as the default...,!366Update prace.md to document the change from qprace to qprod as the default...,!323extended-acls-storage-section
......@@ -14,7 +14,7 @@ Load the VTune kernel modules.
$ qsub ... -l vtune=version_string
```
version_string is VTune version e.g. 2017_update2
version_string is VTune version e.g. 2019_update4
## Global RAM Disk
......@@ -35,8 +35,27 @@ Configure network for virtualization, create interconnect for fast communication
$ qsub ... -l virt_network=true
```
!!! Warning
Available on Anselm and Salomon nodes only.
[See Tap Interconnect][1]
## MSR-SAFE Support
Load a kernel module that allows saving/restoring values of MSR registers. Uses LLNL MSR-SAFE.
```console
$ qsub ... -l msr=version_string
```
version_string is MSR-SAFE version e.g. 1.4.0
!!! Danger
Hazardous, it causes CPU frequency disruption.
!!! Warning
Available on Salomon and Barbora nodes only.
## x86 Adapt Support
Load a kernel module that allows changing/toggling system parameters stored in MSR and PCI registers of x86 processors.
......@@ -61,6 +80,9 @@ To disable Intel Turbo Boost on CPU
$ qsub ... -l cpu_turbo_boost=false
```
!!! Warning
Available on Anselm and Salomon nodes only.
## Offlining CPU Cores
!!! Info
......@@ -83,4 +105,62 @@ where 'pattern' is a list of core's numbers to offline, separated by the charact
!!! Danger
Hazardous, it causes Lustre threads disruption.
## HDEEM Support
Load the HDEEM software stack. The High Definition Energy Efficiency Monitoring (HDEEM) library is a software interface used to measure power consumption of HPC clusters with bullx blades.
```console
$ qsub ... -l hdeem=version_string
```
version_string is HDEEM version e.g. 2.2.8-1
!!! Warning
Available on Barbora nodes only.
## NVMe Over Fabrics File System
Attach a volume from an NVMe storage and mount it as a file-system. File-system is mounted on /mnt/nvmeof (on the first node of the job).
Barbora cluster provides two NVMeoF storage nodes equipped with NVMe disks. Each storage node contains seven 1.6TB NVMe disks and provides net aggregated capacity of 10.18TiB. Storage space is provided using the NVMe over Fabrics protocol; RDMA network i.e. Infiniband is used for data transfers.
```console
$ qsub ... -l nvmeof=size
```
size is a size of requested volume, PBS size conventions are used, e.g. 10t
Create a shared file-system on the attached NVMe file-system and make it available on all nodes of the job. Append ':shared' to size specification, shared file-system is mounted on /mnt/nvmeof-shared.
```console
$ qsub ... -l nvmeof=size:shared
```
For example:
```console
$ qsub ... -l nvmeof=10t:shared
```
!!! Warning
Available on Barbora nodes only.
## Smart Burst Buffer
Accelerate SCRATCH storage using the Smart Burst Buffer (SBB) technology. A specific Burst Buffer process is launched and Burst Buffer resources (CPUs, memory, flash storage) are allocated on an SBB storage node for acceleration (I/O caching) of SCRATCH data operations. The SBB profile file /lscratch/$PBS_JOBID/sbb.sh is created on the first allocated node of job. For SCRATCH acceleration, the SBB profile file has to be sourced into the shell environment - provided environment variables have to be defined in the process environment.
Barbora cluster provides two SBB storage nodes equipped with NVMe disks. Each storage node contains ten 3.2TB NVMe disks and provides net aggregated capacity of 29.1TiB. Acceleration uses RDMA network i.e. Infiniband is used for data transfers.
```console
$ qsub ... -l sbb=spec
```
spec specifies amount of resources requested for Burst Buffer (CPUs, memory, flash storage), available values are small, medium, and large
Loading SBB profile:
```console
$ source /lscratch/$PBS_JOBID/sbb.sh
```
!!! Warning
Available on Barbora nodes only.
[1]: software/tools/virtualization.md#tap-interconnect
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment