diff --git a/README.md b/README.md
index d87d571fb21f9014412f86f4324a8f3729fbbb41..9bdd090550196c70b915bf1bd6af223a09aab9e3 100644
--- a/README.md
+++ b/README.md
@@ -1,6 +1,4 @@
-# User documentation
-
-test
+# User Documentation
This project contains IT4Innovations user documentation source.
@@ -15,7 +13,7 @@ $ source venv/bin/activate
$ pip install -r requirements.txt
```
-### Package upgrade with pip
+### Package Upgrade With pip
```console
$ pip list -o
@@ -43,7 +41,7 @@ Mellanox
## Mathematical Formulae
-### Formulas are made with:
+### Formulas Are Made With:
* [https://facelessuser.github.io/pymdown-extensions/extensions/arithmatex/](https://facelessuser.github.io/pymdown-extensions/extensions/arithmatex/)
* [https://www.mathjax.org/](https://www.mathjax.org/)
diff --git a/docs.it4i/barbora/compute-nodes.md b/docs.it4i/barbora/compute-nodes.md
index 9915e01c3166dbd9bf12279fe61a156f9e65c448..9e280f60a7581ba489f309deef0c3a37e1cbf687 100644
--- a/docs.it4i/barbora/compute-nodes.md
+++ b/docs.it4i/barbora/compute-nodes.md
@@ -1,18 +1,19 @@
# Compute Nodes
-Barbora is a cluster of x86-64 Intel-based nodes built with the BullSequana Computing technology. The cluster contains three types of compute nodes.
+Barbora is a cluster of x86-64 Intel-based nodes built with the BullSequana Computing technology.
+The cluster contains three types of compute nodes.
## Compute Nodes Without Accelerators
* 192 nodes
* 6912 cores in total
* 2x Intel Cascade Lake 6240, 18-core, 2.6 GHz processors per node
-* 192 GB DDR4 2933MT/s of physical memory per node (12x 16 GB)
+* 192 GB DDR4 2933 MT/s of physical memory per node (12x16 GB)
* BullSequana X1120 blade servers
-* 2995,2 GFLOP/s per compute node
+* 2995.2 GFLOP/s per compute node
* 1x 1 GB Ethernet
* 1x HDR100 IB port
-* 3 computes nodes per X1120 blade server
+* 3 compute nodes per X1120 blade server
* cn[1-192]
@@ -22,11 +23,11 @@ Barbora is a cluster of x86-64 Intel-based nodes built with the BullSequana Comp
* 8 nodes
* 192 cores in total
* two Intel Skylake Gold 6126, 12-core, 2.6 GHz processors per node
-* 192 GB DDR4 2933MT/s with ECC of physical memory per node (12x 16 GB)
+* 192 GB DDR4 2933MT/s with ECC of physical memory per node (12x16 GB)
* 4x GPU accelerator NVIDIA Tesla V100-SXM2 per node
* Bullsequana X410-E5 NVLink-V blade servers
-* 1996,8 GFLOP/s per compute nodes
-* GPU-tp-GPU All-to-All NVLINK 2.0, GPU-Direct
+* 1996.8 GFLOP/s per compute nodes
+* GPU-to-GPU All-to-All NVLINK 2.0, GPU-Direct
* 1 GB Ethernet
* 2x HDR100 IB ports
* cn[193-200]
@@ -37,8 +38,8 @@ Barbora is a cluster of x86-64 Intel-based nodes built with the BullSequana Comp
* 1x BullSequana X808 server
* 128 cores in total
-* 8 Intel Skylake 8153, 16-core, 2.0 GHz, 125W
-* 6144 GiB DDR4 2667MT/s of physical memory per node (92x 64 GB)
+* 8 Intel Skylake 8153, 16-core, 2.0 GHz, 125 W
+* 6144 GiB DDR4 2667 MT/s of physical memory per node (92x64 GB)
* 2x HDR100 IB port
* 8192 GFLOP/s
* cn[201]
@@ -47,19 +48,21 @@ Barbora is a cluster of x86-64 Intel-based nodes built with the BullSequana Comp
## Compute Node Summary
-| Node type | Count | Range | Memory | Cores | Queues |
-| ---------------------------- | ----- | ----------- | ------ | ----------- | -------------------------- |
-| Nodes without an accelerator | 192 | cn[1-192] | 192GB | 36 @ 2.6 GHz | qexp, qprod, qlong, qfree |
-| Nodes with a GPU accelerator | 8 | cn[193-200] | 192GB | 24 @ 2.6 GHz | qnvidia |
-| Fat compute nodes | 1 | cn[201] | 6144GiB | 128 @ 2.0 GHz | qfat |
+| Node type | Count | Range | Memory | Cores |
+| ---------------------------- | ----- | ----------- | -------- | ------------- |
+| Nodes without an accelerator | 192 | cn[1-192] | 192 GB | 36 @ 2.6 GHz |
+| Nodes with a GPU accelerator | 8 | cn[193-200] | 192 GB | 24 @ 2.6 GHz |
+| Fat compute nodes | 1 | cn[201] | 6144 GiB | 128 @ 2.0 GHz |
## Processor Architecture
-Barbora is equipped with Intel Cascade Lake processors Intel Xeon 6240 (nodes without accelerators), Intel Skylake Gold 6126 (nodes with accelerators) and Intel Skylake Platinum 8153.
+Barbora is equipped with Intel Cascade Lake processors Intel Xeon 6240 (nodes without accelerators),
+Intel Skylake Gold 6126 (nodes with accelerators) and Intel Skylake Platinum 8153.
### Intel [Cascade Lake 6240][d]
-Cascade Lake core is largely identical to that of [Skylake's][a]. For in-depth detail of the Skylake core/pipeline see [Skylake (client) § Pipeline][b].
+Cascade Lake core is largely identical to that of [Skylake's][a].
+For in-depth detail of the Skylake core/pipeline see [Skylake (client) § Pipeline][b].
Xeon Gold 6240 is a 64-bit 18-core x86 multi-socket high performance server microprocessor set to be introduced by Intel in late 2018. This chip supports up to 4-way multiprocessing. The Gold 6240, which is based on the Cascade Lake microarchitecture and is manufactured on a 14 nm process, sports 2 AVX-512 FMA units as well as three Ultra Path Interconnect links. This microprocessor, which operates at 2.6 GHz with a TDP of 150 W and a turbo boost frequency of up to 3.9 GHz, supports up 1 TB of hexa-channel DDR4-2933 ECC memory.
@@ -116,23 +119,23 @@ Barbora is equipped with an [NVIDIA Tesla V100-SXM2][g] accelerator.
-|NVIDIA Tesla V100-SXM2||
-| --- | --- |
-| GPU Architecture | NVIDIA Volta |
-| NVIDIA Tensor| Cores: 640 |
-| NVIDIA CUDA® Cores | 5 120 |
-| Double-Precision Performance | 7.8 TFLOP/s |
-| Single-Precision Performance | 15.7 TFLOP/s |
-| Tensor Performance | 125 TFLOP/s |
-| GPU Memory | 16 GB HBM2 |
-| Memory Bandwidth | 900 GB/sec |
-| ECC | Yes |
-| Interconnect Bandwidth | 300 GB/sec |
-| System Interface | NVIDIA NVLink |
-| Form Factor | SXM2 |
-| Max Power Consumption | 300 W |
-| Thermal Solution | Passive |
-| Compute APIs | CUDA, DirectCompute,OpenCLTM, OpenACC |
+| NVIDIA Tesla V100-SXM2 | |
+| ---------------------------- | -------------------------------------- |
+| GPU Architecture | NVIDIA Volta |
+| NVIDIA Tensor Cores | 640 |
+| NVIDIA CUDA® Cores | 5120 |
+| Double-Precision Performance | 7.8 TFLOP/s |
+| Single-Precision Performance | 15.7 TFLOP/s |
+| Tensor Performance | 125 TFLOP/s |
+| GPU Memory | 16 GB HBM2 |
+| Memory Bandwidth | 900 GB/sec |
+| ECC | Yes |
+| Interconnect Bandwidth | 300 GB/sec |
+| System Interface | NVIDIA NVLink |
+| Form Factor | SXM2 |
+| Max Power Consumption | 300 W |
+| Thermal Solution | Passive |
+| Compute APIs | CUDA, DirectCompute, OpenCLTM, OpenACC |
[a]: https://en.wikichip.org/wiki/intel/microarchitectures/skylake_(server)#Core
[b]: https://en.wikichip.org/wiki/intel/microarchitectures/skylake_(client)#Pipeline
diff --git a/docs.it4i/barbora/hardware-overview.md b/docs.it4i/barbora/hardware-overview.md
index cc3f350fa76989096226cfc88296917e06210d19..3d7a1c83914cbf8abc09f86e2cb921923c300cf0 100644
--- a/docs.it4i/barbora/hardware-overview.md
+++ b/docs.it4i/barbora/hardware-overview.md
@@ -1,8 +1,15 @@
# Hardware Overview
-The Barbora cluster consists of 201 computational nodes named **cn[1-201]** of which 192 are regular compute nodes, 8 are GPU Tesla V100 accelerated nodes and 1 is a fat node. Each node is a powerful x86-64 computer, equipped with 36/24/128 cores (18-core Intel Cascade Lake 6240 / 12-core Intel Skylake Gold 6126 / 16-core Intel Skylake 8153), at least 192 GB of RAM. User access to the Barbora cluster is provided by two login nodes **login[1,2]**. The nodes are interlinked through high speed InfiniBand and Ethernet networks.
+The Barbora cluster consists of 201 computational nodes named **cn[001-201]**
+of which 192 are regular compute nodes, 8 are GPU Tesla V100 accelerated nodes and 1 is a fat node.
+Each node is a powerful x86-64 computer, equipped with 36/24/128 cores
+(18-core Intel Cascade Lake 6240 / 12-core Intel Skylake Gold 6126 / 16-core Intel Skylake 8153), at least 192 GB of RAM.
+User access to the Barbora cluster is provided by two login nodes **login[1,2]**.
+The nodes are interlinked through high speed InfiniBand and Ethernet networks.
-The Fat node is equipped with a large amount (6144 GB) of memory. Virtualization infrastructure provides resources to run long-term servers and services in virtual mode. The Accelerated nodes, Fat node, and Virtualization infrastructure are available [upon request][a] from a PI.
+The fat node is equipped with 6144 GB of memory.
+Virtualization infrastructure provides resources for running long-term servers and services in virtual mode.
+The Accelerated nodes, fat node, and virtualization infrastructure are available [upon request][a] from a PI.
**There are three types of compute nodes:**
@@ -10,14 +17,17 @@ The Fat node is equipped with a large amount (6144 GB) of memory. Virtualization
* 8 compute nodes with a GPU accelerator - 4x NVIDIA Tesla V100-SXM2
* 1 fat node - equipped with 6144 GB of RAM
-[More about Compute nodes][1].
+[More about compute nodes][1].
GPU and accelerated nodes are available upon request, see the [Resources Allocation Policy][2].
-All of these nodes are interconnected through fast InfiniBand and Ethernet networks. [More about the Network][3].
-Every chassis provides an InfiniBand switch, marked **isw**, connecting all nodes in the chassis, as well as connecting the chassis to the upper level switches.
+All of these nodes are interconnected through fast InfiniBand and Ethernet networks.
+[More about the computing network][3].
+Every chassis provides an InfiniBand switch, marked **isw**, connecting all nodes in the chassis,
+as well as connecting the chassis to the upper level switches.
-User access to Barbora is provided by two login nodes: login1 and login2. [More about accessing the cluster][5].
+User access to Barbora is provided by two login nodes: login1 and login2.
+[More about accessing the cluster][5].
The parameters are summarized in the following tables:
@@ -32,20 +42,20 @@ The parameters are summarized in the following tables:
| RAM | min. 192 GB |
| Local disk drive | no |
| Compute network | InfiniBand HDR |
-| w/o accelerator | 192, cn[1-192] |
+| w/o accelerator | 192, cn[001-192] |
| GPU accelerated | 8, cn[193-200] |
| Fat compute nodes | 1, cn[201] |
-| **In total** | |
+| **In total** | |
| Total theoretical peak performance (Rpeak) | 848.8448 TFLOP/s |
| Total amount of RAM | 44.544 TB |
-| Node | Processor | Memory | Accelerator |
-| ---------------- | --------------------------------------- | ------ | ---------------------- |
-| w/o accelerator | 2 x Intel Cascade Lake 6240, 2.6 GHz | 192 GB | - |
-| GPU accelerated | 2 x Intel Skylake Gold 6126, 2.6 GHz | 192 GB | NVIDIA Tesla V100-SXM2 |
-| Fat compute node | 2 x Intel Skylake Platinum 8153, 2.0 GHz | 6144 GB | - |
+| Node | Processor | Memory | Accelerator |
+| ---------------- | --------------------------------------- | ------ | ---------------------- |
+| Regular node | 2x Intel Cascade Lake 6240, 2.6 GHz | 192GB | - |
+| GPU accelerated | 2x Intel Skylake Gold 6126, 2.6 GHz | 192GB | NVIDIA Tesla V100-SXM2 |
+| Fat compute node | 2x Intel Skylake Platinum 8153, 2.0 GHz | 6144GB | - |
-For more details refer to [Compute nodes][1], [Storage][4], [Visualization servers][6], and [Network][3].
+For more details refer to the sections [Compute Nodes][1], [Storage][4], [Visualization Servers][6], and [Network][3].
[1]: compute-nodes.md
[2]: ../general/resources-allocation-policy.md
diff --git a/docs.it4i/barbora/introduction.md b/docs.it4i/barbora/introduction.md
index fd300804fc3f256ada81fe415f4b3805dd884943..874ccb1f3e951b222a8c8d1231ece872f3855542 100644
--- a/docs.it4i/barbora/introduction.md
+++ b/docs.it4i/barbora/introduction.md
@@ -2,11 +2,11 @@
Welcome to Barbora supercomputer cluster. The Barbora cluster consists of 201 compute nodes, totaling 7232 compute cores with 44544 GB RAM, giving over 848 TFLOP/s theoretical peak performance.
-Nodes are interconnected through a fully non-blocking fat-tree InfiniBand network, and are equipped with Intel Cascade Lake processors. A few nodes are also equipped with NVIDIA Tesla V100-SXM2 Read more in [Hardware Overview][1].
+Nodes are interconnected through a fully non-blocking fat-tree InfiniBand network, and are equipped with Intel Cascade Lake processors. A few nodes are also equipped with NVIDIA Tesla V100-SXM2. Read more in [Hardware Overview][1].
The cluster runs with an operating system compatible with the Red Hat [Linux family][a]. We have installed a wide range of software packages targeted at different scientific domains. These packages are accessible via the [modules environment][2].
-The user data shared file-system and job data shared file-system are available to users.
+The user data shared file system and job data shared file system are available to users.
The [PBS Professional Open Source Project][b] workload manager provides [computing resources allocations and job execution][3].
diff --git a/docs.it4i/barbora/network.md b/docs.it4i/barbora/network.md
index 58b28ec3cb7fb3f79e1a081bc6eedf8d35619322..0f9ab3766767b32c1526f2b46f8ee8ffe6b5efd3 100644
--- a/docs.it4i/barbora/network.md
+++ b/docs.it4i/barbora/network.md
@@ -2,9 +2,11 @@
All of the compute and login nodes of Barbora are interconnected through a [InfiniBand][a] HDR 200 Gbps network and a Gigabit Ethernet network.
-Compute nodes and the service infrastructure is connected by the HDR100 technology that allows one 200Gbps HDR port (aggregation 4x 50Gbps) to be divided into two HDR100 ports with 100Gbps (2x 50Gbps) bandwidth.
+Compute nodes and the service infrastructure is connected by the HDR100 technology
+that allows one 200 Gbps HDR port (aggregation 4x 50 Gbps) to be divided into two HDR100 ports with 100 Gbps (2x 50 Gbps) bandwidth.
-The cabling between the L1 and L2 layer is realized by HDR cabling, connecting the end devices is realized by so called Y or splitter cable (1x HRD200 - 2x HDR100).
+The cabling between the L1 and L2 layer is realized by HDR cabling,
+connecting the end devices is realized by so called Y or splitter cable (1x HRD200 - 2x HDR100).
@@ -21,9 +23,9 @@ The cabling between the L1 and L2 layer is realized by HDR cabling, connecting t
**Performance**
-* 40x HDR 200Gb/s ports in a 1U switch
-* 80x HDR100 100Gb/s ports in a 1U switch
-* 16Tb/s aggregate switch throughput
+* 40x HDR 200 Gb/s ports in a 1U switch
+* 80x HDR100 100 Gb/s ports in a 1U switch
+* 16 Tb/s aggregate switch throughput
* Up to 15.8 billion messages-per-second
* 90ns switch latency
diff --git a/docs.it4i/barbora/visualization.md b/docs.it4i/barbora/visualization.md
index 172231d4827956f9ca835ca8bf82ee40fc380ac5..fe72dc8a3c42295ebc06a80600d88b612469bbe3 100644
--- a/docs.it4i/barbora/visualization.md
+++ b/docs.it4i/barbora/visualization.md
@@ -4,8 +4,8 @@ Remote visualization with [VirtualGL][3] is available on two nodes.
* 2 nodes
* 32 cores in total
-* 2x Intel Skylake Gold 6130 – 16core@2,1GHz processors per node
-* 192 GB DDR4 2667MT/s of physical memory per node (12x 16 GB)
+* 2x Intel Skylake Gold 6130 – 16-core@2,1 GHz processors per node
+* 192 GB DDR4 2667 MT/s of physical memory per node (12x 16 GB)
* BullSequana X450-E5 blade servers
* 2150.4 GFLOP/s per compute node
* 1x 1 GB Ethernet and 2x 10 GB Ethernet
diff --git a/docs.it4i/cs/specifications.md b/docs.it4i/cs/specifications.md
index cea360f479a9f24fa891b83a849a016b91356144..bc257f62ad000348116681db0f97a9639f91da51 100644
--- a/docs.it4i/cs/specifications.md
+++ b/docs.it4i/cs/specifications.md
@@ -27,11 +27,12 @@ consists of 8 compute nodes with the following per-node parameters:
- 1x Infiniband HDR100 interface
- connected via 16x PCI-e Gen3 slot to the CPU
-## Partition 02 - Intel (Ice Lake, NVDIMMs + Bitware FPGAs)
+## Partition 02 - Intel (Ice Lake, NVDIMMs) <!--- + Bitware FPGAs) -->
The partition is based on the Intel Ice Lake x86 architecture.
-The key technologies installed are Intel NVDIMM memories and Intel FPGA accelerators.
-The partition contains two servers each with two FPGA accelerators.
+It contains two servers with Intel NVDIMM memories.
+ <!--- The key technologies installed are Intel NVDIMM memories. and Intel FPGA accelerators.
+The partition contains two servers each with two FPGA accelerators. -->
Each server has the following parameters:
@@ -42,8 +43,11 @@ Each server has the following parameters:
- 1x Infiniband HDR100 interface
- connected to CPU 8x PCI-e Gen4 interface
- 3.2 TB NVMe local storage – mixed use type
-- 2x FPGA accelerators
- - Bitware [520N-MX][1]
+
+<!---
+2x FPGA accelerators
+Bitware [520N-MX][1]
+-->
In addition, the servers has the following parameters:
diff --git a/docs.it4i/environment-and-modules.md b/docs.it4i/environment-and-modules.md
index 6e75748776e894dc2f40d31b82e7d5ce7c141d96..dfb2f13e5a8df0db99e37e7edeb3d623a585a6dc 100644
--- a/docs.it4i/environment-and-modules.md
+++ b/docs.it4i/environment-and-modules.md
@@ -41,9 +41,9 @@ fi
### Application Modules
-In order to configure your shell for running particular application on clusters, we use a Module package interface.
+In order to configure your shell for running a particular application on clusters, we use a module package interface.
-Application modules on clusters are built using [EasyBuild][1]. The modules are divided into the following structure:
+Application modules on clusters are built using [EasyBuild][1]. The modules are divided into the following groups:
```
base: Default module class
@@ -72,7 +72,7 @@ Application modules on clusters are built using [EasyBuild][1]. The modules are
```
!!! note
- The modules set up the application paths, library paths and environment variables for running particular application.
+ The modules set up the application paths, library paths and environment variables for running a particular application.
The modules may be loaded, unloaded, and switched according to momentary needs. For details, see [lmod][2].
diff --git a/docs.it4i/general/energy.md b/docs.it4i/general/energy.md
index 35c9208675e0aaf9de92601b25618f2fbb684845..ea8f0ceae0fa865d19b7f108715627c8b94a5c37 100644
--- a/docs.it4i/general/energy.md
+++ b/docs.it4i/general/energy.md
@@ -19,7 +19,7 @@ The CPU core and GPU streaming multiprocessors frequency limit is implemented fo
### Performance Impact
The performance impact depends on the [arithmetic intensity][1] of the executed workload.
-The [arithmetic intensity][2] is a measure of floating-point operations (FLOPs) performed by a given code (or code section) relative to the amount of memory accesses (Bytes) that are required to support those operations. It is defined as a FLOP per Byte ratio (F/B). Arithmetic intensity is a characteristic of the computational algorithm.
+The [arithmetic intensity][2] is a measure of floating-point operations (FLOPs) performed by a given code (or code section) relative to the amount of memory accesses (Bytes) that are required to support those operations. It is defined as a FLOP per Byte ratio (F/B).Arithmetic intensity is a characteristic of the computational algorithm.
In general, the processor frequency [capping][3] has low performance impact for memory bound computations (arithmetic intensity below the [ridge point][2]). For processor bound computations (arithmetic intensity above the [ridge point][2]), the impact is proportional to the frequency reduction.
diff --git a/docs.it4i/general/resource-accounting.md b/docs.it4i/general/resource-accounting.md
index e8de68e1b817dd282c7cabe91144df3e9f91440a..55a9ef3c90e4acd06b8f7cce145235ce77884c28 100644
--- a/docs.it4i/general/resource-accounting.md
+++ b/docs.it4i/general/resource-accounting.md
@@ -45,12 +45,14 @@ $$
All jobs are accounted in normalized core-hours, using factor F valid at the time of the execution:
-| System | F | Validity |
-| --------------| ---: | -------- |
-| Karolina | 1.00 | 2021-08-02 to 2022-09-06 |
-| Barbora CPU | 1.40 | 2020-02-01 to 2022-09-06 |
-| Barbora GPU | 4.50 | 2020-04-01 to 2022-09-06 |
-| DGX-2 | 5.50 | 2020-04-01 to 2022-09-06 |
+| System | F |
+| --------------| ---: |
+| Karolina | 1.00 |
+| Barbora CPU | 1.40 |
+| Barbora GPU | 4.50 |
+| DGX-2 | 5.50 |
+
+Factors are valid as of July 9, 2022.
The normalized core-hours were introduced to treat systems of different age on equal footing.
Normalized core-hour is an accounting tool to discount the legacy systems.
diff --git a/docs.it4i/general/resources-allocation-policy.md b/docs.it4i/general/resources-allocation-policy.md
index 035e549127f2d629e59424d8eb57bccfbbd56033..e76c47a8221e8026403ee5be258a720e6ff039fd 100644
--- a/docs.it4i/general/resources-allocation-policy.md
+++ b/docs.it4i/general/resources-allocation-policy.md
@@ -1,4 +1,4 @@
-# Resources Allocation Policy
+# Resource Allocation Policy
## Job Queue Policies
diff --git a/docs.it4i/job-features.md b/docs.it4i/job-features.md
index ca9c0d3c35f7fc4da000b8a7a2d553d151ee4ed0..858015b254a502b998ac4fbc96654ff7092329e2 100644
--- a/docs.it4i/job-features.md
+++ b/docs.it4i/job-features.md
@@ -8,7 +8,7 @@ $ qsub... -l feature=req
## Xorg
-[Xorg][2] is a free and open source implementation of the X Window System imaging server maintained by the X.Org Foundation. Xorg is vailable only for Karolina accelerated nodes Acn[01-72].
+[Xorg][2] is a free and open source implementation of the X Window System imaging server maintained by the X.Org Foundation. Xorg is available only for Karolina accelerated nodes Acn[01-72].
```console
$ qsub ... -l xorg=True
@@ -83,7 +83,8 @@ N = number of compute nodes in the job.
## MSR-SAFE Support
-Load a kernel module that allows saving/restoring values of MSR registers. Uses LLNL MSR-SAFE.
+Load a kernel module that allows saving/restoring values of MSR registers.
+Uses [LLNL MSR-SAFE][a].
```console
$ qsub ... -l msr=version_string
@@ -121,7 +122,7 @@ where `PATTERN` is a list of core's numbers to offline, separated by the charact
## HDEEM Support
-Load the HDEEM software stack. The High Definition Energy Efficiency Monitoring (HDEEM) library is a software interface used to measure power consumption of HPC clusters with bullx blades.
+Load the HDEEM software stack. The [High Definition Energy Efficiency Monitoring][b] (HDEEM) library is a software interface used to measure power consumption of HPC clusters with bullx blades.
```console
$ qsub ... -l hdeem=version_string
@@ -181,3 +182,6 @@ $ source /lscratch/$PBS_JOBID/sbb.sh
[1]: software/tools/virtualization.md#tap-interconnect
[2]: general/accessing-the-clusters/graphical-user-interface/xorg.md
+
+[a]: https://software.llnl.gov/news/2019/04/29/msrsafe-1.3.0/
+[b]: https://tu-dresden.de/zih/forschung/projekte/hdeem
diff --git a/docs.it4i/karolina/compute-nodes.md b/docs.it4i/karolina/compute-nodes.md
index 50a5b73e002aefa3f5d075e4fa1a04693d584d2a..d07f4cad22ee5068ab292fb4aac24ab2efe8d465 100644
--- a/docs.it4i/karolina/compute-nodes.md
+++ b/docs.it4i/karolina/compute-nodes.md
@@ -61,11 +61,11 @@ Cloud compute nodes support both the research and operation of the Infrastructur
## Compute Node Summary
-| Node type | Count | Range | Memory | Cores | Queues (?) |
-| ---------------------------- | ----- | ------------ | ------- | -------------- | -------------------------- |
-| Nodes without an accelerator | 720 | Cn[001-720] | 256 GB | 128 @ 2.6 GHz | qexp, qprod, qlong, qfree |
-| Nodes with a GPU accelerator | 72 | Acn[01-72] | 1024 GB | 128 @ 2.45 GHz | qnvidia |
-| Data analytics nodes | 1 | Sdf1 | 24 TB | 768 @ 2.9 GHz | qfat |
+| Node type | Count | Range | Memory | Cores |
+| ---------------------------- | ----- | ------------ | ------- | -------------- |
+| Nodes without an accelerator | 720 | Cn[001-720] | 256 GB | 128 @ 2.6 GHz |
+| Nodes with a GPU accelerator | 72 | Acn[01-72] | 1024 GB | 128 @ 2.45 GHz |
+| Data analytics nodes | 1 | Sdf1 | 24 TB | 768 @ 2.9 GHz |
| Cloud partiton | 36 | CLn[01-36] | 256 GB | 128 @ 2.6 GHz | |
## Processor Architecture
diff --git a/docs.it4i/karolina/hardware-overview.md b/docs.it4i/karolina/hardware-overview.md
index 196337318451c964a0bf0655a4e648d4f429948d..75f7636c6c3b769f72a95f4c5d035dc1994d5716 100644
--- a/docs.it4i/karolina/hardware-overview.md
+++ b/docs.it4i/karolina/hardware-overview.md
@@ -17,7 +17,7 @@ The parameters are summarized in the following tables:
| Operating system | Linux |
| **Compute nodes** | |
| Total | 829 |
-| Processor cores | 128/768 (2x32 cores/2x64 cores/32x24 cores) |
+| Processor cores | 128/768 (2x64 cores/32x24 cores) |
| RAM | min. 256 GB |
| Local disk drive | no |
| Compute network | InfiniBand HDR |
diff --git a/docs.it4i/karolina/introduction.md b/docs.it4i/karolina/introduction.md
index 47bc340d3f8ecc6fb9b3b88c594373a917214c89..d0162bf22ad1406a8568180757138882d36901ed 100644
--- a/docs.it4i/karolina/introduction.md
+++ b/docs.it4i/karolina/introduction.md
@@ -1,12 +1,12 @@
# Introduction
-Karolina is the latest and most powerful supercomputer cluster built for IT4Innovations in Q2 of 2021. The Karolina cluster consists of 829 compute nodes, totaling 106,752 compute cores with 313 TB RAM, giving over 15.7 PFLOP/s theoretical peak performance and is ranked in the top 10 of the most powerful supercomputers in Europe.
+Karolina is the latest and most powerful supercomputer cluster built for IT4Innovations in Q2 of 2021. The Karolina cluster consists of 829 compute nodes, totaling 106,752 compute cores with 313 TB RAM, giving over 15.7 PFLOP/s theoretical peak performance.
-Nodes are interconnected through a fully non-blocking fat-tree InfiniBand network, and are equipped with AMD Zen 2, Zen3, and Intel Cascade Lake architecture processors. Seventy two nodes are also equipped with NVIDIA A100 accelerators. Read more in [Hardware Overview][1].
+Nodes are interconnected through a fully non-blocking fat-tree InfiniBand network, and are equipped with AMD Zen 2, Zen 3, and Intel Cascade Lake architecture processors. Seventy two nodes are also equipped with NVIDIA A100 accelerators. Read more in [Hardware Overview][1].
The cluster runs with an operating system compatible with the Red Hat [Linux family][a]. We have installed a wide range of software packages targeted at different scientific domains. These packages are accessible via the [modules environment][2].
-The user data shared file-system and job data shared file-system are available to users.
+The user data shared file system and job data shared file-system are available to users.
The [PBS Professional Open Source Project][b] workload manager provides [computing resources allocations and job execution][3].
@@ -14,7 +14,7 @@ Read more on how to [apply for resources][4], [obtain login credentials][5] and
[1]: hardware-overview.md
[2]: ../environment-and-modules.md
-[3]: ../general/resources-allocation-policy.md
+[3]: ../general/job-submission-and-execution.md
[4]: ../general/applying-for-resources.md
[5]: ../general/obtaining-login-credentials/obtaining-login-credentials.md
[6]: ../general/shell-and-data-access.md
diff --git a/docs.it4i/karolina/network.md b/docs.it4i/karolina/network.md
index 7981a2afe6709d7f7e91031ea0a5ab2009abfa59..f74d0accb6a26b126ebe5ceef9430d628e87f2a4 100644
--- a/docs.it4i/karolina/network.md
+++ b/docs.it4i/karolina/network.md
@@ -1,8 +1,8 @@
# Network
-All of the compute and login nodes of Karolina are interconnected through an [InfiniBand][a] HDR 200 Gbps network and a Gigabit Ethernet network.
+All of the compute and login nodes of Karolina are interconnected through an [InfiniBand][a] HDR 200Gbps network and a gigabit ethernet network.
-The Compute network is configured as a non-blocking Fat Tree which consists of 60 x 40-ports Mellanox Quantum™ HDR switches (40 Leaf HDR switches and 20 Spine HDR switches).
+The compute network is configured as a non-blocking Fat Tree which consists of 60 x 40-ports Mellanox Quantum™ HDR switches (40 Leaf HDR switches and 20 Spine HDR switches).
<br>*For a higher resolution, open the image in a new browser tab.*
@@ -11,7 +11,7 @@ Compute nodes and the service infrastructure is connected by the HDR100 technolo
**The compute network has the following parameters**
* 100Gbps
-* Latencies less than 10 microseconds (0.6 μs end-to-end, <90ns switch hop)
+* Latencies less than 10 microseconds (0.6μs end-to-end, <90ns switch hop)
* Adaptive routing support
* MPI communication support
* IP protocol support (IPoIB)
@@ -19,8 +19,8 @@ Compute nodes and the service infrastructure is connected by the HDR100 technolo
## Mellanox Quantum™ QM8790 40-Ports Switch
-Mellanox provides the world’s smartest switch, enabling in-network computing through the Co-Design Scalable Hierarchical Aggregation and Reduction Protocol (SHARP)™ technology.
-QM8790 has the highest fabric performance available in the market with up to 16Tb/s of non-blocking bandwidth with sub-130ns port-to-port latency
+[Mellanox][b] provides the world’s smartest switch, enabling in-network computing through the Co-Design Scalable Hierarchical Aggregation and Reduction Protocol (SHARP)™ technology.
+QM8790 has the highest fabric performance available in the market with up to 16Tb/s of non-blocking bandwidth with sub-130ns port-to-port latency.
**Performance**
@@ -43,3 +43,4 @@ QM8790 has the highest fabric performance available in the market with up to 16T
* VL mapping (VL2VL)
[a]: http://en.wikipedia.org/wiki/InfiniBand
+[b]: https://network.nvidia.com/files/doc-2020/pb-qm8790.pdf
diff --git a/docs.it4i/karolina/storage.md b/docs.it4i/karolina/storage.md
index 65c41b049e3b02e72cd1b008a37191594d573c75..a97b86b14723bd130d6bb30b3d7ef7438567a207 100644
--- a/docs.it4i/karolina/storage.md
+++ b/docs.it4i/karolina/storage.md
@@ -4,15 +4,14 @@ Karolina cluster provides two main shared filesystems, [HOME filesystem][1] and
## Archiving
-Do not use shared filesystems as a backup for large amount of data or long-term archiving mean. The academic staff and students of research institutions in the Czech Republic can use the CESNET storage service, which is available via SSHFS.
-For more information, see the [CESNET storage][6] section.
+Shared filesystems should not be used as a backup for large amount of data or long-term data storage. The academic staff and students of research institutions in the Czech Republic can use the [CESNET storage][6] service, which is available via SSHFS.
### HOME File System
The HOME filesystem is an HA cluster of two active-passive NFS servers. This filesystem contains users' home directories `/home/username`. Accessible capacity is 31 TB, shared among all users. Individual users are restricted by filesystem usage quotas, set to 25 GB per user. Should 25 GB prove insufficient, contact [support][d], the quota may be increased upon request.
!!! note
- The HOME filesystem is intended for preparation, evaluation, processing and storage of data generated by active Projects.
+ The HOME filesystem is intended for preparation, evaluation, processing and storage of data generated by active projects.
The files on HOME filesystem will not be deleted until the end of the [user's lifecycle][4].
@@ -20,7 +19,7 @@ The filesystem is backed up, so that it can be restored in case of a catastrophi
| HOME filesystem | |
| -------------------- | ------------------------------ |
-| Accesspoint | /home/username |
+| Mountpoint | /home/username |
| Capacity | 31 TB |
| Throughput | 1.93 GB/s write, 3.1 GB/s read |
| User space quota | 25 GB |
@@ -53,7 +52,7 @@ The filesystem is backed up, so that it can be restored in case of a catastrophi
### SCRATCH File System
-The SCRATCH filesystem is realized as a parallel Lustre filesystem. It is accessible via the Infiniband network and is available from all login and computational nodes. Extended ACLs are provided on the Lustre filesystems for sharing data with other users using fine-grained control. For basic information about Lustre, see the [Understanding the Lustre Filesystem][7] subsection of the Barbora's storage documentation.*
+The SCRATCH filesystem is realized as a parallel Lustre filesystem. It is accessible via the Infiniband network and is available from all login and compute nodes. Extended ACLs are provided on the Lustre filesystems for sharing data with other users using fine-grained control. For basic information about Lustre, see the [Understanding the Lustre Filesystems][7] subsection of the Barbora's storage documentation.
The SCRATCH filesystem is mounted in directory /scratch. Users may freely create subdirectories and files on the filesystem. Accessible capacity is 1000 TB, shared among all users. Users are restricted by PROJECT quotas set to 20 TB. The purpose of this quota is to prevent runaway programs from filling the entire filesystem and deny service to other users. Should 20 TB prove insufficient, contact [support][d], the quota may be increased upon request.
@@ -111,12 +110,12 @@ Configuration of the storage:
### PROJECT File System
-The PROJECT data storage is a central storage for projects'/users' data on IT4Innovations that is accessible from all clusters.
-For more information, see the [PROJECT storage][9] section.
+The PROJECT data storage is a central storage for projects' and users' data at IT4Innovations that is accessible from all clusters.
+For more information, see the [PROJECT Data Storage][9] section.
### Disk Usage and Quota Commands
-For more information about Disk usage and user quotas, see the Barbora's [storage section][8].
+For more information about disk usage and user quotas, see the Barbora's [storage section][8].
### Extended ACLs
@@ -130,7 +129,7 @@ For more information, see the [Access Control List][10] section of the documenta
### TMP
-Each node is equipped with local /tmp directory of few GB capacity. The /tmp directory should be used to work with small temporary files. Old files in /tmp directory are automatically purged.
+Each node is equipped with a local `/tmp` directory of few GB capacity. The `/tmp` directory should be used to work with small temporary files. Old files in the `/tmp` directory are automatically purged.
## Summary
diff --git a/docs.it4i/software/cae/comsol/comsol-multiphysics.md b/docs.it4i/software/cae/comsol/comsol-multiphysics.md
index d4b610109413207774066a2c16dcca865c0c827c..c71bf25b9a17e6ac8201e87de777c95085bcc300 100644
--- a/docs.it4i/software/cae/comsol/comsol-multiphysics.md
+++ b/docs.it4i/software/cae/comsol/comsol-multiphysics.md
@@ -2,7 +2,7 @@
## Introduction
-[COMSOL][a] is a powerful environment for modelling and solving various engineering and scientific problems based on partial differential equations. COMSOL is designed to solve coupled or multiphysics phenomena. For many standard engineering problems, COMSOL provides add-on products such as electrical, mechanical, fluid flow, and chemical applications.
+[COMSOL][a] is a powerful environment for modelling and solving various engineering and scientific problems based on partial differential equations. COMSOL is designed to solve coupled or multiphysics phenomena. For many standard engineering problems, COMSOL provides add-on products (modules) such as electrical, mechanical, fluid flow, and chemical applications.
* [Structural Mechanics Module][b],
* [Heat Transfer Module][c],
@@ -14,28 +14,22 @@ COMSOL also allows an interface support for equation-based modelling of partial
## Execution
-The latest available COMSOL version on the clusters is 5.2. There are two variants of the release:
+There are two types of license:
* **Non commercial** or so called **EDU variant**, which can be used for research and educational purposes.
* **Commercial** or so called **COM variant**, which can used also for commercial activities. **COM variant** has only subset of features compared to the **EDU variant** available. More about licensing [here][1].
-To load the default `COMSOL` module ([**not** recommended][4]), use:
-
-```console
-$ ml COMSOL
-```
-
-By default, the **EDU variant** will be loaded. If you needs other version or variant, load the particular version. To obtain the list of available versions, use:
+By default, the **EDU variant** is loaded. If you needs other version or variant, load the particular version. To obtain the list of available versions, use:
```console
$ ml av COMSOL
```
-To prepare COMSOL jobs in the interactive mode, we recommend using COMSOL on the compute nodes via the PBS Pro scheduler. To run the COMSOL Desktop GUI on Windows.
+To prepare COMSOL jobs in the interactive mode, we recommend using COMSOL on the compute nodes via the PBS Pro scheduler.
!!! Note
- We recommend using the [Virtual Network Computing (VNC)][2].
+ To run the COMSOL Desktop GUI on Windows, we recommend using the [Virtual Network Computing (VNC)][2].
Example for Karolina:
@@ -65,7 +59,7 @@ you need to run COMSOL with additional parameters:
$ comsol -3drend sw
```
-To run COMSOL in batch mode without the COMSOL Desktop GUI environment, utilize the default (comsol.pbs) job script and execute it via the `qsub` command:
+To run COMSOL in batch mode without the COMSOL Desktop GUI environment, utilize the following (`comsol.pbs`) job script and execute it via the `qsub` command:
```bash
#!/bin/bash
@@ -116,13 +110,13 @@ After that, start your COMSOL Desktop GUI session and fill required information
## LiveLink for MATLAB
-COMSOL is a software package for the numerical solution of partial differential equations. LiveLink for MATLAB allows connection to the COMSOL API (Application Programming Interface) with the benefits of the programming language and computing environment of the MATLAB.
+COMSOL is a software package for the numerical solution of partial differential equations. LiveLink for MATLAB allows connection to the COMSOL API (Application Programming Interface) with the benefits of the programming language and computing environment of MATLAB.
LiveLink for MATLAB is available in both **EDU** and **COM** **variant** of the COMSOL release. On the clusters there is 1 commercial (**COM**) and 5 educational (**EDU**) licenses of LiveLink for MATLAB (see the [ISV Licenses][3]). The following example shows how to start COMSOL model from MATLAB via LiveLink in the interactive mode.
```console
$ qsub -I -X -A PROJECT_ID -q qexp -l select=1:ncpus=128:mpiprocs=128
-$ ml MATLAB/R2015b COMSOL/5.2.0-EDU
+$ ml <matlab_module> and <comsol_module>
$ comsol -3drend sw server MATLAB
```
@@ -147,7 +141,7 @@ echo '**PBS_NODEFILE***END*********'
text_nodes < cat $PBS_NODEFILE
-ml MATLAB/R2015b COMSOL/5.2.0-EDU
+ml <matlab_module> and <comsol_module>
ntask=$(wc -l $PBS_NODEFILE)
@@ -156,10 +150,10 @@ cd $EBROOTCOMSOL/mli
matlab -nodesktop -nosplash -r "mphstart; addpath /scratch/project/PROJECT_ID; test_job"
```
-This example shows how to run LiveLink for MATLAB with the following configuration: 3 nodes and 128 cores per node. A working directory has to be created before submitting (comsol_matlab.pbs) job script into the queue. The input file (test_job.m) has to be in the working directory or a full path to the input file has to be specified. The Matlab command option (`-r ”mphstart”`) created a connection with a COMSOL server using the default port number.
+This example shows how to run LiveLink for MATLAB with the following configuration: 3 nodes and 128 cores per node. A working directory has to be created before submitting (comsol_matlab.pbs) job script into the queue. The input file (test_job.m) has to be in the working directory or a full path to the input file has to be specified. The MATLAB command option (`-r ”mphstart”`) created a connection with a COMSOL server using the default port number.
[1]: licensing-and-available-versions.md
-[2]: ../../../general/accessing-the-clusters/graphical-user-interface/x-window-system.md
+[2]: ../../../general/accessing-the-clusters/graphical-user-interface/vnc.md
[3]: ../../isv_licenses.md
[4]: ../../../modules/lmod/#loading-modules
diff --git a/docs.it4i/software/isv_licenses.md b/docs.it4i/software/isv_licenses.md
index 2a02e5f8e0f6bab52c609507e0b59f8e248b7eda..544dafa7c89912f32ef49c966a66070bcef488b1 100644
--- a/docs.it4i/software/isv_licenses.md
+++ b/docs.it4i/software/isv_licenses.md
@@ -2,9 +2,9 @@
## Guide to Managing Independent Software Vendor Licenses
-On IT4I clusters, there are also installed commercial software applications, also known as ISV (Independent Software Vendor), which are subjects to licensing. The licenses are limited and their usage may be restricted only to some users or user groups.
+On IT4I clusters, there are also installed commercial software applications, also known as ISV (Independent Software Vendor), which are subjects to licensing. Licenses are limited and their usage may be restricted only to some users or user groups, or based on other conditions.
-Currently, Flex License Manager based licensing is supported on the cluster for products ANSYS, Comsol, and MATLAB. More information about the applications can be found in the general software section.
+Currently, [Flex License Manager][c] based licensing is supported on the cluster for products ANSYS, Comsol, and MATLAB. More information about the applications can be found in the general software section.
If an ISV application was purchased for educational (research) purposes and also for commercial purposes, then there are always two separate versions maintained and suffix "edu" is used in the name of the non-commercial version.
@@ -43,7 +43,7 @@ To list all Ansys modules, use:
lmstat -i -c 1055@license.it4i.cz
```
-For other applications' licenses, change the port number in the command according to the **Port** column in [this list][b].
+For other applications' licenses, change the port number in the command according to the **Port** column on the [licelin website][b] (requires IT4I VPN).
## License Aware Job Scheduling
@@ -71,12 +71,13 @@ Do not hesitate to ask IT4I support for support of additional license features y
Run an interactive PBS job with 1 Matlab EDU license:
```console
-$ qsub -I -q qprod -A PROJECT_ID -l select=2 -l license__matlab-edu__MATLAB=1
+qsub -I -q qprod -A PROJECT_ID -l select=2 -l license__matlab-edu__MATLAB=1
```
-The license is used and accounted only with the real usage of the product. So in this example, the general Matlab is used after Matlab is run by the user and not at the time, when the shell of the interactive job is started. In addition, the Distributed Computing licenses are used at the time, when the user uses the distributed parallel computation in Matlab (e. g. issues pmode start, matlabpool, etc.).
+The license is used and accounted only with the real usage of the product. So in this example, the general MATLAB is used after MATLAB is run by the user and not at the time, when the shell of the interactive job is started. In addition, the Distributed Computing licenses are used at the time, when the user uses the distributed parallel computation in MATLAB (e.g. issues pmode start, matlabpool, etc.).
[1]: #Licence
[a]: https://extranet.it4i.cz/rsweb/barbora/licenses
[b]: http://licelin.it4i.cz/list/
+[c]: https://www.revenera.com/software-monetization/products/software-licensing/flexnet-licensing
diff --git a/docs.it4i/software/sdk/nvhpc.md b/docs.it4i/software/sdk/nvhpc.md
new file mode 100644
index 0000000000000000000000000000000000000000..e47dcb4814943dc519590296c8c8e3cfbae59460
--- /dev/null
+++ b/docs.it4i/software/sdk/nvhpc.md
@@ -0,0 +1,87 @@
+<style type="text/css">
+.tg {border-collapse:collapse;border-spacing:0;}
+.tg td{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:12px;
+ overflow:hidden;padding:10px 5px;word-break:normal;}
+.tg th{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:12px;
+ font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;}
+.tg .tg-lzqt{background-color:#656565;border-color:inherit;color:#ffffff;font-weight:bold;text-align:center;vertical-align:top}
+.tg .tg-c3ow{border-color:inherit;text-align:center;vertical-align:top}
+.tg .tg-7btt{border-color:inherit;font-weight:bold;text-align:center;vertical-align:top}
+</style>
+
+# NVIDIA HPC SDK
+
+The NVIDIA HPC Software Development Kit includes the proven compilers, libraries, and software tools
+essential to maximizing developer productivity and the performance and portability of HPC applications.
+
+## Installed Versions
+
+Different versions are available on Karolina, Barbora, and DGX-2.
+For the current version use the command:
+
+```console
+ml av nvhpc
+```
+
+## Components
+
+Below is the list of components in the NVIDIA HPC SDK.
+
+<table class="tg">
+<thead>
+ <tr>
+ <th class="tg-lzqt" colspan="7">Development</th>
+ <th class="tg-lzqt" colspan="2">Analysis</th>
+ </tr>
+</thead>
+<tbody>
+ <tr>
+ <td class="tg-7btt">Programming<br>Models</td>
+ <td class="tg-7btt" colspan="2">Compilers</td>
+ <td class="tg-7btt">Core<br>Libraries</td>
+ <td class="tg-7btt" colspan="2">Math<br>Libraries</td>
+ <td class="tg-7btt">Communication<br>Libraries</td>
+ <td class="tg-7btt">Profilers</td>
+ <td class="tg-7btt">Debuggers</td>
+ </tr>
+ <tr>
+ <td class="tg-c3ow"><a href="https://docs.nvidia.com/hpc-sdk/compilers/c++-parallel-algorithms/index.html" target="blank">Standard C++</a> & <a href="https://docs.nvidia.com/hpc-sdk/compilers/cuda-fortran-prog-guide/index.html" target="">Fortran</a></td>
+ <td class="tg-c3ow"><a href="https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html" target="blank">nvcc</a></td>
+ <td class="tg-c3ow"><a href="https://docs.nvidia.com/hpc-sdk/compilers/hpc-compilers-user-guide/index.html" target="blank">nvc</a></td>
+ <td class="tg-c3ow"><a href="https://nvidia.github.io/libcudacxx/" target="blank">libcu++</a></td>
+ <td class="tg-c3ow"><a href="https://docs.nvidia.com/cuda/cublas/index.html#abstract" target="blank">cuBLAS</a></td>
+ <td class="tg-c3ow"><a href="https://docs.nvidia.com/cuda/cutensor/index.html" target="blank">cuTENSOR</a></td>
+ <td class="tg-c3ow"><a href="https://docs.nvidia.com/hpc-sdk/compilers/hpc-compilers-user-guide/index.html#mpi-use" target="blank">Open MPI</a></td>
+ <td class="tg-c3ow">Nsight</td>
+ <td class="tg-c3ow"><a href="https://docs.nvidia.com/cuda/cuda-gdb/index.html" target="blank">Cuda-gdb</a></td>
+ </tr>
+ <tr>
+ <td class="tg-c3ow"><a href="https://docs.nvidia.com/hpc-sdk/compilers/openacc-gs/index.html" target="blank">OpenACC</a> & <a href="https://docs.nvidia.com/hpc-sdk/compilers/hpc-compilers-user-guide/index.html#openmp-use" target="blank">OpenMP</a></td>
+ <td class="tg-c3ow" colspan="2"><a href="https://docs.nvidia.com/hpc-sdk/compilers/hpc-compilers-user-guide/index.html" target="blank">nvc++</a></td>
+ <td class="tg-c3ow"><a href="https://docs.nvidia.com/cuda/thrust/" target="blank">Thrust</a></td>
+ <td class="tg-c3ow"><a href="https://docs.nvidia.com/cuda/cusparse/index.html#abstract" target="blank">cuSPARSE</a></td>
+ <td class="tg-c3ow"><a href="https://docs.nvidia.com/cuda/cusolver/index.html#abstract" target="blank">cuSOLVER</a></td>
+ <td class="tg-c3ow"><a href="https://docs.nvidia.com/nvshmem/" target="blank">NVSHMEM</a></td>
+ <td class="tg-c3ow"><a href="https://docs.nvidia.com/nsight-systems/" target="blank">Systems</a></td>
+ <td class="tg-c3ow">Host</td>
+ </tr>
+ <tr>
+ <td class="tg-c3ow"><a href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html" target="blank">CUDA</a></td>
+ <td class="tg-c3ow" colspan="2"><a href="https://docs.nvidia.com/hpc-sdk/compilers/hpc-compilers-user-guide/index.html" target="blank">nvfortran</a></td>
+ <td class="tg-c3ow"><a href="https://docs.nvidia.com/cuda/cub/index.html" target="blank">CUB</a></td>
+ <td class="tg-c3ow"><a href="https://docs.nvidia.com/cuda/cufft/index.html#abstract" target="blank">cuFFT</a></td>
+ <td class="tg-c3ow"><a href="https://docs.nvidia.com/cuda/curand/index.html" target="blank">cuRAND</a></td>
+ <td class="tg-c3ow"><a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/index.html" target="blank">NCCL</a></td>
+ <td class="tg-c3ow"><a href="https://docs.nvidia.com/nsight-compute/" target="blank">Compute</a></td>
+ <td class="tg-c3ow">Device</td>
+ </tr>
+</tbody>
+</table>
+
+## References
+
+[NVIDIA HPC SDK homepage][1]<br>
+[Documentation][2]
+
+[1]: https://developer.nvidia.com/hpc-sdk
+[2]: https://docs.nvidia.com/hpc-sdk/index.html
diff --git a/mkdocs.yml b/mkdocs.yml
index c06134997321c92e5355952cdbbd941780468c88..9a59f54637ef3e980673ba6e572f340fec6c6613 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -74,7 +74,7 @@ nav:
- Run Jobs:
- Introduction: general/resource_allocation_and_job_execution.md
- Resources Allocation:
- - Resources Allocation Policy: general/resources-allocation-policy.md
+ - Resource Allocation Policy: general/resources-allocation-policy.md
- Karolina Queues: general/karolina-queues.md
- Barbora Queues: general/barbora-queues.md
- Resource Accounting Policy: general/resource-accounting.md
@@ -224,6 +224,7 @@ nav:
- HDF5: software/numerical-libraries/hdf5.md
- Intel Numerical Libraries: software/numerical-libraries/intel-numerical-libraries.md
- PETSc: software/numerical-libraries/petsc.md
+ - NVIDIA HPC SDK: software/sdk/nvhpc.md
- Languages:
- Java: software/lang/java.md
- C#: software/lang/csc.md