Sweet caroline doc merge

docs.it4i/general/shell-and-data-access.md

@@ -7,6 +7,16 @@ All IT4Innovations clusters are accessed by the SSH protocol via login nodes at
 !!! note
     The **cluster-name.it4i.cz** alias is currently not available through VPN connection. Use **loginX.cluster-name.it4i.cz** when connected to VPN.
 
+### Karolina Cluster
+
+| Login address           | Port | Protocol | Login node                            |
+| ----------------------- | ---- | -------- | ------------------------------------- |
+| karolina.it4i.cz        | 22   | SSH      | round-robin DNS record for login[1-4] |
+| login1.karolina.it4i.cz | 22   | SSH      | login1                                |
+| login2.karolina.it4i.cz | 22   | SSH      | login2                                |
+| login3.karolina.it4i.cz | 22   | SSH      | login3                                |
+| login4.karolina.it4i.cz | 22   | SSH      | login4                                |
+
 ### Barbora Cluster
 
 | Login address             | Port | Protocol | Login node                            |
@@ -101,9 +111,19 @@ Last login: Tue Jul 9 15:57:38 2013 from your-host.example.com
 
 Data in and out of the system may be transferred by SCP and SFTP protocols.
 
+### Karolina Cluster
+
+| Address                 | Port | Protocol  |
+| ----------------------- | ---- | --------- |
+| karolina.it4i.cz        | 22   | SCP, SFTP |
+| login1.karolina.it4i.cz | 22   | SCP, SFTP |
+| login2.karolina.it4i.cz | 22   | SCP, SFTP |
+| login3.karolina.it4i.cz | 22   | SCP, SFTP |
+| login4.karolina.it4i.cz | 22   | SCP, SFTP |
+
 ### Barbora Cluster
 
-| Address                  | Port | Protocol |
+| Address                   | Port | Protocol |
 | ------------------------- | ---- | ------- |
 | barbora.it4i.cz           | 22   | SCP     |
 | login1.barbora.it4i.cz    | 22   | SCP     |

docs.it4i/karolina/compute-nodes.md

+# Compute Nodes
+
+Karolina is a cluster of x86-64 AMD- and Intel-based nodes built with the HPE technology. The cluster contains four types of compute nodes.
+
+## Compute Nodes Without Accelerators
+
+Standard compute nodes without accelerators (such as GPUs or FPGAs) are based on the x86 CPU architecture and provide quick accessibility for the users and their existing codes.
+
+* 720 nodes
+* 92,160 cores in total
+* 2x AMD Zen 2 EPYC™ 7H12, 64-core, 2.6 GHz processors per node
+* 256 GB DDR4 3200MT/s of physical memory per node
+* 5,324.8 GFLOP/s per compute node
+* 1x 100 Gb/s Ethernet
+* 1x 100 Gb/s IB port
+* Cn[001-720]
+
+![](img/apolloproliant.png)
+
+## Compute Nodes With a GPU Accelerator
+
+Accelerated compute nodes deliver most of the compute power usable for HPC as well as excellent performance in HPDA and AI workloads, especially in the learning phase of Deep Neural Networks.
+
+* 70 nodes
+* 4,480 cores in total
+* 2x AMD Zen 2 EPYC™ 7452, 32-core, 2.35 GHz processors per node
+* 512 GB DDR4 3200MT/s of physical memory per node
+* 8x GPU accelerator NVIDIA A100 per node
+* 2,406.4 GFLOP/s per compute node
+* GPU-to-GPU All-to-All NVLINK 2.0, GPU-Direct(?)
+* 4x 200 Gb/s Ethernet
+* 4x 200 Gb/s IB port
+* Acn[01-70]
+
+![](img/hpeapollo6500.png)
+
+## Data Analytics Compute Node
+
+Data analytics compute node is oriented on supporting huge memory jobs by implementing a NUMA SMP system with large cache coherent memory.
+
+* 1x HPE Superdome Flex server
+* 768 cores in total
+* 32x Intel® Xeon® Platinum, 24-core, 2.9 GHz, 205W
+* 24 TiB DDR4 2993MT/s of physical memory per node
+* 2x 200 Gb/s Ethernet
+* 2x 200 Gb/s IB port
+* 71.2704 TFLOP/s
+* DAcn1
+
+![](img/superdomeflex.png)
+
+## Cloud Compute Node
+
+Cloud compute nodes support both the research and operation of the Infrastructure/HPC as a Service. It is intended for provision and operation of cloud technologies like OpenStack and Kubernetes.
+
+* 36 nodes
+* 4,608 cores in total
+* 2x AMD Zen 2 EPYC™ 7H12, 64-core, 2.6 GHz processors per node
+* 256 GB DDR4 3200MT/s of physical memory per node
+* HPE ProLiant XL225n Gen10 Plus servers
+* 5,324.8 GFLOP/s per compute node
+* 1x 100 Gb/s Ethernet
+* 1x 100 Gb/s IB port
+* CLn[01-36]
+
+## Compute Node Summary
+
+| Node type                    | Count | Range       | Memory  | Cores          | Queues (?)                  |
+| ---------------------------- | ----- | ----------- | ------  | -----------    | -------------------------- |
+| Nodes without an accelerator | 720   | Cn[001-720]  | 256 GB  | 128 @ 2.6 GHz  | qexp, qprod, qlong, qfree  |
+| Nodes with a GPU accelerator | 70    | Acn[01-70]   | 512 GB  | 64 @ 2.35 GHz  | qnvidia                    |
+| Data analytics nodes         | 1     | DAcn1        | 24 TB   | 768 @ 2.9 GHz  | qfat                       |
+| Cloud partiton               | 36    | CLn[01-36]   | 256 GB  | 128 @ 2.6 GHz  |                            |
+
+## Processor Architecture
+
+Karolina is equipped with AMD Zen 2 EPYC™ 7H12 (nodes without accelerators, Cloud partiton), AMD Zen 2 EPYC™ 7452 (nodes with accelerators), and Intel Cascade Lake Xeon-SC 8268 (Data analytics partition).
+
+### AMD [Zen 2 EPYC™ 7H12][d]
+
+EPYC™ 7H12 is a 64-bit 64-core x86 server microprocessor designed and introduced by AMD in late 2019. This multi-chip processor, which is based on the Zen 2 microarchitecture, incorporates logic fabricated TSMC 7 nm process and I/O fabricated on GlobalFoundries 14 nm process. The 7H12 has a TDP of 280 W with a base frequency of 2.6 GHz and a boost frequency of up to 3.3 GHz. This processor supports up to two-way SMP and up to 4 TiB of eight channels DDR4-3200 memory per socket.
+
+* **Family**: EPYC™
+* **Cores**: 64
+* **Threads**: 128
+* **L1I Cache**: 2 MiB, 64x32 KiB, 8-way set associative
+* **L1D Cache**: 2 MiB, 64x32 KiB, 8-way set associative
+* **L2 Cache**: 32 MiB, 64x512 KiB, 8-way set associative, write-back
+* **L3 Cache**: 256 MiB, 16x16 MiB
+* **Instructions**(?): x86-64, MOVBE, MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, POPCNT, AVX, AVX2, AES, PCLMUL, FSGSBASE, RDRND, FMA3, F16C, BMI, BMI2, VT-x, VT-d, TXT, TSX, RDSEED, ADCX, PREFETCHW, CLFLUSHOPT, XSAVE, SGX, MPX, AVX-512 (New instructions for [Vector Neural Network Instructions][c])
+* **Frequency**: 2.6 GHz
+* **Max turbo**: 3.3 GHz
+* **Process**: 7 nm, 14 nm
+* **TDP**: 280 W
+
+### AMD [Zen 2 EPYC™ 7452][e]
+
+EPYC™ 7452 is a 64-bit dotriaconta-core x86 server microprocessor designed and introduced by AMD in mid-2019. This multi-chip processor, which is based on the Zen 2 microarchitecture, incorporates logic fabricated TSMC 7 nm process and I/O fabricated on GlobalFoundries 14 nm process. The 7452 has a TDP of 155 W with a base frequency of 2.2 GHz and a boost frequency of up to 3.35 GHz. This processor supports up to two-way SMP and up to 4 TiB of eight channels DDR4-3200 memory per socket.
+
+* **Family**: EPYC™
+* **Cores**: 32
+* **Threads**: 64
+* **L1I Cache**: 1 MiB, 32x32 KiB, 8-way set associative
+* **L1D Cache**: 1 MiB,	32x32 KiB, 8-way set associative
+* **L2 Cache**: 16 MiB, 32x512 KiB, 8-way set associative
+* **L3 Cache**: 128 MiB, 18x16 MiB, 8-way set associative, write-back
+* **Instructions**: x86-64, MOVBE, MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, POPCNT, AVX, AVX2, AES, PCLMUL, FSGSBASE, RDRND, FMA3, F16C, BMI, BMI2, VT-x, VT-d, TXT, TSX, RDSEED, ADCX, PREFETCHW, CLFLUSHOPT, XSAVE, SGX, MPX, AVX-512
+* **Frequency**: 2.35 GHz
+* **Max turbo**: 3.35 GHz
+* **Process**: 7 nm, 14 nm
+* **TDP**: 155 W
+
+### Intel [Skylake Platinum 8268][f]
+
+Xeon Platinum 8268 is a 64-bit 24-core x86 high-performance server microprocessor introduced by Intel in early 2019. The Platinum 8268 is based on the Cascade Lake microarchitecture and is manufactured on a 14 nm process. This chip supports 8-way multiprocessing, sports 2 AVX-512 FMA units as well as three Ultra Path Interconnect links. This microprocessor supports up 1 TiB of hexa-channel DDR4-2933 memory, operates at 2.9 GHz with a TDP of 205 W and features a turbo boost frequency of up to 3.9 GHz.
+
+* **Family**: Xeon Platinum
+* **Cores**: 24
+* **Threads**: 48
+* **L1I Cache**: 768 KiB, 24x32 KiB, 8-way set associative
+* **L1D Cache**: 768 KiB, 24x32 KiB, 8-way set associative, write-back
+* **L2 Cache**: 24 MiB, 24x1 MiB, 16-way set associative, write-back
+* **L3 Cache**: 35.75 MiB, 26x1.375 MiB, 11-way set associative, write-back
+* **Instructions**: x86-64, MOVBE, MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, POPCNT, AVX, AVX2, AES, PCLMUL, FSGSBASE, RDRND, FMA3, F16C, BMI, BMI2, VT-x, VT-d, TXT, TSX, RDSEED, ADCX, PREFETCHW, CLFLUSHOPT, XSAVE, SGX, MPX, AVX-512
+* **Frequency**: 2.9 GHz
+* **Max turbo**: 3.9 GHz
+* **Process**: 14 nm
+* **TDP**: 205 W
+
+## GPU Accelerator
+
+Karolina is equipped with an [NVIDIA A100][g] accelerator.
+
+|NVIDIA A100||
+| --- | --- |
+| GPU Architecture | NVIDIA Ampere |
+| NVIDIA Tensor| Cores: 432 |
+| NVIDIA CUDA® Cores | 6912 |
+| Double-Precision Performance | 9.7 TFLOP/s |
+| Single-Precision Performance | 19.5 TFLOP/s |
+| Tensor Performance | 312 TFLOP/s |
+| GPU Memory | 40 GB HBM2 |
+| Memory Bandwidth | 1555 GB/sec |
+| ECC | Yes |
+| Interconnect Bandwidth | 600 GB/sec |
+| System Interface | NVIDIA NVLink |
+| Form Factor | SXM4 | (sxm 8?)
+| Max Power Consumption | 400 W |
+| Thermal Solution | Passive |
+| Compute APIs | CUDA, DirectCompute,OpenCLTM, OpenACC | (?)
+
+[c]: https://en.wikichip.org/wiki/x86/avx512vnni
+[d]: https://en.wikichip.org/wiki/amd/epyc/7h12
+[e]: https://en.wikichip.org/wiki/amd/epyc/7452
+[f]: https://en.wikichip.org/wiki/intel/xeon_platinum/8268
+[g]: https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/a100-80gb-datasheet-update-nvidia-us-1521051-r2-web.pdf

docs.it4i/karolina/hardware-overview.md

+# Hardware Overview
+
+Karolina consists of 827 computational nodes of which 720 are universal compute nodes (**Cn[001-720]**), 70 are NVIDIA A100 accelerated nodes (**Acn[01-70]**), 1 is a data analytics node (**DAcn1**), and 36 are cloud partitions (**CLn[01-36]**). Each node is a powerful x86-64 computer, equipped with 64/128/768 cores (64-core AMD EPYC™ 7H12 / 32-core AMD EPYC™ 7452 / 24-core Intel Xeon-SC 8268) and at least 256 GB of RAM.
+
+[User access][5] to Karolina is provided by four login nodes **login[1-4]**. The nodes are interlinked through high speed InfiniBand and Ethernet networks.
+
+The Accelerated nodes, Data analytics node, and Cloud nodes are available [upon request][a] from a PI. For more information about accessing the nodes, see also the [Resources Allocation Policy][2] section.
+
+For more technical information, see the [Compute Nodes][1] section.
+
+The parameters are summarized in the following tables:
+
+| **In general**                              |                                                |
+| ------------------------------------------- | ---------------------------------------------- |
+| Primary purpose                             | High Performance Computing                     |
+| Architecture of compute nodes               | x86-64                                         |
+| Operating system                            | Linux                                          |
+| **Compute nodes**                           |                                                |
+| Total                                       | 827                                            |
+| Processor cores                             | 64/128/768 (2x32 cores/2x64 cores/32x24 cores) |
+| RAM                                         | min. 256 GB                                    |
+| Local disk drive                            | no                                             |
+| Compute network                             | InfiniBand HDR                                 |
+| Universal compute node                      | 720, Cn[001-720]                               |
+| Accelerated compute nodes                   | 70, Acn[01-70]                                 |
+| Data analytics compute nodes                | 1, DAcn1                                       |
+| Cloud compute nodes                         | 36, CLn[01-36]                                 |
+| **In total**                                |                                                |
+| Total theoretical peak performance  (Rpeak) | 15.2 PFLOP/s                                   |
+| Total amount of RAM                         | 248 TB                                         |
+
+| Node                     | Processor                                | Memory  | Accelerator            |
+| ------------------------ | ---------------------------------------  | ------  | ---------------------- |
+| Universal compute node   | 2 x AMD Zen 2 EPYC™ 7H12, 2.6 GHz        | 256 GB  | -                      |
+| Accelerated compute node | 2 x AMD Zen 2 EPYC™ 7452, 2.35 GHz       | 512 GB  | NVIDIA A100            |
+| Data analytics node      | 32 x Intel Xeon-SC 8268, 2.9 GHz         | 24 TB   | -                      |
+| Cloud compute node       | 2 x AMD Zen 2 EPYC™ 7H12, 2.6 GHz        | 256 GB  | -                      |
+
+[1]: compute-nodes.md
+[2]: ../general/resources-allocation-policy.md
+[3]: network.md
+[4]: storage.md
+[5]: ../general/shell-and-data-access.md
+[6]: visualization.md
+
+[a]: https://support.it4i.cz/rt

docs.it4i/karolina/introduction.md

+# Introduction
+
+Karolina is the latest and most powerful supercomputer cluster built for IT4Innovations in Q2 of 2021. The Karolina cluster consists of 827 compute nodes, totaling 102,016 compute cores with 248 TB RAM, giving over 15.2 PFLOP/s theoretical peak performance and is ranked in the top 10 of the most powerful supercomputers in Europe.
+
+Nodes are interconnected through a fully non-blocking fat-tree InfiniBand network, and are equipped with AMD Zen 2 and Intel Cascade Lake architecture processors. Seventy nodes are also equipped with NVIDIA A100 accelerators. Read more in [Hardware Overview][1].
+
+The cluster runs with an operating system compatible with the Red Hat [Linux family][a]. We have installed a wide range of software packages targeted at different scientific domains. These packages are accessible via the [modules environment][2].
+
+The user data shared file-system and job data shared file-system are available to users.
+
+The [PBS Professional Open Source Project][b] workload manager provides [computing resources allocations and job execution][3].
+
+Read more on how to [apply for resources][4], [obtain login credentials][5] and [access the cluster][6].
+
+[1]: hardware-overview.md
+[2]: ../environment-and-modules.md
+[3]: ../general/resources-allocation-policy.md
+[4]: ../general/applying-for-resources.md
+[5]: ../general/obtaining-login-credentials/obtaining-login-credentials.md
+[6]: ../general/shell-and-data-access.md
+
+[a]: http://upload.wikimedia.org/wikipedia/commons/1/1b/Linux_Distribution_Timeline.svg
+[b]: https://www.pbspro.org/

docs.it4i/karolina/network.md

+# Network
+
+All of the compute and login nodes of Karolina are interconnected through a [Mellanox][c] [InfiniBand][a] HDR 200 Gbps network and a Gigabit Ethernet network.
+
+The Compute network is configured as a non-blocking Fat Tree which consists of 60 x 40-ports Mellanox Quantum™ HDR switches (40 Leaf HDR switches and 20 Spine HDR switches).
+
+![](img/compute_network_topology.png)
+
+Compute nodes and the service infrastructure is connected by the HDR100 technology that allows one 200Gbps HDR port (aggregation 4x 50Gbps) divided into 2 HDR100 ports with 100Gbps (2x 50Gbps) bandwidth. The cabling between the L1 and L2 layer is realized by HDR cabling, connecting the end devices is realized by so called Y or splitter cable (1x HDR200 - 2x HDR100).
+
+**The compute network has the following parameters**
+
+* 100Gbps
+* Latencies less than 10 microseconds (0.6 μs end-to-end, <90ns switch hop)
+* Adaptive routing support
+* MPI communication support
+* IP protocol support (IPoIB)
+* Support for SCRATCH Data Storage and NVMe over Fabric Data Storage.
+
+## Mellanox Quantum™ QM8790 40-Ports Switch
+
+Mellanox provides the world’s smartest switch, enabling in-network computing through the Co-Design Scalable Hierarchical Aggregation and Reduction Protocol (SHARP)™ technology.
+QM8790 has the highest fabric performance available in the market with up to 16Tb/s of non-blocking bandwidth with sub-130ns port-to-port latency
+
+**Performance**
+
+* 40 x HDR200 200Gb/s ports in a 1U switch
+* 80 x HDR100 100Gb/s ports (using splitter cables)
+* 16Tb/s aggregate switch throughput
+* Sub-130ns switch latency – Optimized design
+
+**Optimized Design**
+
+* 1+1 redundant & hot-swappable power
+* N+1 redundant & hot-swappable fans
+* 80 gold+ and energy star certified power supplies
+
+**Advanced Design**
+
+* Adaptive routing
+* Congestion control
+* Collective offloads (Mellanox SHARP™ technology)
+* VL mapping (VL2VL)
+
+[a]: http://en.wikipedia.org/wiki/InfiniBand
+[c]: http://www.mellanox.com/

docs.it4i/karolina/storage.md

+# Storage
+
+Karolina cluster provides two main shared filesystems, [HOME filesystem][1] and [SCRATCH filesystem][2], and has access to IT4Innovations' central PROJECT storage, as well. All login and compute nodes may access the same data on shared file systems. Compute nodes are also equipped with local (non-shared) scratch, RAM disk, and TMP file systems.
+
+## Archiving
+
+Do not use shared filesystems as a backup for large amount of data or long-term archiving mean. The academic staff and students of research institutions in the Czech Republic can use the CESNET storage service, which is available via SSHFS.
+For more information, see the [CESNET storage][6] section.
+
+### HOME File System
+
+The HOME filesystem is an HA cluster of two active-passive NFS servers. This filesystem contains users' home directories `/home/username`. Accessible capacity is 25 TB, shared among all users. Individual users are restricted by filesystem usage quotas, set to 25 GB per user. Should 25 GB prove insufficient, contact [support][d], the quota may be lifted upon request.
+
+!!! note
+    The HOME filesystem is intended for preparation, evaluation, processing and storage of data generated by active Projects.
+
+The files on HOME filesystem will not be deleted until the end of the [user's lifecycle][4].
+
+The filesystem is backed up, so that it can be restored in case of a catastrophic failure resulting in significant data loss. However, this backup is not intended to restore old versions of user data or to restore (accidentally) deleted files.
+
+| HOME filesystem      |                 |
+| -------------------- | --------------- |
+| Accesspoint          | /home/username  |
+| Capacity             | 25 TB           |
+| Throughput           | 1.2 GB/s        |
+| User space quota     | 25 GB           |
+| User inodes quota    | 500 k           |
+| Protocol             | NFS             |
+
+ Configuration of the storage:
+
+**2x NFS server HPE ProLiant DL325 Gen10 Plus**
+
+* 1x AMD EPYC 7302P (3.0GHz/16-core/155W)
+* 8x 16GB (1x16GB) Dual Rank x8 DDR4-3200 CAS-22-22-22
+* 2x 240GB SATA 6G Read Intensive SFF (2.5in) SC SSD – (HW RAID1)
+* 1x Smart Array E208i-a SR Gen10 (No Cache) 12G SAS Modular LH Controller
+* 1x HPE SN1100Q 16Gb Dual Port Fibre Channel Host Bus Adapter
+* 1x Intel I350-T4 Ethernet 1Gb 4-port BASE-T OCP3 Adapter
+* ILO5
+* 1x InfiniBand HDR100/Ethernet 100Gb 2-port QSFP56 PCIe4 x16 MCX653106A-ECAT Adapter
+* 2x 500W Flex Slot Platinum Hot Plug Low Halogen Power Supply Kit
+* OS: Red Hat Enterprise Linux Server
+
+**1x Storage array HPE MSA 2060 16Gb Fibre Channel SFF Storage**
+
+* 1x Base MSA 2060 SFF Storage Drive Enclosure
+* 22x MSA 1.92TB SAS 12G SFF (2.5in) M2 SSD
+* 1x MSA 16Gb Short Wave Fibre Channel SFP+ 4-pack Transceiver
+* Dual-controller, 4x 16Gb FC host interface
+* LAN connectivity 2x 1Gb/s
+* Redundant, hot-swap power supplies
+
+### SCRATCH File System
+
+The SCRATCH filesystem is realized as a parallel Lustre filesystem. It is accessible via the Infiniband network and is available from all login and computational nodes. Extended ACLs are provided on the Lustre filesystems for sharing data with other users using fine-grained control. For basic information about Lustre, see the [Understanding the Lustre Filesystem][7] subsection of the Barbora's storage documentation.*
+
+The SCRATCH filesystem is mounted in directory /scratch. Users may freely create subdirectories and files on the filesystem. Accessible capacity is 1000 TB, shared among all users. Individual users are restricted by filesystem usage quotas, set to 9.3 TB per user. The purpose of this quota is to prevent runaway programs from filling the entire filesystem and deny service to other users. Should 9.3 TB prove insufficient, contact [support][d], the quota may be lifted upon request.
+
+!!! note
+    The Scratch filesystem is intended for temporary scratch data generated during the calculation as well as for high-performance access to input and output files. All I/O intensive jobs must use the SCRATCH filesystem as their working directory.
+
+    Users are advised to save the necessary data from the SCRATCH filesystem to HOME filesystem after the calculations and clean up the scratch files.
+
+!!! warning
+    Files on the SCRATCH filesystem that are **not accessed for more than 90 days** will be automatically **deleted**.
+
+| SCRATCH filesystem   |           |
+| -------------------- | --------- |
+| Mountpoint           | /scratch  |
+| Capacity             | 1000 TB   |
+| Throughput           | 1000 GB/s |
+| User space quota     | 9,3 TB    |
+| User inodes quota    | 10 M      |
+| Default stripe size  | 1 MB      |
+| Default stripe count | 1         |
+| Protocol             | Lustre    |
+
+Configuration of the storage:
+
+**1x SMU - ClusterStor 2U/24 System Management Unit Storage Controller**
+
+* 5x Cray ClusterStor 1.6TB NVMe x4 Lanes Mixed Use SFF (2.5in) U.2 with Carrier
+* 2x Cray ClusterStor InfiniBand HDR/Ethernet 200Gb 1-port QSFP PCIe4 Adapter (Mellanox ConnectX-6)
+
+**1x MDU - ClusterStor 2U/24 Metadata Unit Storage Controller**
+
+* 24x Cray ClusterStor 1.6TB NVMe x4 Lanes Mixed Use SFF (2.5in) U.2 with Carrier
+* 2x Cray ClusterStor InfiniBand HDR/Ethernet 200Gb 1-port QSFP PCIe4 Adapter (Mellanox ConnectX-6)
+
+**24x SSU-F - ClusterStor 2U24 Scalable Storage Unit Flash Storage Controller**
+
+* 24x Cray ClusterStor 3.2TB NVMe x4 Lanes Mixed Use SFF (2.5in) U.2 with Carrier
+* 4x Cray ClusterStor InfiniBand HDR/Ethernet 200Gb 1-port QSFP PCIe4 Adapter (Mellanox ConnectX-6)
+
+**2x LMN - Aruba 6300M 48-port 1GbE**
+
+* Aruba X371 12VDC 250W 100-240VAC Power-to-Port Power Supply
+
+### PROJECT File System
+
+The PROJECT data storage is a central storage for projects'/users' data on IT4Innovations that is accessible from all clusters.
+For more information, see the [PROJECT storage][9] section.
+
+### Disk Usage and Quota Commands
+
+For more information about Disk usage and user quotas, see the Barbora's [storage section][8].
+
+### Extended ACLs
+
+Extended ACLs provide another security mechanism beside the standard POSIX ACLs, which are defined by three entries (for owner/group/others). Extended ACLs have more than the three basic entries. In addition, they also contain a mask entry and may contain any number of named user and named group entries.
+
+ACLs on a Lustre file system work exactly like ACLs on any Linux file system. They are manipulated with the standard tools in the standard manner.
+
+For more information, see the [Access Control List][10] section of the documentation.
+
+## Local Filesystems
+
+### TMP
+
+Each node is equipped with local /tmp directory of few GB capacity. The /tmp directory should be used to work with small temporary files. Old files in /tmp directory are automatically purged.
+
+## Summary
+
+| Mountpoint | Usage                     | Protocol | Net Capacity   | Throughput | Limitations | Access                  | Services                    |        |
+| ---------- | ------------------------- | -------- | -------------- | ---------- | ----------- | ----------------------- | --------------------------- | ------ |
+| /home      | home directory            | NFS      | 25 TB        | 1.2 GB/s     | Quota 25 GB | Compute and login nodes | backed up                   |        |
+| /scratch   | cluster shared jobs' data | Lustre   | 1000 TB        | 1000 GB/s  | Quota 9.3 TB| Compute and login nodes | files older 90 days removed |        |
+| /tmp       | local temporary files     | local    | ------   | -------   | none        | Compute / login nodes   | auto                        | purged |
+
+[1]: #home-file-system
+[2]: #scratch-file-system
+[4]: ../general/obtaining-login-credentials/obtaining-login-credentials.md
+[5]: #project-file-system
+[6]: ../storage/cesnet-storage.md
+[7]: ../barbora/storage.md#understanding-the-lustre-filesystems
+[8]: ../barbora/storage.md#disk-usage-and-quota-commands
+[9]: ../storage/project-storage.md
+[10]: ../storage/standard-file-acl.md
+
+[a]: http://www.nas.nasa.gov
+[b]: http://www.nas.nasa.gov/hecc/support/kb/Lustre_Basics_224.html#striping
+[c]: http://doc.lustre.org/lustre_manual.xhtml#managingstripingfreespace
+[d]: https://support.it4i.cz/rt
+[e]: http://man7.org/linux/man-pages/man1/nfs4_setfacl.1.html
+[l]: http://man7.org/linux/man-pages/man1/nfs4_getfacl.1.html

docs.it4i/karolina/visualization.md

+# Visualization Servers
+
+Karolina includes two nodes for remote visualization with [VirtualGL 2][3] and TurboVNC 2.
+
+* 64 cores in total
+* 2x AMD EPYC™ 7452 32-core, 2.35 GHz processors per node
+* 256GiB DDR4 RAM, 3200MT/s, ECC of physical memory per node (12x 16 GB)
+* HPE ProLiant DL385 Gen10 Plus servers
+* 2406.4 GFLOP/s per compute node
+* NVIDIA Quadro RTX 6000 card with OpenGL support
+* 2x 100 GB Ethernet and 1x 1 GB Ethernet
+* 1x HDR100 IB port
+* 2x SSD 480 GB in RAID1
+
+![](img/proliantdl385.png)
+
+## NVIDIA® Quadro RTX™ 6000
+
+* GPU Memory: 24 GB GDDR6
+* Memory Interface: 384-bit
+* Memory Bandwidth: Up to 672 GB/s
+* NVIDIA® CUDA® Cores: 4,608
+* NVIDIA® Tensor Cores: 576
+* NVIDIA® RT Cores: 72
+* System Interface: PCI Express 3.0 x16
+* Max Power Consumption: 295 W
+* Thermal Solution: Active
+* Form Factor: 111 mm W x 267 mm L, Dual Slot, Full Height
+* Display Connectors: 4x DP 1.4 + DVI-D DL
+* Graphics APIs: Shader Model 5.1, OpenGL 4.6, DirectX 12.0, Vulkan 1.1,
+* Compute APIs: CUDA, DirectCompute, OpenCL™
+* Floating-Point Performance-Single Precision: 16.3 TFLOP/s, Peak
+* Tensor Performance: 130.5 TFLOP/s
+
+![](img/qrtx6000.png)
+
+## Resource Allocation Policy ??
+
+| queue | active project | project resources | nodes | min ncpus | priority | authorization | walltime |
+|-------|----------------|-------------------|-------|-----------|----------|---------------|----------|
+| qviz Visualization queue | yes | none required | - | - | 150 | no | 1h/8h |
+
+## References
+
+* [Graphical User Interface][1]
+* [VPN Access][2]
+
+[1]: ../general/shell-and-data-access.md#graphical-user-interface
+[2]: ../general/shell-and-data-access.md#vpn-access
+[3]: ../software/viz/vgl.md

mkdocs.yml

@@ -77,6 +77,13 @@ nav:
       - Standard File ACL: storage/standard-file-acl.md
       - NFSv4 File ACL: storage/nfs4-file-acl.md
   - Clusters:
+    - Karolina:          
+      - Introduction: karolina/introduction.md
+      - Hardware Overview: karolina/hardware-overview.md
+      - Compute Nodes: karolina/compute-nodes.md
+      - Storage: karolina/storage.md
+      - Network: karolina/network.md
+      - Visualization Servers: karolina/visualization.md
     - Barbora:          
       - Introduction: barbora/introduction.md
       - Hardware Overview: barbora/hardware-overview.md