Skip to content
Snippets Groups Projects
Commit 84f26260 authored by David Hrbáč's avatar David Hrbáč
Browse files

Merge branch 'brano_ramdisk' into 'master' 

Global Ramdisk

See merge request !186
parents 8745bac2 d8cae268
No related branches found
No related tags found
6 merge requests!368Update prace.md to document the change from qprace to qprod as the default...,!367Update prace.md to document the change from qprace to qprod as the default...,!366Update prace.md to document the change from qprace to qprod as the default...,!323extended-acls-storage-section,!196Master,!186Global Ramdisk
Hide whitespace changes
Inline Side-by-side
docs.it4i/img/global_ramdisk.png 0 → 100644
+ 0
− 0
View file @ 84f26260
docs.it4i/img/global_ramdisk.png

9.72 KiB

docs.it4i/salomon/storage.md
+ 71
− 12
View file @ 84f26260
......@@ -302,34 +302,93 @@ The TEMP workspace is hosted on SCRATCH file system. The SCRATCH is realized as
## RAM Disk
Every computational node is equipped with file system realized in memory, so called RAM disk.
### Local RAM Disk
!!! note
Use RAM disk in case you need really fast access to your data of limited size during your calculation. Be very careful, use of RAM disk file system is at the expense of operational memory.
Every computational node is equipped with file system realized in memory, so called RAM disk.
The local RAM disk is mounted as /ramdisk and is accessible to user at /ramdisk/$PBS_JOBID directory.
The local RAM disk file system is intended for temporary scratch data generated during the calculation as well as for high performance access to input and output files. Size of RAM disk file system is limited. Be very careful, use of RAM disk file system is at the expense of operational memory. It is not recommended to allocate large amount of memory and use large amount of data in RAM disk file system at the same time.
The RAM disk is private to a job and local to node, created when the job starts and deleted at the job end.
!!! note
The local RAM disk directory /ramdisk/$PBS_JOBID will be deleted immediately after the calculation end. Users should take care to save the output data from within the jobscript.
| RAM disk | |
The local RAM disk file system is intended for temporary scratch data generated during the calculation as well as
for high-performance access to input and output files. Size of RAM disk file system is limited.
It is not recommended to allocate large amount of memory and use large amount of data in RAM disk file system at the same time.
!!! warning
Be very careful, use of RAM disk file system is at the expense of operational memory.
| Local RAM disk | |
| ----------- | ------------------------------------------------------------------------------------------------------- |
| Mountpoint | /ramdisk |
| Accesspoint | /ramdisk/$PBS_JOBID |
| Capacity | 120 GB |
| Capacity | 110 GB |
| Throughput | over 1.5 GB/s write, over 5 GB/s read, single thread, over 10 GB/s write, over 50 GB/s read, 16 threads |
| User quota | none |
### Global RAM Disk
The Global RAM disk spans the local RAM disks of all the nodes within a single job.
![Global RAM disk](../img/global_ramdisk.png)
The Global RAM disk deploys
BeeGFS On Demand parallel filesystem, using local RAM disks as a storage backend.
The Global RAM disk is mounted at /mnt/global_ramdisk.
!!! note
The global RAM disk is on-demand. It has to be activated by **global_ramdisk=true** in the qsub command.
```console
$ qsub -q qprod -l select=4,global_ramdisk=true ./jobscript
```
This command would submit 4 node job in qprod queue, once running a 440GB RAM disk shared across the 4 nodes will be created.
The RAM disk will be accessible at /mnt/global_ramdisk, files written to this RAM disk will be visible on all 4 nodes.
The file system is private to a job and shared among the nodes, created when the job starts and deleted at the job end.
!!! note
The Global RAM disk will be deleted immediately after the calculation end. Users should take care to save the output data from within the jobscript.
The files on the Global RAM disk will be equally striped across all the nodes, using 512k stripe size.
Check the Global RAM disk status:
```console
$ beegfs-df -p /mnt/global_ramdisk
$ beegfs-ctl --mount=/mnt/global_ramdisk --getentryinfo /mnt/global_ramdisk
```
Use Global RAM disk in case you need very large RAM disk space. The Global RAM disk allows for high performance sharing of data among compute nodes
within a job.
!!! warning
Be very careful, use of Global RAM disk file system is at the expense of operational memory.
| Global RAM disk | |
| ------------------ | --------------------------------------------------------------------------|
| Mountpoint | /mnt/global_ramdisk |
| Accesspoint | /mnt/global_ramdisk |
| Capacity | N*110 GB |
| Throughput | 3*(N+1) GB/s, 2GB/s single POSIX thread |
| User quota | none |
N = number of compute nodes in the job.
## Summary
| Mountpoint | Usage | Protocol | Net Capacity| Throughput | Limitations | Access | Service |
| ------------- | ------------------------------ | ----------- | ------- | -------- | ------------ | ----------------------- | --------------------------- |
| /home | home directory | NFS, 2-Tier | 0.5 PB | 6 GB/s | Quota 250GB | Compute and login nodes | backed up |
| /scratch/work | large project files | Lustre | 1.69 PB | 30 GB/s | Quota | Compute and login nodes | none |
| /scratch/temp | job temporary data | Lustre | 1.69 PB | 30 GB/s | Quota 100 TB | Compute and login nodes | files older 90 days removed |
| /ramdisk | job temporary data, node local | local | 120GB | 90 GB/s | none | Compute nodes | purged after job ends |
| Mountpoint | Usage | Protocol | Net Capacity| Throughput | Limitations | Access | Service |
| ------------------- | ------------------------------ | ----------- | ------------| -------------- | ------------ | --------------------------- | --------------------------- |
| /home | home directory | NFS, 2-Tier | 0.5 PB | 6 GB/s | Quota 250GB | Compute and login nodes | backed up |
| /scratch/work | large project files | Lustre | 1.69 PB | 30 GB/s | Quota | Compute and login nodes | none |
| /scratch/temp | job temporary data | Lustre | 1.69 PB | 30 GB/s | Quota 100 TB | Compute and login nodes | files older 90 days removed |
| /ramdisk | job temporary data, node local | tmpfs | 110GB | 90 GB/s | none | Compute nodes, node local | purged after job ends |
| /mnt/global_ramdisk | job temporary data | BeeGFS | N*110GB | 3*(N+1) GB/s | none | Compute nodes, job shared | purged after job ends |
N = number of compute nodes in the job.
## CESNET Data Storage
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment