diff --git a/docs.it4i/barbora/storage.md b/docs.it4i/barbora/storage.md index 315bde28965b032672ab6c45c61d27872e16b4f2..d9fbddfb80fc0fe629cc844aa568b40f0352e125 100644 --- a/docs.it4i/barbora/storage.md +++ b/docs.it4i/barbora/storage.md @@ -3,7 +3,7 @@ !!! warning Cluster integration in the progress. The resulting settings may vary. The documentation will be updated. -There are three main shared file systems on Barbora cluster, the [HOME][1], [SCRATCH][2], and [PROJECT][5]. All login and compute nodes may access same data on shared file systems. Compute nodes are also equipped with local (non-shared) scratch, RAM disk, and tmp file systems. +There are three main shared file systems on Barbora cluster: [HOME][1], [SCRATCH][2], and [PROJECT][5]. All login and compute nodes may access same data on shared file systems. Compute nodes are also equipped with local (non-shared) scratch, RAM disk, and tmp file systems. ## Archiving @@ -21,18 +21,18 @@ A user file on the [Lustre filesystem][a] can be divided into multiple chunks (s When a client (a compute node from your job) needs to create or access a file, the client queries the metadata server (MDS) and the metadata target (MDT) for the layout and location of the [file's stripes][b]. Once the file is opened and the client obtains the striping information, the MDS is no longer involved in the file I/O process. The client interacts directly with the object storage servers (OSSes) and OSTs to perform I/O operations such as locking, disk allocation, storage, and retrieval. -If multiple clients try to read and write the same part of a file at the same time, the Lustre distributed lock manager enforces coherency so that all clients see consistent results. +If multiple clients try to read and write the same part of a file at the same time, the Lustre distributed lock manager enforces coherency, so that all clients see consistent results. There is default stripe configuration for Barbora Lustre filesystems. However, users can set the following stripe parameters for their own directories or files to get optimum I/O performance: -1. stripe_size: the size of the chunk in bytes; specify with k, m, or g to use units of KB, MB, or GB, respectively; the size must be an even multiple of 65,536 bytes; default is 1MB for all Barbora Lustre filesystems -1. stripe_count the number of OSTs to stripe across; default is 1 for Barbora Lustre filesystems one can specify -1 to use all OSTs in the filesystem. -1. stripe_offset The index of the OST where the first stripe is to be placed; default is -1 which results in random selection; using a non-default value is NOT recommended. +1. `stripe_size` the size of the chunk in bytes; specify with k, m, or g to use units of KB, MB, or GB, respectively; the size must be an even multiple of 65,536 bytes; default is 1MB for all Barbora Lustre filesystems +1. `stripe_count` the number of OSTs to stripe across; default is 1 for Barbora Lustre filesystems one can specify -1 to use all OSTs in the filesystem. +1. `stripe_offset` the index of the OST where the first stripe is to be placed; default is -1 which results in random selection; using a non-default value is NOT recommended. !!! note Setting stripe size and stripe count correctly for your needs may significantly affect the I/O performance. -Use the lfs getstripe for getting the stripe parameters. Use the lfs setstripe command for setting the stripe parameters to get optimal I/O performance. The correct stripe setting depends on your needs and file access patterns. +Use the `lfs getstripe` command for getting the stripe parameters. Use `lfs setstripe` for setting the stripe parameters to get optimal I/O performance. The correct stripe setting depends on your needs and file access patterns. ```console $ lfs getstripe dir|filename @@ -54,7 +54,7 @@ stripe_count: 10 stripe_size: 1048576 stripe_offset: -1 In this example, we view the current stripe setting of the /scratch/username/ directory. The stripe count is changed to all OSTs and verified. All files written to this directory will be striped over 10 OSTs -Use lfs check OSTs to see the number and status of active OSTs for each filesystem on Barbora. Learn more by reading the man page: +Use `lfs check ostss` to see the number and status of active OSTs for each filesystem on Barbora. Learn more by reading the man page: ```console $ lfs check osts @@ -64,9 +64,9 @@ $ man lfs ### Hints on Lustre Stripping !!! note - Increase the stripe_count for parallel I/O to the same file. + Increase the `stripe_count` for parallel I/O to the same file. -When multiple processes are writing blocks of data to the same file in parallel, the I/O performance for large files will improve when the stripe_count is set to a larger value. The stripe count sets the number of OSTs to which the file will be written. By default, the stripe count is set to 1. While this default setting provides for efficient access of metadata (for example to support the ls -l command), large files should use stripe counts of greater than 1. This will increase the aggregate I/O bandwidth by using multiple OSTs in parallel instead of just one. A rule of thumb is to use a stripe count approximately equal to the number of gigabytes in the file. +When multiple processes are writing blocks of data to the same file in parallel, the I/O performance for large files will improve when the `stripe_count` is set to a larger value. The stripe count sets the number of OSTs to which the file will be written. By default, the stripe count is set to 1. While this default setting provides for efficient access of metadata (for example to support the `ls -l` command), large files should use stripe counts of greater than 1. This will increase the aggregate I/O bandwidth by using multiple OSTs in parallel instead of just one. A rule of thumb is to use a stripe count approximately equal to the number of gigabytes in the file. Another good practice is to make the stripe count be an integral factor of the number of processes performing the write in parallel, so that you achieve load balance among the OSTs. For example, set the stripe count to 16 instead of 15 when you have 64 processes performing the writes. @@ -125,10 +125,10 @@ The filesystem is backed up, so that it can be restored in case of a catastrophi The SCRATCH is realized as Lustre parallel file system and is available from all login and computational nodes. There are 5 OSTs dedicated for the SCRATCH file system. -The SCRATCH filesystem is mounted in directory /scratch. Users may freely create subdirectories and files on the filesystem. Accessible capacity is 282 TB, shared among all users. Individual users are restricted by filesystem usage quotas, set to 9,3 TB per user. The purpose of this quota is to prevent runaway programs from filling the entire filesystem and deny service to other users. Should 9,3 TB prove insufficient, contact [support][d], the quota may be lifted upon request. +The SCRATCH filesystem is mounted in directory /scratch. Users may freely create subdirectories and files on the filesystem. Accessible capacity is 282 TB, shared among all users. Individual users are restricted by filesystem usage quotas, set to 9.3 TB per user. The purpose of this quota is to prevent runaway programs from filling the entire filesystem and deny service to other users. Should 9.3 TB prove insufficient, contact [support][d], the quota may be lifted upon request. !!! note - The Scratch filesystem is intended for temporary scratch data generated during the calculation as well as for high performance access to input and output files. All I/O intensive jobs must use the SCRATCH filesystem as their working directory. + The Scratch filesystem is intended for temporary scratch data generated during the calculation as well as for high-performance access to input and output files. All I/O intensive jobs must use the SCRATCH filesystem as their working directory. Users are advised to save the necessary data from the SCRATCH filesystem to HOME filesystem after the calculations and clean up the scratch files. @@ -186,7 +186,7 @@ Entries limit: 0 ``` In this example, we view current size limits and space occupied on the /home and /scratch filesystem, for a particular user executing the command. -Note that limits are imposed also on number of objects (files, directories, links, etc...) that are allowed to create. +Note that limits are imposed also on number of objects (files, directories, links, etc.) that are allowed to create. To have a better understanding of where the space is exactly used, you can use following command to find out. @@ -290,7 +290,7 @@ Once registered for CESNET Storage, you may [access the storage][j] in number of The SSHFS provides a very convenient way to access the CESNET Storage. The storage will be mounted onto a local directory, exposing the vast CESNET Storage as if it was a local removable hard drive. Files can be than copied in and out in a usual fashion. -First, create the mount point +First, create the mount point: ```console $ mkdir cesnet @@ -331,9 +331,9 @@ $ fusermount -u cesnet ### RSYNC Access !!! info - RSYNC provides delta transfer for best performance, can resume interrupted transfers + RSYNC provides delta transfer for best performance and can resume interrupted transfers. -RSYNC is a fast and extraordinarily versatile file copying tool. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. RSYNC is widely used for backups and mirroring and as an improved copy command for everyday use. +RSYNC is a fast and extraordinarily versatile file copying tool. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. RSYNC is widely used for backups and mirroring and as an improved copy command for everyday use. RSYNC finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated.