storage.md 21.5 KB
Newer Older
David Hrbáč's avatar
David Hrbáč committed
1
# Storage
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
2

David Hrbáč's avatar
David Hrbáč committed
3
There are two main shared file systems on Anselm cluster, the [HOME][1] and [SCRATCH][2]. All login and compute nodes may access same data on shared file systems. Compute nodes are also equipped with local (non-shared) scratch, RAM disk and tmp file systems.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
4

David Hrbáč's avatar
David Hrbáč committed
5
## Archiving
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
6

Lukáš Krupčík's avatar
Dgx  
Lukáš Krupčík committed
7
Don't use shared filesystems as a backup for large amount of data or long-term archiving mean. The academic staff and students of research institutions in the Czech Republic can use [CESNET storage service][3], which is available via SSHFS.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
8

David Hrbáč's avatar
David Hrbáč committed
9
## Shared Filesystems
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
10

David Hrbáč's avatar
David Hrbáč committed
11
Anselm computer provides two main shared filesystems, the [HOME filesystem][1] and the [SCRATCH filesystem][2]. Both HOME and SCRATCH filesystems are realized as a parallel Lustre filesystem. Both shared file systems are accessible via the Infiniband network. Extended ACLs are provided on both Lustre filesystems for the purpose of sharing data with other users using fine-grained control.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
12

David Hrbáč's avatar
David Hrbáč committed
13
### Understanding the Lustre Filesystems
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
14

David Hrbáč's avatar
David Hrbáč committed
15
A user file on the [Lustre filesystem][a] can be divided into multiple chunks (stripes) and stored across a subset of the object storage targets (OSTs) (disks). The stripes are distributed among the OSTs in a round-robin fashion to ensure load balancing.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
16

David Hrbáč's avatar
David Hrbáč committed
17
When a client (a compute node from your job) needs to create or access a file, the client queries the metadata server (MDS) and the metadata target (MDT) for the layout and location of the [file's stripes][b]. Once the file is opened and the client obtains the striping information, the MDS is no longer involved in the file I/O process. The client interacts directly with the object storage servers (OSSes) and OSTs to perform I/O operations such as locking, disk allocation, storage, and retrieval.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
18 19 20 21 22

If multiple clients try to read and write the same part of a file at the same time, the Lustre distributed lock manager enforces coherency so that all clients see consistent results.

There is default stripe configuration for Anselm Lustre filesystems. However, users can set the following stripe parameters for their own directories or files to get optimum I/O performance:

Lukáš Krupčík's avatar
Lukáš Krupčík committed
23 24 25
1. stripe_size: the size of the chunk in bytes; specify with k, m, or g to use units of KB, MB, or GB, respectively; the size must be an even multiple of 65,536 bytes; default is 1MB for all Anselm Lustre filesystems
1. stripe_count the number of OSTs to stripe across; default is 1 for Anselm Lustre filesystems one can specify -1 to use all OSTs in the filesystem.
1. stripe_offset The index of the OST where the first stripe is to be placed; default is -1 which results in random selection; using a non-default value is NOT recommended.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
26

David Hrbáč's avatar
David Hrbáč committed
27
!!! note
Lukáš Krupčík's avatar
Lukáš Krupčík committed
28
    Setting stripe size and stripe count correctly for your needs may significantly impact the I/O performance you experience.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
29

Lukáš Krupčík's avatar
Lukáš Krupčík committed
30
Use the lfs getstripe for getting the stripe parameters. Use the lfs setstripe command for setting the stripe parameters to get optimal I/O performance The correct stripe setting depends on your needs and file access patterns.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
31

Lukáš Krupčík's avatar
Lukáš Krupčík committed
32
```console
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
33
$ lfs getstripe dir|filename
Lukáš Krupčík's avatar
Lukáš Krupčík committed
34
$ lfs setstripe -s stripe_size -c stripe_count -o stripe_offset dir|filename
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
35 36 37 38
```

Example:

Lukáš Krupčík's avatar
Lukáš Krupčík committed
39
```console
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
40 41 42 43 44 45 46 47 48 49 50 51 52 53
$ lfs getstripe /scratch/username/
/scratch/username/
stripe_count:   1 stripe_size:    1048576 stripe_offset:  -1

$ lfs setstripe -c -1 /scratch/username/
$ lfs getstripe /scratch/username/
/scratch/username/
stripe_count:  10 stripe_size:    1048576 stripe_offset:  -1
```

In this example, we view current stripe setting of the /scratch/username/ directory. The stripe count is changed to all OSTs, and verified. All files written to this directory will be striped over 10 OSTs

Use lfs check OSTs to see the number and status of active OSTs for each filesystem on Anselm. Learn more by reading the man page

Lukáš Krupčík's avatar
Lukáš Krupčík committed
54
```console
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
55 56 57 58 59 60
$ lfs check osts
$ man lfs
```

### Hints on Lustre Stripping

David Hrbáč's avatar
David Hrbáč committed
61
!!! note
Lukáš Krupčík's avatar
Lukáš Krupčík committed
62
    Increase the stripe_count for parallel I/O to the same file.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
63 64 65 66 67

When multiple processes are writing blocks of data to the same file in parallel, the I/O performance for large files will improve when the stripe_count is set to a larger value. The stripe count sets the number of OSTs the file will be written to. By default, the stripe count is set to 1. While this default setting provides for efficient access of metadata (for example to support the ls -l command), large files should use stripe counts of greater than 1. This will increase the aggregate I/O bandwidth by using multiple OSTs in parallel instead of just one. A rule of thumb is to use a stripe count approximately equal to the number of gigabytes in the file.

Another good practice is to make the stripe count be an integral factor of the number of processes performing the write in parallel, so that you achieve load balance among the OSTs. For example, set the stripe count to 16 instead of 15 when you have 64 processes performing the writes.

David Hrbáč's avatar
David Hrbáč committed
68
!!! note
Lukáš Krupčík's avatar
Lukáš Krupčík committed
69
    Using a large stripe size can improve performance when accessing very large files
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
70 71 72

Large stripe size allows each client to have exclusive access to its own part of a file. However, it can be counterproductive in some cases if it does not match your I/O pattern. The choice of stripe size has no effect on a single-stripe file.

David Hrbáč's avatar
David Hrbáč committed
73
Read more [here][c].
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
74 75 76

### Lustre on Anselm

Lukáš Krupčík's avatar
->  
Lukáš Krupčík committed
77
The architecture of Lustre on Anselm is composed of two metadata servers (MDS) and four data/object storage servers (OSS). Two object storage servers are used for file system HOME and another two object storage servers are used for file system SCRATCH.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
78 79 80

 Configuration of the storages

Lukáš Krupčík's avatar
* -> *  
Lukáš Krupčík committed
81
* HOME Lustre object storage
Lukáš Krupčík's avatar
Lukáš Krupčík committed
82 83 84 85 86
  * One disk array NetApp E5400
  * 22 OSTs
  * 227 2TB NL-SAS 7.2krpm disks
  * 22 groups of 10 disks in RAID6 (8+2)
  * 7 hot-spare disks
Lukáš Krupčík's avatar
* -> *  
Lukáš Krupčík committed
87
* SCRATCH Lustre object storage
Lukáš Krupčík's avatar
Lukáš Krupčík committed
88 89 90 91 92
  * Two disk arrays NetApp E5400
  * 10 OSTs
  * 106 2TB NL-SAS 7.2krpm disks
  * 10 groups of 10 disks in RAID6 (8+2)
  * 6 hot-spare disks
Lukáš Krupčík's avatar
* -> *  
Lukáš Krupčík committed
93
* Lustre metadata storage
Lukáš Krupčík's avatar
Lukáš Krupčík committed
94 95 96 97
  * One disk array NetApp E2600
  * 12 300GB SAS 15krpm disks
  * 2 groups of 5 disks in RAID5
  * 2 hot-spare disks
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
98

David Hrbáč's avatar
David Hrbáč committed
99
### HOME File System
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
100

David Hrbáč's avatar
David Hrbáč committed
101
The HOME filesystem is mounted in directory /home. Users home directories /home/username reside on this filesystem. Accessible capacity is 320TB, shared among all users. Individual users are restricted by filesystem usage quotas, set to 250GB per user. If 250GB should prove as insufficient for particular user, contact [support][d], the quota may be lifted upon request.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
102

David Hrbáč's avatar
David Hrbáč committed
103
!!! note
Lukáš Krupčík's avatar
Lukáš Krupčík committed
104
    The HOME filesystem is intended for preparation, evaluation, processing and storage of data generated by active Projects.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
105 106 107

The HOME filesystem should not be used to archive data of past Projects or other unrelated data.

David Hrbáč's avatar
David Hrbáč committed
108
The files on HOME filesystem will not be deleted until end of the [users lifecycle][4].
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
109

David Hrbáč's avatar
David Hrbáč committed
110
The filesystem is backed up, such that it can be restored in case of catasthropic failure resulting in significant data loss. This backup however is not intended to restore old versions of user data or to restore (accidentaly) deleted files.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
111 112 113 114

The HOME filesystem is realized as Lustre parallel filesystem and is available on all login and computational nodes.
Default stripe size is 1MB, stripe count is 1. There are 22 OSTs dedicated for the HOME filesystem.

David Hrbáč's avatar
David Hrbáč committed
115
!!! note
Lukáš Krupčík's avatar
Lukáš Krupčík committed
116
    Setting stripe size and stripe count correctly for your needs may significantly impact the I/O performance you experience.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
117

David Hrbáč's avatar
David Hrbáč committed
118
| HOME filesystem      |        |
Lukáš Krupčík's avatar
Lukáš Krupčík committed
119
| -------------------- | ------ |
David Hrbáč's avatar
David Hrbáč committed
120 121 122
| Mountpoint           | /home  |
| Capacity             | 320 TB |
| Throughput           | 2 GB/s |
Marek Chrastina's avatar
Marek Chrastina committed
123 124
| User space quota     | 250 GB |
| User inodes quota    | 500 k  |
David Hrbáč's avatar
David Hrbáč committed
125 126 127
| Default stripe size  | 1 MB   |
| Default stripe count | 1      |
| Number of OSTs       | 22     |
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
128

David Hrbáč's avatar
David Hrbáč committed
129
### SCRATCH File System
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
130

David Hrbáč's avatar
David Hrbáč committed
131
The SCRATCH filesystem is mounted in directory /scratch. Users may freely create subdirectories and files on the filesystem. Accessible capacity is 146TB, shared among all users. Individual users are restricted by filesystem usage quotas, set to 100TB per user. The purpose of this quota is to prevent runaway programs from filling the entire filesystem and deny service to other users. If 100TB should prove as insufficient for particular user, contact [support][d], the quota may be lifted upon request.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
132

David Hrbáč's avatar
David Hrbáč committed
133
!!! note
Lukáš Krupčík's avatar
->  
Lukáš Krupčík committed
134
    The Scratch filesystem is intended for temporary scratch data generated during the calculation as well as for high performance access to input and output files. All I/O intensive jobs must use the SCRATCH filesystem as their working directory.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
135

Lukáš Krupčík's avatar
Lukáš Krupčík committed
136
    Users are advised to save the necessary data from the SCRATCH filesystem to HOME filesystem after the calculations and clean up the scratch files.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
137

138
!!! warning
David Hrbáč's avatar
David Hrbáč committed
139
    Files on the SCRATCH filesystem that are **not accessed for more than 90 days** will be automatically **deleted**.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
140 141 142

The SCRATCH filesystem is realized as Lustre parallel filesystem and is available from all login and computational nodes. Default stripe size is 1MB, stripe count is 1. There are 10 OSTs dedicated for the SCRATCH filesystem.

David Hrbáč's avatar
David Hrbáč committed
143
!!! note
Lukáš Krupčík's avatar
Lukáš Krupčík committed
144
    Setting stripe size and stripe count correctly for your needs may significantly impact the I/O performance you experience.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
145

David Hrbáč's avatar
David Hrbáč committed
146
| SCRATCH filesystem   |          |
Lukáš Krupčík's avatar
Lukáš Krupčík committed
147
| -------------------- | -------- |
David Hrbáč's avatar
David Hrbáč committed
148
| Mountpoint           | /scratch |
Marek Chrastina's avatar
Marek Chrastina committed
149 150
| Capacity             | 146 TB   |
| Throughput           | 6 GB/s   |
Marek Chrastina's avatar
Marek Chrastina committed
151
| User space quota     | 100 TB   |
Marek Chrastina's avatar
Marek Chrastina committed
152 153
| User inodes quota    | 10 M     |
| Default stripe size  | 1 MB     |
David Hrbáč's avatar
David Hrbáč committed
154 155
| Default stripe count | 1        |
| Number of OSTs       | 10       |
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
156

David Hrbáč's avatar
David Hrbáč committed
157
### Disk Usage and Quota Commands
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
158

159
Disk usage and user quotas can be checked and reviewed using following command:
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
160

Lukáš Krupčík's avatar
Lukáš Krupčík committed
161
```console
162
$ it4i-disk-usage
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
163 164
```

165
Example:
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
166

Lukáš Krupčík's avatar
Lukáš Krupčík committed
167
```console
168 169 170 171 172 173
$ it4i-disk-usage -h
# Using human-readable format
# Using power of 1024 for space
# Using power of 1000 for entries

Filesystem:    /home
Marek Chrastina's avatar
Marek Chrastina committed
174 175 176 177
Space used:    112GiB
Space limit:   238GiB
Entries:       15K
Entries limit: 500K
178 179 180

Filesystem:    /scratch
Space used:    0
Marek Chrastina's avatar
Marek Chrastina committed
181
Space limit:   93TiB
182
Entries:       0
Marek Chrastina's avatar
Marek Chrastina committed
183
Entries limit: 10M
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
184 185
```

David Hrbáč's avatar
David Hrbáč committed
186
In this example, we view current size limits and space occupied on the /home and /scratch filesystem, for a particular user executing the command.
187
Note that limits are imposed also on number of objects (files, directories, links, etc...) that are allowed to create.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
188 189 190

To have a better understanding of where the space is exactly used, you can use following command to find out.

Lukáš Krupčík's avatar
Lukáš Krupčík committed
191
```console
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
192 193 194 195 196
$ du -hs dir
```

Example for your HOME directory:

Lukáš Krupčík's avatar
Lukáš Krupčík committed
197
```console
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
198 199 200 201 202 203 204 205 206
$ cd /home
$ du -hs * .[a-zA-z0-9]* | grep -E "[0-9]*G|[0-9]*M" | sort -hr
258M     cuda-samples
15M      .cache
13M      .mozilla
5,5M     .eclipse
2,7M     .idb_13.0_linux_intel64_app
```

David Hrbáč's avatar
David Hrbáč committed
207
This will list all directories which are having MegaBytes or GigaBytes of consumed space in your actual (in this example HOME) directory. List is sorted in descending order from largest to smallest files/directories.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
208 209 210

To have a better understanding of previous commands, you can read manpages.

Lukáš Krupčík's avatar
Lukáš Krupčík committed
211
```console
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
212 213 214
$ man lfs
```

Lukáš Krupčík's avatar
Lukáš Krupčík committed
215
```console
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
216 217 218 219 220
$ man du
```

### Extended ACLs

David Hrbáč's avatar
David Hrbáč committed
221
Extended ACLs provide another security mechanism beside the standard POSIX ACLs which are defined by three entries (for owner/group/others). Extended ACLs have more than the three basic entries. In addition, they also contain a mask entry and may contain any number of named user and named group entries.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
222 223 224

ACLs on a Lustre file system work exactly like ACLs on any Linux file system. They are manipulated with the standard tools in the standard manner. Below, we create a directory and allow a specific user access.

Lukáš Krupčík's avatar
Lukáš Krupčík committed
225
```console
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
226 227 228
[vop999@login1.anselm ~]$ umask 027
[vop999@login1.anselm ~]$ mkdir test
[vop999@login1.anselm ~]$ ls -ld test
Lukáš Krupčík's avatar
->  
Lukáš Krupčík committed
229
drwxr-x--- 2 vop999 vop999 4096 Nov 5 14:17 test
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
230 231 232 233 234 235 236 237 238 239
[vop999@login1.anselm ~]$ getfacl test
# file: test
# owner: vop999
# group: vop999
user::rwx
group::r-x
other::---

[vop999@login1.anselm ~]$ setfacl -m user:johnsm:rwx test
[vop999@login1.anselm ~]$ ls -ld test
Lukáš Krupčík's avatar
->  
Lukáš Krupčík committed
240
drwxrwx---+ 2 vop999 vop999 4096 Nov 5 14:17 test
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
241 242 243 244 245 246 247 248 249 250 251
[vop999@login1.anselm ~]$ getfacl test
# file: test
# owner: vop999
# group: vop999
user::rwx
user:johnsm:rwx
group::r-x
mask::rwx
other::---
```

David Hrbáč's avatar
David Hrbáč committed
252
Default ACL mechanism can be used to replace setuid/setgid permissions on directories. Setting a default ACL on a directory (-d flag to setfacl) will cause the ACL permissions to be inherited by any newly created file or subdirectory within the directory. Refer to this page for more information on Linux ACL at [RedHat guide][e].
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
253

David Hrbáč's avatar
David Hrbáč committed
254
## Local Filesystems
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
255 256 257

### Local Scratch

David Hrbáč's avatar
David Hrbáč committed
258
!!! note
Lukáš Krupčík's avatar
Lukáš Krupčík committed
259
    Every computational node is equipped with 330GB local scratch disk.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
260 261 262 263 264

Use local scratch in case you need to access large amount of small files during your calculation.

The local scratch disk is mounted as /lscratch and is accessible to user at /lscratch/$PBS_JOBID directory.

Lukáš Krupčík's avatar
->  
Lukáš Krupčík committed
265
The local scratch filesystem is intended for temporary scratch data generated during the calculation as well as for high performance access to input and output files. All I/O intensive jobs that access large number of small files within the calculation must use the local scratch filesystem as their working directory. This is required for performance reasons, as frequent access to number of small files may overload the metadata servers (MDS) of the Lustre filesystem.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
266

David Hrbáč's avatar
David Hrbáč committed
267
!!! note
Lukáš Krupčík's avatar
Lukáš Krupčík committed
268
    The local scratch directory /lscratch/$PBS_JOBID will be deleted immediately after the calculation end. Users should take care to save the output data from within the jobscript.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
269

David Hrbáč's avatar
David Hrbáč committed
270
| local SCRATCH filesystem |                      |
Lukáš Krupčík's avatar
Lukáš Krupčík committed
271
| ------------------------ | -------------------- |
David Hrbáč's avatar
David Hrbáč committed
272 273
| Mountpoint               | /lscratch            |
| Accesspoint              | /lscratch/$PBS_JOBID |
Marek Chrastina's avatar
Marek Chrastina committed
274 275
| Capacity                 | 330 GB               |
| Throughput               | 100 MB/s             |
David Hrbáč's avatar
David Hrbáč committed
276
| User quota               | none                 |
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
277

David Hrbáč's avatar
David Hrbáč committed
278
### RAM Disk
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
279 280 281

Every computational node is equipped with filesystem realized in memory, so called RAM disk.

David Hrbáč's avatar
David Hrbáč committed
282
!!! note
Lukáš Krupčík's avatar
Lukáš Krupčík committed
283
    Use RAM disk in case you need really fast access to your data of limited size during your calculation. Be very careful, use of RAM disk filesystem is at the expense of operational memory.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
284 285 286

The local RAM disk is mounted as /ramdisk and is accessible to user at /ramdisk/$PBS_JOBID directory.

David Hrbáč's avatar
David Hrbáč committed
287
The local RAM disk filesystem is intended for temporary scratch data generated during the calculation as well as for high performance access to input and output files. Size of RAM disk filesystem is limited. Be very careful, use of RAM disk filesystem is at the expense of operational memory.  It is not recommended to allocate large amount of memory and use large amount of data in RAM disk filesystem at the same time.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
288

David Hrbáč's avatar
David Hrbáč committed
289
!!! note
Lukáš Krupčík's avatar
Lukáš Krupčík committed
290
    The local RAM disk directory /ramdisk/$PBS_JOBID will be deleted immediately after the calculation end. Users should take care to save the output data from within the jobscript.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
291

Marek Chrastina's avatar
Marek Chrastina committed
292 293 294 295 296 297 298
| RAM disk    |                                                                                                          |
| ----------- | -------------------------------------------------------------------------------------------------------- |
| Mountpoint  | /ramdisk                                                                                                 |
| Accesspoint | /ramdisk/$PBS_JOBID                                                                                      |
| Capacity    | 60 GB at compute nodes without accelerator, 90 GB at compute nodes with accelerator, 500 GB at fat nodes |
| Throughput  | over 1.5 GB/s write, over 5 GB/s read, single thread, over 10 GB/s write, over 50 GB/s read, 16 threads  |
| User quota  | none                                                                                                     |
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
299

David Hrbáč's avatar
David Hrbáč committed
300
### Tmp
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
301 302 303

Each node is equipped with local /tmp directory of few GB capacity. The /tmp directory should be used to work with small temporary files. Old files in /tmp directory are automatically purged.

David Hrbáč's avatar
David Hrbáč committed
304
## Summary
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
305

Marek Chrastina's avatar
Marek Chrastina committed
306 307 308 309 310 311 312
| Mountpoint | Usage                     | Protocol | Net Capacity   | Throughput | Space/Inodes quota       | Access                  | Services                    |        |
| ---------- | ------------------------- | -------- | -------------- | ---------- | ------------------------ | ----------------------- | --------------------------- | ------ |
| /home      | home directory            | Lustre   | 320 TiB        | 2 GB/s     | 250 GB / 500 k | Compute and login nodes | backed up                   |        |
| /scratch   | cluster shared jobs' data | Lustre   | 146 TiB        | 6 GB/s     | 100 TB / 10 M  | Compute and login nodes | files older 90 days removed |        |
| /lscratch  | node local jobs' data     | local    | 330 GB         | 100 MB/s   | none / none              | Compute nodes           | purged after job ends       |        |
| /ramdisk   | node local jobs' data     | local    | 60, 90, 500 GB | 5-50 GB/s  | none / none              | Compute nodes           | purged after job ends       |        |
| /tmp       | local temporary files     | local    | 9.5 GB         | 100 MB/s   | none / none              | Compute and login nodes | auto                        | purged |
David Hrbáč's avatar
David Hrbáč committed
313 314

## CESNET Data Storage
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
315 316 317

Do not use shared filesystems at IT4Innovations as a backup for large amount of data or long-term archiving purposes.

David Hrbáč's avatar
David Hrbáč committed
318
!!! note
David Hrbáč's avatar
David Hrbáč committed
319
    The IT4Innovations does not provide storage capacity for data archiving. Academic staff and students of research institutions in the Czech Republic can use [CESNET Storage service][f].
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
320 321 322 323 324 325 326

The CESNET Storage service can be used for research purposes, mainly by academic staff and students of research institutions in the Czech Republic.

User of data storage CESNET (DU) association can become organizations or an individual person who is either in the current employment relationship (employees) or the current study relationship (students) to a legal entity (organization) that meets the “Principles for access to CESNET Large infrastructure (Access Policy)”.

User may only use data storage CESNET for data transfer and storage which are associated with activities in science, research, development, the spread of education, culture and prosperity. In detail see “Acceptable Use Policy CESNET Large Infrastructure (Acceptable Use Policy, AUP)”.

David Hrbáč's avatar
David Hrbáč committed
327
The service is documented [here][g]. For special requirements contact directly CESNET Storage Department via e-mail [du-support(at)cesnet.cz][h].
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
328 329 330

The procedure to obtain the CESNET access is quick and trouble-free.

David Hrbáč's avatar
David Hrbáč committed
331
## CESNET Storage Access
David Hrbáč's avatar
David Hrbáč committed
332

David Hrbáč's avatar
David Hrbáč committed
333
### Understanding CESNET Storage
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
334

David Hrbáč's avatar
David Hrbáč committed
335
!!! note
Lukáš Krupčík's avatar
Dgx  
Lukáš Krupčík committed
336
    It is very important to understand the CESNET storage before uploading data. [Read][i] first.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
337

David Hrbáč's avatar
David Hrbáč committed
338
Once registered for CESNET Storage, you may [access the storage][j] in number of ways. We recommend the SSHFS and RSYNC methods.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
339 340 341

### SSHFS Access

David Hrbáč's avatar
David Hrbáč committed
342
!!! note
Lukáš Krupčík's avatar
Lukáš Krupčík committed
343
    SSHFS: The storage will be mounted like a local hard drive
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
344

Lukáš Krupčík's avatar
->  
Lukáš Krupčík committed
345
The SSHFS provides a very convenient way to access the CESNET Storage. The storage will be mounted onto a local directory, exposing the vast CESNET Storage as if it was a local removable hard drive. Files can be than copied in and out in a usual fashion.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
346

David Hrbáč's avatar
David Hrbáč committed
347
First, create the mount point
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
348

Lukáš Krupčík's avatar
Lukáš Krupčík committed
349 350
```console
$ mkdir cesnet
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
351 352
```

353
Mount the storage. Note that you can choose among the ssh.du1.cesnet.cz (Plzen), ssh.du2.cesnet.cz (Jihlava), ssh.du3.cesnet.cz (Brno) Mount tier1_home **(only 5120 MB !)**:
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
354

Lukáš Krupčík's avatar
Lukáš Krupčík committed
355 356
```console
$ sshfs username@ssh.du1.cesnet.cz:. cesnet/
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
357 358 359 360
```

For easy future access from Anselm, install your public key

Lukáš Krupčík's avatar
Lukáš Krupčík committed
361 362
```console
$ cp .ssh/id_rsa.pub cesnet/.ssh/authorized_keys
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
363 364 365 366
```

Mount tier1_cache_tape for the Storage VO:

Lukáš Krupčík's avatar
Lukáš Krupčík committed
367 368
```console
$ sshfs username@ssh.du1.cesnet.cz:/cache_tape/VO_storage/home/username cesnet/
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
369 370 371 372
```

View the archive, copy the files and directories in and out

Lukáš Krupčík's avatar
Lukáš Krupčík committed
373 374 375 376
```console
$ ls cesnet/
$ cp -a mydir cesnet/.
$ cp cesnet/myfile .
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
377 378
```

Lukáš Krupčík's avatar
Cisteni  
Lukáš Krupčík committed
379
Once done, remember to unmount the storage
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
380

Lukáš Krupčík's avatar
Lukáš Krupčík committed
381 382
```console
$ fusermount -u cesnet
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
383 384
```

Lukáš Krupčík's avatar
fix  
Lukáš Krupčík committed
385
### RSYNC Access
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
386

Lukáš Krupčík's avatar
Cisteni  
Lukáš Krupčík committed
387 388
!!! info
    RSYNC provides delta transfer for best performance, can resume interrupted transfers
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
389

390
RSYNC is a fast and extraordinarily versatile file copying tool. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination.  RSYNC is widely used for backups and mirroring and as an improved copy command for everyday use.
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
391

392 393
RSYNC finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time.  Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated.

David Hrbáč's avatar
David Hrbáč committed
394
[More about RSYNC][k].
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
395

David Hrbáč's avatar
David Hrbáč committed
396
Transfer large files to/from CESNET storage, assuming membership in the Storage VO
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
397

Lukáš Krupčík's avatar
Lukáš Krupčík committed
398 399 400
```console
$ rsync --progress datafile username@ssh.du1.cesnet.cz:VO_storage-cache_tape/.
$ rsync --progress username@ssh.du1.cesnet.cz:VO_storage-cache_tape/datafile .
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
401 402
```

David Hrbáč's avatar
David Hrbáč committed
403
Transfer large directories to/from CESNET storage, assuming membership in the Storage VO
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
404

Lukáš Krupčík's avatar
Lukáš Krupčík committed
405 406 407
```console
$ rsync --progress -av datafolder username@ssh.du1.cesnet.cz:VO_storage-cache_tape/.
$ rsync --progress -av username@ssh.du1.cesnet.cz:VO_storage-cache_tape/datafolder .
Pavel Jirásek's avatar
Merged  
Pavel Jirásek committed
408 409
```

David Hrbáč's avatar
David Hrbáč committed
410
Transfer rates of about 28 MB/s can be expected.
David Hrbáč's avatar
David Hrbáč committed
411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427

[1]: #home
[2]: #scratch
[3]: #cesnet-data-storage
[4]: ../general/obtaining-login-credentials/obtaining-login-credentials.md

[a]: http://www.nas.nasa.gov
[b]: http://www.nas.nasa.gov/hecc/support/kb/Lustre_Basics_224.html#striping
[c]: http://doc.lustre.org/lustre_manual.xhtml#managingstripingfreespace
[d]: https://support.it4i.cz/rt
[e]: https://access.redhat.com/documentation/en-US/Red_Hat_Storage/2.0/html/Administration_Guide/ch09s05.html
[f]: https://du.cesnet.cz/
[g]: https://du.cesnet.cz/en/start
[h]: mailto:du-support@cesnet.cz
[i]: https://du.cesnet.cz/en/navody/home-migrace-plzen/start
[j]: https://du.cesnet.cz/en/navody/faq/start
[k]: https://du.cesnet.cz/en/navody/rsync/start#pro_bezne_uzivatele