Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
CESNET Data Storage
===================
Introduction
------------
Do not use shared filesystems at IT4Innovations as a backup for large
amount of data or long-term archiving purposes.
The IT4Innovations does not provide storage capacity for data archiving.
Academic staff and students of research institutions in the Czech
Republic can use [CESNET Storage
service](https://du.cesnet.cz/).
The CESNET Storage service can be used for research purposes, mainly by
academic staff and students of research institutions in the Czech
Republic.
User of data storage CESNET (DU) association can become organizations or
an individual person who is either in the current employment
relationship (employees) or the current study relationship (students) to
a legal entity (organization) that meets the “Principles for access to
CESNET Large infrastructure (Access Policy)”.
User may only use data storage CESNET for data transfer and storage
which are associated with activities in science, research, development,
the spread of education, culture and prosperity. In detail see
“Acceptable Use Policy CESNET Large Infrastructure (Acceptable Use
Policy, AUP)”.
The service is documented at
<https://du.cesnet.cz/wiki/doku.php/en/start>. For special requirements
please contact directly CESNET Storage Department via e-mail
[du-support(at)cesnet.cz](mailto:du-support@cesnet.cz).
The procedure to obtain the CESNET access is quick and trouble-free.
(source
[https://du.cesnet.cz/](https://du.cesnet.cz/wiki/doku.php/en/start "CESNET Data Storage"))
CESNET storage access
---------------------
### Understanding Cesnet storage
It is very important to understand the Cesnet storage before uploading
data. Please read
<https://du.cesnet.cz/en/navody/home-migrace-plzen/start> first.
Once registered for CESNET Storage, you may [access the
storage](https://du.cesnet.cz/en/navody/faq/start) in
number of ways. We recommend the SSHFS and RSYNC methods.
### SSHFS Access
SSHFS: The storage will be mounted like a local hard drive
The SSHFS provides a very convenient way to access the CESNET Storage.
The storage will be mounted onto a local directory, exposing the vast
CESNET Storage as if it was a local removable harddrive. Files can be
than copied in and out in a usual fashion.
First, create the mountpoint
$ mkdir cesnet
Mount the storage. Note that you can choose among the ssh.du1.cesnet.cz
(Plzen), ssh.du2.cesnet.cz (Jihlava), ssh.du3.cesnet.cz (Brno)
Mount tier1_home **(only 5120M !)**:
$ sshfs username@ssh.du1.cesnet.cz:. cesnet/
For easy future access from Anselm, install your public key
$ cp .ssh/id_rsa.pub cesnet/.ssh/authorized_keys
Mount tier1_cache_tape for the Storage VO:
$ sshfs username@ssh.du1.cesnet.cz:/cache_tape/VO_storage/home/username cesnet/
View the archive, copy the files and directories in and out
$ ls cesnet/
$ cp -a mydir cesnet/.
$ cp cesnet/myfile .
Once done, please remember to unmount the storage
$ fusermount -u cesnet
### Rsync access
Rsync provides delta transfer for best performance, can resume
interrupted transfers
Rsync is a fast and extraordinarily versatile file copying tool. It is
famous for its delta-transfer algorithm, which reduces the amount of
data sent over the network by sending only the differences between the
source files and the existing files in the destination. Rsync is widely
used for backups and mirroring and as an improved copy command for
everyday use.
Rsync finds files that need to be transferred using a "quick check"
algorithm (by default) that looks for files that have changed in size or
in last-modified time. Any changes in the other preserved attributes
(as requested by options) are made on the destination file directly when
the quick check indicates that the file's data does not need to be
updated.
More about Rsync at
<https://du.cesnet.cz/en/navody/rsync/start#pro_bezne_uzivatele>
Transfer large files to/from Cesnet storage, assuming membership in the
Storage VO
$ rsync --progress datafile username@ssh.du1.cesnet.cz:VO_storage-cache_tape/.
$ rsync --progress username@ssh.du1.cesnet.cz:VO_storage-cache_tape/datafile .
Transfer large directories to/from Cesnet storage, assuming membership
in the Storage VO
$ rsync --progress -av datafolder username@ssh.du1.cesnet.cz:VO_storage-cache_tape/.
$ rsync --progress -av username@ssh.du1.cesnet.cz:VO_storage-cache_tape/datafolder .
Transfer rates of about 28MB/s can be expected.