Skip to content
Snippets Groups Projects
dice.md 10 KiB
Newer Older
  • Learn to ignore specific revisions
  • Jan Siwiec's avatar
    Jan Siwiec committed
    # What Is DICE Project?
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    
    DICE (Developing Innovation and Creativity in Education) is an international project co-funded by the Erasmus+ Programme of the European Union.
    
    Its main goals are related to fostering innovation and creativity in education.
    
    This project attempts to be a leading initiative that supports educational community to make progress in the field of innovation and creativity through effective, authentic, practical, sustainable actions.
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    This project has been funded with support from the European Commission.
    For more information, see [http://www.diceproject.eu/][b].
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    **IT4Innovations participates in DICE. DICE uses the iRods software**
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    The integrated Rule-Oriented Data System (iRODS) is an open source data management software used by research organizations and government agencies worldwide. iRODS is released as a production-level distribution aimed at deployment in mission critical environments. It virtualizes data storage resources, so users can take control of their data, regardless of where and on what device the data is stored. As data volumes grow and data services become more complex, iRODS is serving an increasingly important role in data management, see [https://irods.org/][c].
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    ## How to Put Your Data to Our Server
    
    
    **Prerequisities:**
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    
    We have to be sure about your identity, this will be done throught this steps:
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    
    - at the first, fill this request for [EUDAT][h]
    ![](img/eudat_request.jpg)
    - sign to [MyAccessID][g] , after that, please create account at [B2ACCESS][d] , pages requests valid personal certificate (e.g from GEANT)
    ![](img/B2ACCESS.jpg)
    ![](img/crypto.jpg)
    ![](img/eudat.jpg)
    - after successfull login to B2Access, please sign in to our [AAI][f] throught your B2Access account. You can set new password. this new password will work for cyberduck and icommands.
    - finally, contact our support@it4i.cz by mail, we will create your account at our iRODS server.
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    ![](img/aai2.jpg)
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    ![](img/aai3-passwd.jpg)
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ### ***IT4I User (You Have IT4I Account With Access to Cluster Already)***
    
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - iRODS clients are provided as a module on our ***Karolina*** cluster ( Barbora is in progress )
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - module irodsfs contains config file for irodsfs nad for icommands too
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - PAM passwords is password, you set in aai.it4i.cz
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    
    ***How to mount your collection***
    
    ```console
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ssh some_user@karolina.it4i.cz
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ml irodsfs
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ```
    
    
    - config files for irodsfs and icommands will be loaded
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - now you can choose between Fuse client or iCommands
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ***Fuse***
    
    ```console
    
    ssh some_user@karolina.it4i.cz
    [some_use@login4.karolina ~]$ ml irodsfs
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    
    
    irodsfs configuration file has been created at /home/dvo0012/.irods/config.yml
    iRODS environment file has been created at /home/dvo0012/.irods/irods_environment.json
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    
    
    to start irodsfs, run:          irodsfs -config ~/.irods/config.yml ~/IRODS
    to start iCommands, run:        iinit
    
    For more information, see https://docs.it4i.cz/dice/
    ```
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    
    
    To mount your iRODS collection to ~/IRODS, run
    ```console
    [some_user@login4.karolina ~]$ irodsfs -config ~/.irods/config.yml ~/IRODS
    time="2022-08-04 08:54:13.222836" level=info msg="Logging to /tmp/irodsfs_cblmq5ab1lsaj31vrv20.log" function=processArguments package=main
    Password:
    time="2022-08-04 08:54:18.698811" level=info msg="Found FUSE Device. Starting iRODS FUSE Lite." function=parentMain package=main
    time="2022-08-04 08:54:18.699080" level=info msg="Running the process in the background mode" function=parentRun package=main
    time="2022-08-04 08:54:18.699544" level=info msg="Process id = 27145" function=parentRun package=main
    time="2022-08-04 08:54:18.699572" level=info msg="Sending configuration data" function=parentRun package=main
    time="2022-08-04 08:54:18.699730" level=info msg="Successfully sent configuration data to background process" function=parentRun package=main
    time="2022-08-04 08:54:18.922490" level=info msg="Successfully started background process" function=parentRun package=main
    ```
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    
    To unmount it, run
    
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ```
    
    you can work with fuse as an ordinary directory  ( ls, cd, cp, mv etc )
    
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    
    ```console
    
    ssh some_user@karolina.it4i.cz
    [some_use@login4.karolina ~]$ ml irodsfs
    irodsfs configuration file has been created at /home/dvo0012/.irods/config.yml.
          to start irods fs run: irodsfs -config ~/.irods/config.yml ~/IRODS
    
    iCommands environment file has been created at /home/$USER/.irods/irods_environment.json.
          to start iCommands run: iinit
    
    [some_use@login4.karolina ~]$ iinit
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    Enter your current PAM password:
    ```
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - put you password you set in keycloak
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ```console
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    /IT4I/home/some_user:
      test.1
      test.2
      test.3
      test.4
    ```
    
    - upload `iput` , download `iget`
    - for more help, type `ihelp`
    
    ### ***Users Without IT4I Account (No Access to Cluster)***
    
    - we suppose, you will upload your data from your local pc/vm
    
    #### You Need a Client to Connect to iRods Server
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - There are many iRods clients, but we recommend theese:
    
    - Cyberduck - for windows / mac, GUI . Password from aai.it4i.cz (keycloak) works
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - Fuse (irodsfs lite) - for linux, cli
    
    - iCommands - for linux, cli . Password from aai.it4i.cz (keycloak) works
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    - download at [https://cyberduck.io/download/][i]
    
    - download connection profile for IT4I irods server [irods.cyberduckprofile][1]
    - left doubleclick at this profile file opens connection
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    #### Irodsfs Lite
    
    - work with your irods collection like ordinary directory
    - this is linux client only, basic knowledge of commandline is neccessary
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - PAM passwords is password, you set in aai.it4i.cz
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    
    ```console
    cd ~
    wget https://docs.it4i.cz/irods1.it4i.cz.crt [k]
    wget https://github.com/cyverse/irodsfs/releases/download/v0.7.6/irodsfs_amd64_linux_v0.7.6.tar
    tar -xvf ~/irodsfs_amd64_linux_v0.7.6.tar
    mkdir ~/IRODS
    ```
    
    settings:
    
    ```console
    
    [some_user@local_pc ~]$ vim ~/.rods/config.yml
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ```
    
    ```console
    host: irods1.it4i.cz
    port: 1247
    proxy_user: some_user
    client_user: some_user
    zone: IT4I
    
    authscheme: "pam"
    ssl_ca_cert_file: "~/irods1.it4i.cz.crt"
    ssl_encryption_key_size: 32
    ssl_encryption_algorithm: "AES-256-CBC"
    ssl_encryption_salt_size: 8
    ssl_encryption_hash_rounds: 16
    
    path_mappings:
      - irods_path: /IT4I/home/some_user
        mapping_path: /
        resource_type: dir
    ```
    
    ***How to start - mount your collection:***
    
    ```console
    
    [some_user@local_pc ~]$ ./irodsfs -config ~/.irods/config.yml ~/IRODS
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    time="2022-07-29 09:51:11.720831" level=info msg="Logging to /tmp/irodsfs_cbhp2rucso0ef0s7dtl0.log" function=processArguments package=main
    Password:
    
    time="2022-07-29 09:51:17.691988" level=info msg="Found FUSE Device. Starting iRODS FUSE Lite." function=parentMain package=main
    time="2022-07-29 09:51:17.692683" level=info msg="Running the process in the background mode" function=parentRun package=main
    time="2022-07-29 09:51:17.693381" level=info msg="Process id = 74772" function=parentRun package=main
    time="2022-07-29 09:51:17.693421" level=info msg="Sending configuration data" function=parentRun package=main
    time="2022-07-29 09:51:17.693772" level=info msg="Successfully sent configuration data to background process" function=parentRun package=main
    time="2022-07-29 09:51:18.008166" level=info msg="Successfully started background process" function=parentRun package=main
    ```
    
    ***How to put your data to iRODS***
    
    ```console
    
    [some_use@local_pc ~]$ cp test1G.txt ~/IRODS
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ```
    
    It works as ordinary filesystem
    
    ```console
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    total 0
    -rwx------ 1 some_user some_user 1073741824 Nov  4  2021 test1G.txt
    ```
    
    ***How to stop - unmout your collection:***
    
    ```console
    
    [some_user@local_pc ~]$ fusermount -u ~/IRODS
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ```
    
    
    - this is linux client only, basic knowledge of commandline is neccessary
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - manual is for centos7
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - for ubuntu20 - there is [http://docs.snic.se/wiki/IRODS_iCommands_installation_on_Ubuntu_20.04#Authenticate_and_test_iRODS_iCommands_client][j]
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - ***please do NOT install icommands 4.3.0, it is NOT working with PAM***
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    ```console
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    sudo rpm --import https://packages.irods.org/irods-signing-key.asc
    sudo wget -qO - https://packages.irods.org/renci-irods.yum.repo | sudo tee /etc/yum.repos.d/renci-irods.yum.repo
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    sudo yum install epel-release -y
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    sudo yum install python-psutil python-jsonschema
    
    sudo yum install irods-icommands-4.2.7-1.x86_64
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    mkdir ~/.irods/
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    cd ~ && wget https://docs.it4i.cz/irods1.it4i.cz.crt [k]
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    ```console
    
    [some_user@local_pc ~]$ vim ~/.irods/irods_environment.json
    
        "irods_host": "irods1.it4i.cz",
    
        "irods_port": 1247,
    
        "irods_user_name": "some_user",
    
        "irods_zone_name": "IT4I",
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
        "irods_authentication_scheme": "PAM",
    
        "irods_ssl_verify_server": "cert",
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
        "irods_ssl_ca_certificate_file": "~/irods1.it4i.cz.crt",
    
        "irods_encryption_algorithm": "AES-256-CBC",
        "irods_encryption_key_size": 32,
        "irods_encryption_num_hash_rounds": 16,
        "irods_encryption_salt_size": 8
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    ```console
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    /some_user/.irods
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    drwx------. 2 some_user some_user 136 Sep 29 08:53 .
    dr-xr-x---. 6 some_user some_user 206 Sep 29 08:53 ..
    -rw-r--r--. 1 some_user some_user 253 Sep 29 08:14 irods_environment.json
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    **How to Start:**
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    **step 1:**
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    ```console
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    Enter your current PAM password:
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
      file.jpg
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    ```
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ***how to put your data to IRODS***
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    /IT4I/home/some_user:
      cesnet.crt
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ***How to download data***
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ls -la ~
    -rw-r--r--. 1 some_user some_user 1464 Jul 20 13:44 cesnet.crt
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    For more commands, use the `ihelp` command.
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    [b]: http://www.diceproject.eu/
    [c]: https://irods.org/
    [d]: https://b2access.eudat.eu/
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    [f]: https://aai.it4i.cz/realms/IT4i_AAI/account/#/
    
    [g]: https://mms.myaccessid.org/profile/
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    [h]: https://www.eudat.eu/contact-support-request?Service=B2SAFE
    [i]: https://cyberduck.io/download/
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    [j]: http://docs.snic.se/wiki/IRODS_iCommands_installation_on_Ubuntu_20.04#Authenticate_and_test_iRODS_iCommands_client
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    [k]: https://docs.it4i.cz/irods1.it4i.cz.crt