Skip to content
Snippets Groups Projects
dice.md 9.17 KiB
Newer Older
  • Learn to ignore specific revisions
  • Jan Siwiec's avatar
    Jan Siwiec committed
    # What Is DICE Project?
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    
    DICE (Developing Innovation and Creativity in Education) is an international project co-funded by the Erasmus+ Programme of the European Union.
    
    Its main goals are related to fostering innovation and creativity in education.
    
    This project attempts to be a leading initiative that supports educational community to make progress in the field of innovation and creativity through effective, authentic, practical, sustainable actions.
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    This project has been funded with support from the European Commission.
    For more information, see [http://www.diceproject.eu/][b].
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    **IT4Innovations participates in DICE. DICE uses the iRods software**
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    The integrated Rule-Oriented Data System (iRODS) is an open source data management software used by research organizations and government agencies worldwide. iRODS is released as a production-level distribution aimed at deployment in mission critical environments. It virtualizes data storage resources, so users can take control of their data, regardless of where and on what device the data is stored. As data volumes grow and data services become more complex, iRODS is serving an increasingly important role in data management, see [https://irods.org/][c].
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    ## How to Put Your Data to Our Server
    
    
    **Prerequisities:**
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    
    We have to be sure about your identity, this will be done throught this steps:
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - at the first, fill this request for eudat [https://www.eudat.eu/contact-support-request?Service=B2SAFE][h]
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    - sign to MyAccessId [https://ds.myaccessid.org/][g] , after that, please create account at B2Access [https://b2access.eudat.eu/][d]
    
    - after successfull login to B2Access, please sign in to our AAI at : [https://aai.it4i.cz/realms/IT4i_AAI/account/#/][f] throught your B2Access account. You can set new password. this new password will work for cyberduck and icommands.
    
    - finally, contact our support@it4i.cz by crypted mail, we will create account at our iRods server and send you crypted mail.
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    ![](img/aai2.jpg)
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    ![](img/aai3-passwd.jpg)
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ### ***IT4I User (You Have IT4I Account With Access to Cluster Already)***
    
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - iRODS clients are provided as a module on our ***Karolina*** cluster ( Barbora is in progress )
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - module irodsfs contains config file for irodsfs nad for icommands too
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - PAM passwords is password, you set in aai.it4i.cz
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    
    ***How to mount your collection***
    
    ```console
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ssh some_user@karolina.it4i.cz
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ml irodsfs
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ```
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - config files for irodsfs and icommands will be loaded 
    - now you can choose between Fuse client or iCommands
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ***Fuse***
    
    ```console
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    $ ml irodsfs
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    
    The following have been reloaded with a version change:
      1) iRODS/0.7.3 => iRODS/0.7.6
    
    iRODS configuration file has been created at /home/some_user/.config/iRODS/config.yml.
    
    To mount your iRODS repository at ~/IRODS, run
            irodsfs -config ~/.config/iRODS/config.yml ~/IRODS
    
    To unmount it, run
            fusermount -u ~/IRODS
    
    $ irodsfs -config ~/.config/iRODS/config.yml ~/IRODS
    time="2022-08-01 12:44:22.980139" level=info msg="Logging to /tmp/irodsfs_cbjqt1ib1ls3g1gtnnog.log" function=processArguments package=main
    Password:
    time="2022-08-01 12:44:27.319063" level=info msg="Found FUSE Device. Starting iRODS FUSE Lite." function=parentMain package=main
    time="2022-08-01 12:44:27.319319" level=info msg="Running the process in the background mode" function=parentRun package=main
    time="2022-08-01 12:44:27.319721" level=info msg="Process id = 64406" function=parentRun package=main
    time="2022-08-01 12:44:27.319752" level=info msg="Sending configuration data" function=parentRun package=main
    time="2022-08-01 12:44:27.319909" level=info msg="Successfully sent configuration data to background process" function=parentRun package=main
    time="2022-08-01 12:44:27.555090" level=info msg="Successfully started background process" function=parentRun package=main
    ```
    
    you can work with fuse as an ordinary directory  ( ls, cd, cp, mv etc )
    
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ***iCommands***  
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    
    ```console
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ml irodsfs
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    iinit
    Enter your current PAM password:
    ```
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - put you password you set in keycloak
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ```console
    ils
    /IT4I/home/some_user:
      test.1
      test.2
      test.3
      test.4
    ```
    
    - upload `iput` , download `iget`
    - for more help, type `ihelp`
    
    ### ***Users Without IT4I Account (No Access to Cluster)***
    
    - we suppose, you will upload your data from your local pc/vm
    
    #### You Need a Client to Connect to iRods Server
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - There are many iRods clients, but we recommend theese:
    
    - Cyberduck - for windows / mac, GUI . Password from aai.it4i.cz (keycloak) works
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - Fuse (irodsfs lite) - for linux, cli
    
    - iCommands - for linux, cli . Password from aai.it4i.cz (keycloak) works
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    - download at [https://cyberduck.io/download/][i]
    
    - download connection profile for IT4I irods server [irods.cyberduckprofile][1]
    - left doubleclick at this profile file opens connection
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    #### Irodsfs Lite
    
    - work with your irods collection like ordinary directory
    - this is linux client only, basic knowledge of commandline is neccessary
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - PAM passwords is password, you set in aai.it4i.cz
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    
    ```console
    cd ~
    wget https://docs.it4i.cz/irods1.it4i.cz.crt [k]
    wget https://github.com/cyverse/irodsfs/releases/download/v0.7.6/irodsfs_amd64_linux_v0.7.6.tar
    tar -xvf ~/irodsfs_amd64_linux_v0.7.6.tar
    mkdir ~/IRODS
    ```
    
    settings:
    
    ```console
    vim ~/config.yml
    ```
    
    ```console
    host: irods1.it4i.cz
    port: 1247
    proxy_user: some_user
    client_user: some_user
    zone: IT4I
    
    authscheme: "pam"
    ssl_ca_cert_file: "~/irods1.it4i.cz.crt"
    ssl_encryption_key_size: 32
    ssl_encryption_algorithm: "AES-256-CBC"
    ssl_encryption_salt_size: 8
    ssl_encryption_hash_rounds: 16
    
    path_mappings:
      - irods_path: /IT4I/home/some_user
        mapping_path: /
        resource_type: dir
    ```
    
    ***How to start - mount your collection:***
    
    ```console
    ./irodsfs -config ~/config.yml ~/IRODS
    time="2022-07-29 09:51:11.720831" level=info msg="Logging to /tmp/irodsfs_cbhp2rucso0ef0s7dtl0.log" function=processArguments package=main
    Password:
    
    time="2022-07-29 09:51:17.691988" level=info msg="Found FUSE Device. Starting iRODS FUSE Lite." function=parentMain package=main
    time="2022-07-29 09:51:17.692683" level=info msg="Running the process in the background mode" function=parentRun package=main
    time="2022-07-29 09:51:17.693381" level=info msg="Process id = 74772" function=parentRun package=main
    time="2022-07-29 09:51:17.693421" level=info msg="Sending configuration data" function=parentRun package=main
    time="2022-07-29 09:51:17.693772" level=info msg="Successfully sent configuration data to background process" function=parentRun package=main
    time="2022-07-29 09:51:18.008166" level=info msg="Successfully started background process" function=parentRun package=main
    ```
    
    ***How to put your data to iRODS***
    
    ```console
    cp test1G.txt ~/IRODS
    ```
    
    It works as ordinary filesystem
    
    ```console
    ls -la ~/IRODS
    total 0
    -rwx------ 1 some_user some_user 1073741824 Nov  4  2021 test1G.txt
    ```
    
    ***How to stop - unmout your collection:***
    
    ```console
    fusermount -u ~/IRODS
    ```
    
    
    - this is linux client only, basic knowledge of commandline is neccessary
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - manual is for centos7
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - for ubuntu20 - there is [http://docs.snic.se/wiki/IRODS_iCommands_installation_on_Ubuntu_20.04#Authenticate_and_test_iRODS_iCommands_client][j]
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    - ***please do NOT install icommands 4.3.0, it is NOT working with PAM***
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    ```console
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    sudo rpm --import https://packages.irods.org/irods-signing-key.asc
    sudo wget -qO - https://packages.irods.org/renci-irods.yum.repo | sudo tee /etc/yum.repos.d/renci-irods.yum.repo
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    sudo yum install epel-release -y
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    sudo yum install python-psutil python-jsonschema
    sudo yum install irods-icommands-4.2.7-1.x86_64 
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    mkdir ~/.irods/
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    cd ~ && wget https://docs.it4i.cz/irods1.it4i.cz.crt [k]
    
    ```
    
    Copy&paste + edit irods_user_name
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    ```console
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    $ vim ~/.irods/irods_environment.json
    
        "irods_host": "irods1.it4i.cz",
    
        "irods_port": 1247,
    
        "irods_user_name": "some_user",
    
        "irods_zone_name": "IT4I",
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
        "irods_authentication_scheme": "PAM",
    
        "irods_ssl_verify_server": "cert",
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
        "irods_ssl_ca_certificate_file": "~/irods1.it4i.cz.crt",
    
        "irods_encryption_algorithm": "AES-256-CBC",
        "irods_encryption_key_size": 32,
        "irods_encryption_num_hash_rounds": 16,
        "irods_encryption_salt_size": 8
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    ```console
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    /some_user/.irods
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    drwx------. 2 some_user some_user 136 Sep 29 08:53 .
    dr-xr-x---. 6 some_user some_user 206 Sep 29 08:53 ..
    -rw-r--r--. 1 some_user some_user 253 Sep 29 08:14 irods_environment.json
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    **How to Start:**
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    **step 1:**
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    ```console
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    Enter your current PAM password:
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
      file.jpg
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    ```
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ***how to put your data to IRODS***
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    $ iput cesnet.crt
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    $ ils
    /IT4I/home/some_user:
      cesnet.crt
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    ***How to download data***
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    $ iget cesnet.crt
    ls -la ~
    -rw-r--r--. 1 some_user some_user 1464 Jul 20 13:44 cesnet.crt
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    For more commands, use the `ihelp` command.
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    [b]: http://www.diceproject.eu/
    [c]: https://irods.org/
    [d]: https://b2access.eudat.eu/
    
    Jan Siwiec's avatar
    Jan Siwiec committed
    [f]: https://aai.it4i.cz/realms/IT4i_AAI/account/#/
    [g]: https://ds.myaccessid.org/
    [h]: https://www.eudat.eu/contact-support-request?Service=B2SAFE
    [i]: https://cyberduck.io/download/
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    [j]: http://docs.snic.se/wiki/IRODS_iCommands_installation_on_Ubuntu_20.04#Authenticate_and_test_iRODS_iCommands_client
    
    Ondrej Dvorak's avatar
    Ondrej Dvorak committed
    [k]: https://docs.it4i.cz/irods1.it4i.cz.crt