Skip to content
Snippets Groups Projects
Select Git revision
  • master default protected
  • dvo0012-dice-patch
  • qfree-update
  • vnode-allocation
  • revert-d95b46c5
  • dvo0012-dice
  • 20180621-revision
  • 20180621-before_revision
8 results
Blame Permalink
Forked from SCS / docs.it4i.cz
1528 commits behind, 378 commits ahead of the upstream repository.
prace.md 13.08 KiB

PRACE User Support

Intro

PRACE users coming to Anselm as to TIER-1 system offered through the DECI calls are in general treated as standard users and so most of the general documentation applies to them as well. This section shows the main differences for quicker orientation, but often uses references to the original documentation. PRACE users who don't undergo the full procedure (including signing the IT4I AuP on top of the PRACE AuP) will not have a password and thus access to some services intended for regular users. This can lower their comfort, but otherwise they should be able to use the TIER-1 system as intended. Please see the Obtaining Login Credentials section, if the same level of access is required.

All general PRACE User Documentation should be read before continuing reading the local documentation here.

Help and Support

If you have any troubles, need information, request support or want to install additional software, please use PRACE Helpdesk.

Information about the local services are provided in the introduction of general user documentation. Please keep in mind, that standard PRACE accounts don't have a password to access the web interface of the local (IT4Innovations) request tracker and thus a new ticket should be created by sending an e-mail to support[at]it4i.cz.

Obtaining Login Credentials

In general PRACE users already have a PRACE account setup through their HOMESITE (institution from their country) as a result of rewarded PRACE project proposal. This includes signed PRACE AuP, generated and registered certificates, etc.

If there's a special need a PRACE user can get a standard (local) account at IT4Innovations. To get an account on the Anselm cluster, the user needs to obtain the login credentials. The procedure is the same as for general users of the cluster, so please see the corresponding section of the general documentation here.

Accessing the cluster

Access with GSI-SSH

For all PRACE users the method for interactive access (login) and data transfer based on grid services from Globus Toolkit (GSI SSH and GridFTP) is supported.

The user will need a valid certificate and to be present in the PRACE LDAP (please contact your HOME SITE or the primary investigator of your project for LDAP account creation).

Most of the information needed by PRACE users accessing the Anselm TIER-1 system can be found here:

Before you start to use any of the services don't forget to create a proxy certificate from your certificate:

    $ grid-proxy-init

To check whether your proxy certificate is still valid (by default it's valid 12 hours), use:

    $ grid-proxy-info

To access Anselm cluster, two login nodes running GSI SSH service are available. The service is available from public Internet as well as from the internal PRACE network (accessible only from other PRACE partners).

Access from PRACE network:

It is recommended to use the single DNS name anselm-prace.it4i.cz which is distributed between the two login nodes. If needed, user can login directly to one of the login nodes. The addresses are:

|Login address|Port|Protocol|Login node| |---|---| |anselm-prace.it4i.cz|2222|gsissh|login1 or login2| |login1-prace.anselm.it4i.cz|2222|gsissh|login1| |login2-prace.anselm.it4i.cz|2222|gsissh|login2|

    $ gsissh -p 2222 anselm-prace.it4i.cz

When logging from other PRACE system, the prace_service script can be used:

    $ gsissh `prace_service -i -s anselm`

Access from public Internet:

It is recommended to use the single DNS name anselm.it4i.cz which is distributed between the two login nodes. If needed, user can login directly to one of the login nodes. The addresses are:

|Login address||Port|Protocol|Login node| |---|---| |anselm.it4i.cz|2222|gsissh|login1 or login2| |login1.anselm.it4i.cz|2222|gsissh|login1| |login2.anselm.it4i.cz|2222|gsissh|login2|

    $ gsissh -p 2222 anselm.it4i.cz

When logging from other PRACE system, the prace_service script can be used:

    $ gsissh `prace_service -e -s anselm`

Although the preferred and recommended file transfer mechanism is using GridFTP, the GSI SSH implementation on Anselm supports also SCP, so for small files transfer gsiscp can be used:

    $ gsiscp -P 2222 _LOCAL_PATH_TO_YOUR_FILE_ anselm.it4i.cz:_ANSELM_PATH_TO_YOUR_FILE_

    $ gsiscp -P 2222 anselm.it4i.cz:_ANSELM_PATH_TO_YOUR_FILE_ _LOCAL_PATH_TO_YOUR_FILE_

    $ gsiscp -P 2222 _LOCAL_PATH_TO_YOUR_FILE_ anselm-prace.it4i.cz:_ANSELM_PATH_TO_YOUR_FILE_

    $ gsiscp -P 2222 anselm-prace.it4i.cz:_ANSELM_PATH_TO_YOUR_FILE_ _LOCAL_PATH_TO_YOUR_FILE_

Access to X11 applications (VNC)

If the user needs to run X11 based graphical application and does not have a X11 server, the applications can be run using VNC service. If the user is using regular SSH based access, please see the section in general documentation.

If the user uses GSI SSH based access, then the procedure is similar to the SSH based access, only the port forwarding must be done using GSI SSH:

    $ gsissh -p 2222 anselm.it4i.cz -L 5961:localhost:5961

Access with SSH

After successful obtainment of login credentials for the local IT4Innovations account, the PRACE users can access the cluster as regular users using SSH. For more information please see the section in general documentation.

File transfers

PRACE users can use the same transfer mechanisms as regular users (if they've undergone the full registration procedure). For information about this, please see the section in the general documentation.

Apart from the standard mechanisms, for PRACE users to transfer data to/from Anselm cluster, a GridFTP server running Globus Toolkit GridFTP service is available. The service is available from public Internet as well as from the internal PRACE network (accessible only from other PRACE partners).

There's one control server and three backend servers for striping and/or backup in case one of them would fail.

Access from PRACE network:

|Login address|Port|Node role| |---|---| |gridftp-prace.anselm.it4i.cz|2812|Front end /control server| |login1-prace.anselm.it4i.cz|2813|Backend / data mover server| |login2-prace.anselm.it4i.cz|2813|Backend / data mover server| |dm1-prace.anselm.it4i.cz|2813|Backend / data mover server|

Copy files to Anselm by running the following commands on your local machine:

    $ globus-url-copy file://_LOCAL_PATH_TO_YOUR_FILE_ gsiftp://gridftp-prace.anselm.it4i.cz:2812/home/prace/_YOUR_ACCOUNT_ON_ANSELM_/_PATH_TO_YOUR_FILE_

Or by using prace_service script:

    $ globus-url-copy file://_LOCAL_PATH_TO_YOUR_FILE_ gsiftp://`prace_service -i -f anselm`/home/prace/_YOUR_ACCOUNT_ON_ANSELM_/_PATH_TO_YOUR_FILE_

Copy files from Anselm:

    $ globus-url-copy gsiftp://gridftp-prace.anselm.it4i.cz:2812/home/prace/_YOUR_ACCOUNT_ON_ANSELM_/_PATH_TO_YOUR_FILE_ file://_LOCAL_PATH_TO_YOUR_FILE_

Or by using prace_service script:

    $ globus-url-copy gsiftp://`prace_service -i -f anselm`/home/prace/_YOUR_ACCOUNT_ON_ANSELM_/_PATH_TO_YOUR_FILE_ file://_LOCAL_PATH_TO_YOUR_FILE_

Access from public Internet:

|Login address|Port|Node role| |---|---| |gridftp.anselm.it4i.cz|2812|Front end /control server| |login1.anselm.it4i.cz|2813|Backend / data mover server| |login2.anselm.it4i.cz|2813|Backend / data mover server| |dm1.anselm.it4i.cz|2813|Backend / data mover server|

Copy files to Anselm by running the following commands on your local machine:

    $ globus-url-copy file://_LOCAL_PATH_TO_YOUR_FILE_ gsiftp://gridftp.anselm.it4i.cz:2812/home/prace/_YOUR_ACCOUNT_ON_ANSELM_/_PATH_TO_YOUR_FILE_

Or by using prace_service script:

    $ globus-url-copy file://_LOCAL_PATH_TO_YOUR_FILE_ gsiftp://`prace_service -e -f anselm`/home/prace/_YOUR_ACCOUNT_ON_ANSELM_/_PATH_TO_YOUR_FILE_

Copy files from Anselm:

    $ globus-url-copy gsiftp://gridftp.anselm.it4i.cz:2812/home/prace/_YOUR_ACCOUNT_ON_ANSELM_/_PATH_TO_YOUR_FILE_ file://_LOCAL_PATH_TO_YOUR_FILE_

Or by using prace_service script:

    $ globus-url-copy gsiftp://`prace_service -e -f anselm`/home/prace/_YOUR_ACCOUNT_ON_ANSELM_/_PATH_TO_YOUR_FILE_ file://_LOCAL_PATH_TO_YOUR_FILE_

Generally both shared file systems are available through GridFTP:

|File system mount point|Filesystem|Comment| |---|---| |/home|Lustre|Default HOME directories of users in format /home/prace/login/| |/scratch|Lustre|Shared SCRATCH mounted on the whole cluster|

More information about the shared file systems is available here.

Usage of the cluster

There are some limitations for PRACE user when using the cluster. By default PRACE users aren't allowed to access special queues in the PBS Pro to have high priority or exclusive access to some special equipment like accelerated nodes and high memory (fat) nodes. There may be also restrictions obtaining a working license for the commercial software installed on the cluster, mostly because of the license agreement or because of insufficient amount of licenses.

For production runs always use scratch file systems, either the global shared or the local ones. The available file systems are described here.

Software, Modules and PRACE Common Production Environment

All system wide installed software on the cluster is made available to the users via the modules. The information about the environment and modules usage is in this section of general documentation.

PRACE users can use the "prace" module to use the PRACE Common Production Environment.

    $ module load prace

Resource Allocation and Job Execution

General information about the resource allocation, job queuing and job execution is in this section of general documentation.

For PRACE users, the default production run queue is "qprace". PRACE users can also use two other queues "qexp" and "qfree".

|queue|Active project|Project resources|Nodes|priority|authorization|walltime| |---|---|---| |qexp Express queue|no|none required|2 reserved, 8 total|high|no|1 / 1h| |qprace Production queue|yes|> 0|178 w/o accelerator|medium|no|24 / 48h| |qfree Free resource queue|yes|none required|178 w/o accelerator|very low|no| 12 / 12h|

qprace**, the PRACE ***: This queue is intended for normal production runs. It is required that active project with nonzero remaining resources is specified to enter the qprace. The queue runs with medium priority and no special authorization is required to use it. The maximum runtime in qprace is 12 hours. If the job needs longer time, it must use checkpoint/restart functionality.

Accounting & Quota

The resources that are currently subject to accounting are the core hours. The core hours are accounted on the wall clock basis. The accounting runs whenever the computational cores are allocated or blocked via the PBS Pro workload manager (the qsub command), regardless of whether the cores are actually used for any calculation. See example in the general documentation.

PRACE users should check their project accounting using the PRACE Accounting Tool (DART).

Users who have undergone the full local registration procedure (including signing the IT4Innovations Acceptable Use Policy) and who have received local password may check at any time, how many core-hours have been consumed by themselves and their projects using the command "it4ifree". Please note that you need to know your user password to use the command and that the displayed core hours are "system core hours" which differ from PRACE "standardized core hours".

!!! Note "Note" The it4ifree command is a part of it4i.portal.clients package, located here: https://pypi.python.org/pypi/it4i.portal.clients

    $ it4ifree
    Password:
         PID    Total   Used   ...by me Free
       -------- ------- ------ -------- -------
       OPEN-0-0 1500000 400644   225265 1099356
       DD-13-1    10000   2606     2606    7394

By default file system quota is applied. To check the current status of the quota use

    $ lfs quota -u USER_LOGIN /home
    $ lfs quota -u USER_LOGIN /scratch

If the quota is insufficient, please contact the support and request an increase.