diff --git a/docs.it4i/apiv2.md b/docs.it4i/apiv2.md new file mode 100644 index 0000000000000000000000000000000000000000..eb3c0304f209ba14eb45e19bb32cb905ef9d11e8 --- /dev/null +++ b/docs.it4i/apiv2.md @@ -0,0 +1,203 @@ +# SCS API v2 + +## Info + +- **OpenAPI:** 3.1.0 +- **Title:** scs-api-2 +- **Version:** 0.1.0 +- **Server URL:** `https://scs.it4i.cz/api/v2` + +## Paths + +### `/dedicated-time` + +#### GET + +- **Summary:** Get dedicated times +- **Description:** Retrieves dedicated time entries, optionally filtered by cluster name or period preset +- **OperationId:** `dedicated_time_handler` + +**Parameters:** + +- `cluster` (query): Filter by cluster name; Available values: karolina, barbora, dgx *(optional)* +- `period` (query): Filter by time period preset; Available values: planned, active *(optional)* + +**Responses:** + +- `200`: List of dedicated time entries +- `400`: Failed to deserialize query, Invalid cluster, Invalid period + Example: + + ```json + { + "message": "Invalid cluster: el_gordo" + } + ``` +- `500`: Failed to retrieve dedicated time due to a server error + Example: + ```json + { + "message": "Failed to retreive dedicated time" + } + ``` + +### `/dedicated-time-calendar` + +#### GET + +- **Summary:** Get dedicated times +- **Description:** Retrieves dedicated time entries and generates a VCalendar response. +- **OperationId:** `dedicated_time_calendar` + +**Responses:** + +- `200`: Dedicated time VCalendar + Example: + + ``` + BEGIN:VCALENDAR + VERSION:2.0 + PRODID:-//SUTD Timetable Calendar//randName//EN + CALSCALE:GREGORIAN + BEGIN:VEVENT + UID:1234@example.com + DTSTAMP:20230101T000000Z + DTSTART:20230101T000000Z + DTEND:20230102T000000Z + SUMMARY:Sample Dedicated Time - Cluster Outage + DESCRIPTION:Sample Dedicated Time - Cluster Outage + END:VEVENT + END:VCALENDAR + ``` + +- `500`: Failed to retrieve dedicated time calendar + Example: + + ```json + { + "message": "Failed to retreive dedicated time calendar" + } + ``` + +### `/motd` + +#### GET + +- **Summary:** Get messages of the day +- **Description:** Retrieves messages of the day, optionally filtered by category +- **OperationId:** `motd` + +**Parameters:** + +- `category` (query): *(optional)* + +**Responses:** + +- `200`: List of motd entries +- `400`: Failed to deserialize query, Invalid motd category +- `500`: Failed to retrieve motd entries due to a server error + Example: + + ```json + { + "message": "Failed to retrieve motd" + } + ``` + +## Components + +### Schemas + +#### DedicatedTime + +```yaml +type: object +required: + - updated_at +properties: + cluster_type: + type: [string, 'null'] + date_efficiency: + type: [string, 'null'] + format: date-time + date_expiration: + type: [string, 'null'] + format: date-time + updated_at: + type: string + format: date-time +``` + +#### Motd + +```yaml +type: object +required: + - id + - author + - category + - created_at + - updated_at + - date_modification + - title + - message_body + - systems +properties: + id: + type: integer + format: int32 + examples: [1] + author: + type: string + examples: [Admin] + category: + type: string + examples: [public-service-announcement] + created_at: + type: string + format: date-time + updated_at: + type: string + format: date-time + date_modification: + type: string + format: date-time + date_efficiency: + type: [string, 'null'] + format: date-time + date_expiration: + type: [string, 'null'] + format: date-time + date_outage_efficiency: + type: [string, 'null'] + format: date-time + date_outage_expiration: + type: [string, 'null'] + format: date-time + title: + type: string + examples: [Important Update] + message_body: + type: string + examples: [We are experiencing some service disruptions.] + systems: + type: array + items: + type: string + examples: [Karolina] +``` + +#### MsgResponse + +```yaml +type: object +description: | + Common struct for DTO-less responses + eg. ```200 {"message":"Operation succeeded"}``` +required: + - message +properties: + message: + type: string + examples: [API response] +``` diff --git a/docs.it4i/general/tools/cli-client-tools.md b/docs.it4i/general/tools/cli-client-tools.md new file mode 100644 index 0000000000000000000000000000000000000000..e4f44d51f058c28b26dbfbf8b945f349dc11989b --- /dev/null +++ b/docs.it4i/general/tools/cli-client-tools.md @@ -0,0 +1,151 @@ +# SCS CLI Client Tools + +## Configuration + +System wide configuration -> `etc/scs-cli-client-tools/config.toml` +User configuration -> `$XDG_CONFIG_HOME/scs-cli-client-tools/config.toml` + +## it4ifree + +`MUNGE_SOCK` - the path to the munged socket. +`CLUSTERNAME` - name of the cluster `it4ifree` is running on. +`API_URL` - set this environment variable to the API address. [default: `https://scs.it4i.cz/api/v2/`] + +--- + +```code +List project resource usage data from SCS IS. +Also available at [https://scs.it4i.cz][a]. + +Usage: it4ifree [OPTIONS] + +Options: + -p, --percent + -P, --project <PROJECT> + -u, --per-user + -s, --status <STATUS> [possible values: active, planned, expired, all] + -S, --show-status Show status of project resource + --no-status Hide status of project resource + -q, --qos Show a column with the QoS value for resource + -i, --icons <ICONS> When enabled shows emojis for better readability [default: auto] [possible values: always, auto, never] + -j, --json Print in json format + -h, --help Print help + -V, --version Print version +``` + +### Example Output of 'it4ifree' + +```code +╭─────────────┬──────────────┬───────────────┬──────────────────────────┬────────────┬───────┬──────────────╮ +│ Project PID ┆ Resource ┆ Resource Type ┆ Validity Period ┆ Allocation ┆ Usage ┆ Time left │ +╞═════════════╪══════════════╪═══════════════╪══════════════════════════╪════════════╪═══════╪══════════════╡ +│ OPEN-21-48 ┆ Barbora CPU ┆ Node Hours ┆ 2024-11-15 to 2025-08-11 ┆ 20000.00 ┆ 0.00 ┆ 145d 14h 25m │ +│ ┆ Karolina CPU ┆ Node Hours ┆ 2024-11-15 to 2025-08-11 ┆ 20000.00 ┆ 0.00 ┆ 145d 14h 25m │ +│ ┆ Karolina GPU ┆ Node Hours ┆ 2024-11-15 to 2025-08-11 ┆ 6900.00 ┆ 0.00 ┆ 145d 14h 25m │ +╰─────────────┴──────────────┴───────────────┴──────────────────────────┴────────────┴───────┴──────────────╯ +``` + +### Example Output of 'it4ifree -u -s all' + +```code +╭───────────────────┬──────────────┬───────────────┬──────────────────────────┬────────────┬───────────────┬────────────┬──────────────╮ +│ Project PID ┆ Resource ┆ Resource Type ┆ Validity Period ┆ Allocation ┆ User / Total ┆ Usage ┆ Time left │ +╞═══════════════════╪══════════════╪═══════════════╪══════════════════════════╪════════════╪═══════════════╪════════════╪══════════════╡ +│ OPEN-21-16 [PI] ┆ Karolina GPU ┆ Node Hours ┆ 2023-01-01 to 2023-12-31 ┆ 50000.00 ┆ =Usage Total= ┆ 25000.00 ┆ 25d 4h 12m │ +│ ┆ ┆ ┆ ┆ ┆ user1 ┆ 10000.00 ┆ │ +│ ┆ ┆ ┆ ┆ ┆ user2 ┆ 7500.00 ┆ │ +│ ┆ ┆ ┆ ┆ ┆ user3 ┆ 7500.00 ┆ │ +│ OPEN-22-05 ┆ Barbora CPU ┆ Legacy NCH ┆ 2022-06-01 to 2023-06-01 ┆ 10000.00 ┆ =Usage Total= ┆ 8000.00 ┆ 118d 18h 49m │ +│ ┆ ┆ ┆ ┆ ┆ user4 ┆ 3000.00 ┆ │ +│ ┆ ┆ ┆ ┆ ┆ user5 ┆ 5000.00 ┆ │ +│ OPEN-23-10 ┆ Karolina CPU ┆ Node Hours ┆ 2023-03-01 to 2024-03-01 ┆ 75000.00 ┆ =Usage Total= ┆ 30000.00 ┆ 15d 11h 19m │ +│ ┆ ┆ ┆ ┆ ┆ user6 ┆ 15000.00 ┆ │ +│ ┆ ┆ ┆ ┆ ┆ user7 ┆ 15000.00 ┆ │ +│ OPEN-20-08 ┆ Barbora GPU ┆ Legacy NCH ┆ 2020-05-01 to 2023-05-01 ┆ 20000.00 ┆ =Usage Total= ┆ 20000.00 ┆ │ +│ ┆ ┆ ┆ ┆ ┆ user8 ┆ 10000.00 ┆ │ +│ ┆ ┆ ┆ ┆ ┆ user9 ┆ 10000.00 ┆ │ +╰───────────────────┴──────────────┴───────────────┴──────────────────────────┴────────────┴───────────────┴────────────┴──────────────╯ +``` + +## it4imotd + +`API_URL` - set this environment variable to the API address. [default: `https://scs.it4i.cz/api/v2/`] + +```code +The command shows IT4I messages of the day + +Usage: it4imotd [OPTIONS] + +Options: + -w, --width <WIDTH> [default: 78] + -c, --category <CATEGORY> [possible values: public-service-announcement, service-recovered-up, critical-service-down, service-hard-down, auxiliry-service-down, planned-outage, service-degraded, important, notice] + -h, --help Print help + -V, --version Print version +``` + +```code + Critical service down: Unplanned Power Outage +Posted: 2024-09-13 12:35:18 +Outage: 2024-12-17 14:27:43 to 2025-12-17 18:27:43 +Affected systems: Barbora, Karolina + +Due to an unplanned power outage, several nodes on both the Karolina and +Barbora clusters went down, which affected some running jobs. We apologize for +any inconvenience. +``` + +## it4ifsusage + +`MUNGE_SOCK` - the path to the munged socket. +`CLUSTERNAME` - name of the cluster `it4ifsusage` is running on. +`API_URL` - set this environment variable to the API address. [default: `https://scs.it4i.cz/api/v2/`] + +```code +The command shows filesystem usage of IT4I cluster storage systems + +User quotas are always active, expiration timestamps signify an end of a temporary quota increase. +Project quotas are active during the lifetime of a project. + +Usage: it4ifsusage [OPTIONS] + +Options: + -s, --status <STATUS> [default: active] [possible values: active, expired, all] + -j, --json Output to json instead of a table + -h, --help Print help + -V, --version Print version +``` + +```code +╭────────────┬───────────────┬─────────────┬────────────┬─────────────┬──────────────┬───────────────┬────────────┬────────────────────────────╮ +│ Quota Type ┆ Cluster / PID ┆ File System ┆ Space Used ┆ Space Limit ┆ Entries Used ┆ Entries Limit ┆ Expiration ┆ Last Update │ +╞════════════╪═══════════════╪═════════════╪════════════╪═════════════╪══════════════╪═══════════════╪════════════╪════════════════════════════╡ +│ User ┆ Barbora ┆ /home ┆ 86 KB ┆ 25 GB ┆ 20 ┆ 500,000 ┆ ┆ 2025-03-19 10:50:20 │ +│ User ┆ Barbora ┆ /scratch ┆ 0 B ┆ 10 TB ┆ 0 ┆ 10,000,000 ┆ ┆ 2025-03-19 10:50:31 │ +│ User ┆ Karolina ┆ /home ┆ 641.7 MB ┆ 25 GB ┆ 15,462 ┆ 500,000 ┆ ┆ 2025-03-19 10:40:34 │ +│ User ┆ Karolina ┆ /scratch ┆ 0 B ┆ 0 B ┆ 0 ┆ 0 ┆ ┆ 2025-03-19 10:40:49 │ +│ Project ┆ OPEN-18-36 ┆ proj3 ┆ 642.9 GB ┆ 1 TB ┆ 577,894 ┆ 1,000,000 ┆ 2039-01-19 ┆ 2025-03-19 10:40:02.722305 │ +╰────────────┴───────────────┴─────────────┴────────────┴─────────────┴──────────────┴───────────────┴────────────┴────────────────────────────╯ +``` + +## it4idedicatedtime + +```code +Usage: it4idedicatedtime [OPTIONS] + +Options: + -s, --status <STATUS> [default: active] [possible values: active, planned] + -c, --cluster <CLUSTER> [default: all] [possible values: all, barbora, karolina, comp-sys, dgx] + -j, --json + -h, --help Print help + -V, --version Print version +``` + +```code +╭─────────┬─────────────────────┬─────────────────────┬─────────────────────╮ +│ Cluster ┆ Start ┆ End ┆ Last Update │ +╞═════════╪═════════════════════╪═════════════════════╪═════════════════════╡ +│ Barbora ┆ 2023-07-19 07:00:00 ┆ 2025-07-23 23:22:00 ┆ 2023-07-16 23:23:33 │ +╰─────────┴─────────────────────┴─────────────────────┴─────────────────────╯ +``` + +[a]: https://scs.it4i.cz \ No newline at end of file diff --git a/docs.it4i/general/tools/portal-clients.md b/docs.it4i/general/tools/portal-clients.md deleted file mode 100644 index 7d1ba60e51d286a96a52294db4eda16eec36cd84..0000000000000000000000000000000000000000 --- a/docs.it4i/general/tools/portal-clients.md +++ /dev/null @@ -1,295 +0,0 @@ -# it4i-portal-clients - -it4i-portal-clients provides simple user-friendly shell interface -to call [IT4I API](https://docs.it4i.cz/apiv1/) requests and display their respond. - -!!! important - Python 2.7 is required. - -Limits are placed on the number of requests you may make to [IT4I API](https://docs.it4i.cz/apiv1/). -Rate limit can be changed without any warning at any time, but the default is **6 requests per minute**. -Exceeding the limit will lead to your IP address being temporarily blocked from making further requests. -The block will automatically be lifted by waiting an hour. - -## List of Available Utilities - -* [it4icheckaccess](#it4icheckaccess) - Shows if IT4I account and/or related project has the access to specified cluster and queue. -* [it4idedicatedtime](#it4idedicatedtime) - Shows IT4I dedicated time. -* [it4ifree](#it4ifree) - Shows some basic information from IT4I Slurm accounting. -* [it4ifsusage](#it4ifsusage) - Shows filesystem usage of IT4I cluster storage systems. -* [it4iuserfsusage](#it4iuserfsusage) - Shows user filesystem usage of IT4I cluster storage systems. -* [it4projectifsusage](#it4iprojectfsusage) - Shows project filesystem usage of IT4I cluster storage systems. -* [it4imotd](#it4imotd) - Shows IT4I messages of the day into formatted text or HTML page (using TAL / Zope Page Template). - -## Installation/Upgrading - -```bash -pip install --upgrade it4i.portal.clients -``` - -## Sample Configuration File main.cfg - -```bash -[main] - -# IT4I API -api_url = https://scs.it4i.cz/api/v1/ -it4ifreetoken = <your_token> -``` - -Username is taken from OS, therefore the script has to be run under the same user login name as you use to log into clusters. - -* System-wide config file path: ```/usr/local/etc/it4i-portal-clients/main.cfg``` -* Local user's config file path: ```~/.it4ifree``` - -## it4icheckaccess - -### Help of IT4ICHECKACCESS - -```console -$ it4icheckaccess -h -usage: it4icheckaccess [-h] -l LOGIN -c CLUSTER -q QUEUE [-p PROJECT] - -The command shows if an IT4I account and/or related project has the access to -specified cluster and queue. Return exit code 99 if access is not granted. - -optional arguments: - -h, --help show this help message and exit - -l LOGIN, --login LOGIN - user login - -c CLUSTER, --cluster CLUSTER - cluster name - -q QUEUE, --queue QUEUE - queue - -p PROJECT, --project PROJECT - project id -``` - -### Example of IT4ICHECKACCESS - -```console -$ it4icheckaccess -l xxx0123 -c barbora -q qcpu_exp -p DD-12-345 -OK Access granted for regular queue. -``` - -## it4idedicatedtime - -### Help of IT4IDEDICATEDTIME - -```console -$ it4idedicatedtime -h -usage: it4idedicatedtime [-h] [-m {active,planned}] - [-c {barbora,karolina}] - -The command shows IT4I dedicated time. By default all planned and active -outages of all clusters are displayed. Return exit code 99 if there is no -outage, otherwise return 0. - -optional arguments: - -h, --help show this help message and exit - -m {active,planned}, --message {active,planned} - select type of dedicated time. Planned contains also - active - -c {barbora,karolina}, --cluster {barbora,karolina} - select cluster -``` - -### Example of IT4IDEDICATEDTIME - -```console -$ it4idedicatedtime -Cluster Start End Last update ---------- ------------------- ------------------- ------------------- -barbora 2024-03-19 08:00:00 2024-03-19 09:30:00 2024-03-08 08:24:33 -karolina 2024-03-19 08:00:00 2024-03-19 09:30:00 2024-03-08 08:23:40 -``` - -## it4ifree - -### Help of IT4IFREE - -```console -$ it4ifree -h -usage: it4ifree [-h] [-p] [-a] - -The command shows some basic information from IT4I SLURM accounting. The -data is related to the current user and to all projects in which user -participates. - -optional arguments: - -h, --help show this help message and exit - -p, --percent - show values in percentage. Projects with unlimited resources are not displayed - -a, --all Show all resources include inactive and future ones. - - -Columns of "Projects I am participating in": - PID: Project ID/account string. - Type: Standard or multiyear project. - Days left: Days till the given project expires. - Total: Core-hours allocated to the given project. - Used: Sum of core-hours used by all project members. - My: Core-hours used by the current user only. - Free: Core-hours that haven't yet been utilized. - -Columns of "Projects I am Primarily Investigating" (if present): - PID: Project ID/account string. - Type: Standard or multiyear project. - Login: Project member's login name. - Used: Project member's used core-hours. -``` - -### Example of IT4IFREE - -```console -$ it4ifree - -Projects I am participating in -============================== -PID Resource type Days left Total Used By me Free ----------- --------------- ------------- -------- -------- -------- -------- -OPEN-XX-XX Karolina GPU 249 42 0 0 42 - Barbora CPU 249 42 5 5 37 - Legacy NCH 249 100 0 0 100 - - -Projects I am Primarily Investigating -===================================== -PID Resource type Login Usage ----------- -------------- ------- -------- -OPEN-XX-XX Barbora CPU user1 3 - Barbora CPU user2 2 - Karolina GPU N/A 0 - Legacy NCH N/A 0 - - -Legend -====== -N/A = No one used this resource yet -Legacy Normalized core hours are in NCH -Everything else is in Node Hours -``` - -## it4ifsusage - -### Help of IT4IFSUSAGE - -```console -$ it4ifsusage -h -usage: it4ifsusage [-h] - -The command shows filesystem usage of IT4I cluster storage systems - -optional arguments: - -h, --help show this help message and exit -``` - -### Example of IT4IFSUSAGE - -```console -$ it4ifsusage -Quota Type Cluster / PID File System Space used Space limit Entries used Entries limit Last update -------------- --------------- ------------- ------------ ------------- -------------- --------------- ------------------- -User barbora /home 2.9 GB 25.0 GB 183 500,000 2024-03-22 16:50:10 -User karolina /home 3.0 MB 25.0 GB 150 500,000 2024-03-22 17:00:07 -User barbora /scratch 0 Bytes 10.0 TB 0 10,000,000 2024-03-22 16:50:28 -User karolina /scratch 0 Bytes 0 Bytes 0 0 2024-03-22 17:00:43 -Project service proj3 1.5 TB 1.0 TB 169,933 198,000 2024-03-22 17:00:02 -``` - -## it4iuserfsusage - -### Help of IT4IUSERFSUSAGE - -```console -$ it4iuserfsusage -h -usage: it4iuserfsusage [-h] [-c {all,barbora, karolina}] - -The command shows user filesystem usage of IT4I cluster storage systems - -optional arguments: - -h, --help show this help message and exit -``` - -### Example of IT4IUSERFSUSAGE - -```console -$ it4iuserfsusage -Cluster File System Space used Space limit Entries used Entries limit Last update ---------------- ------------- ------------ ------------- -------------- --------------- ------------------- -barbora /home 2.9 GB 25.0 GB 183 500,000 2024-03-22 16:50:10 -karolina /home 3.0 MB 25.0 GB 150 500,000 2024-03-22 17:00:07 -barbora /scratch 0 Bytes 10.0 TB 0 10,000,000 2024-03-22 16:50:28 -karolina /scratch 0 Bytes 0 Bytes 0 0 2024-03-22 17:00:43 -``` - -## it4iprojectfsusage - -### Help of IT4IPROJECTFSUSAGE - -```console -$ it4iprojectfsusage -h -usage: it4iprojectfsusage [-h] [-p {PID, all}] - -The command shows project filesystem usage of IT4I cluster storage systems - -optional arguments: - -h, --help show this help message and exit -``` - -### Example of IT4IPROJECTFSUSAGE - -```console -$ it4iprojectfsusage -PID File System Space used Space limit Entries used Entries limit Last update ---------------- ------------- ------------ ------------- -------------- --------------- ------------------- -service proj3 3.1 GB 1.0 TB 5 100,000 2024-03-22 17:00:02 -it4i-x-y proj1 3.1 TB 2.0 TB 5 100,000 2024-03-22 17:00:02 -dd-13-5 proj3 2 GB 3.0 TB 5 100,000 2024-03-22 17:00:02 -projectx proj2 150 TB 4.0 TB 5 100,000 2024-03-22 17:00:02 -``` - -## it4imotd - -### Help of IT4IMOTD - -```console -$ it4imotd -h -usage: it4imotd [-h] [-t TEMPLATE] [-w WIDTH] [-c] - -The command shows IT4I messages of the day into formatted text or HTML page. - -optional arguments: - -h, --help show this help message and exit - -t TEMPLATE, --template TEMPLATE - path to TAL / Zope Page Template, output will be - formatted into HTML page - -w WIDTH, --width WIDTH - maximum line width (intended for text rendering, - default of 78 columns) - -c, --cron sleep from 10 up to 60 seconds prior to any actions - -m {TYPE}, --message {TYPE} - select type of messages - supported types: - all, - public-service-announcement, - service-recovered-up, - critical-service-down, - service-hard-down, - auxiliary-service-down, - planned-outage, - service-degraded, - important, - notice. -``` - -### Example of IT4IMOTD - -```console -$ it4imotd - - Message of the Day (DD/MM/YYYY) - (YYYY-MM-DD hh:mm:ss) - -More on https://... -``` diff --git a/mkdocs.yml b/mkdocs.yml index 8f1dfcb2c5d8634a22cb30439736174cd899aa84..973cbc706dcdd0f55d93c85459f4cc83a7a662d3 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -115,15 +115,15 @@ nav: - Tools: - CodeIT4I: general/tools/codeit4i.md - CI/CD: general/tools/cicd.md + - CLI Client Tools: general/tools/cli-client-tools.md - Data Sharing Tools: general/tools/tools-list.md - - IT4I Portal Client: general/tools/portal-clients.md - OpenCode: general/tools/opencode.md - Technical Information: - SSH Keys: - OpenSSH Keys (UNIX): general/accessing-the-clusters/shell-access-and-data-transfer/ssh-keys.md - PuTTY (Windows): general/accessing-the-clusters/shell-access-and-data-transfer/putty.md - Certificates FAQ: general/obtaining-login-credentials/certificates-faq.md - - API Documentation: apiv1.md + - API Documentation: apiv2.md - Satisfaction and Feedback: general/feedback.md - Energy Saving: general/energy.md - DICE: dice.md