diff --git a/.spelling b/.spelling index b3d535514878bf593dc1b415ac6e466b3a893f20..f703974689746353c5a57b774d9deaa66e8d801b 100644 --- a/.spelling +++ b/.spelling @@ -1,3 +1,7 @@ +AWS +CLI +FAQ +s3cmd GUI EESSI hipBlas diff --git a/docs.it4i/storage/awscli.md b/docs.it4i/storage/awscli.md new file mode 100644 index 0000000000000000000000000000000000000000..bed43703fd22105db86291331ba555791ffeacff --- /dev/null +++ b/docs.it4i/storage/awscli.md @@ -0,0 +1,52 @@ +# AWS CLI + +You can use the AWS command line tools to issue commands or build scripts at your system's command line to perform AWS (including S3) tasks. +The AWS Command Line Interface (AWS CLI) provides commands for a broad set of AWS services. To get started, see the AWS Command Line Interface User Guide. + +For more information, see the official site [https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#BasicsBucket][a]. + + +## How to Load Modules: + +To see loaded modules, use: + +``` +ml +``` + +To load modules, use: + +``` +ml awscli +``` + +To see module info, use: + +``` +ml spider AWS +``` + +For more information on how to work with modules, see +[https://docs.it4i.cz/software/modules/lmod/][1]. + + +## Configuration + +For configuration steps, follow the [e-INFRA CZ guide][b]. + +For the list of basic AWS CLI commands, see [https://docs.e-infra.cz/storage/object-storage/aws-cli/#controls-of-aws-cli---high-level-s3][e]. + +## Useful Links + +* [Getting started][c] +* [Troubleshoot AWS CLI errors][f] +* [Configuration erros][d] + +[1]: https://docs.it4i.cz/software/modules/lmod/ + +[a]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#BasicsBucket +[b]: https://docs.e-infra.cz/storage/object-storage/aws-cli/ +[c]: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html +[d]: https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html +[e]: https://docs.e-infra.cz/storage/object-storage/aws-cli/#controls-of-aws-cli---high-level-s3 +[f]: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-troubleshooting.html diff --git a/docs.it4i/storage/cesnet-s3.md b/docs.it4i/storage/cesnet-s3.md new file mode 100644 index 0000000000000000000000000000000000000000..e6de8b12e934cf606589342ab13f83e58da82448 --- /dev/null +++ b/docs.it4i/storage/cesnet-s3.md @@ -0,0 +1,52 @@ +# CESNET S3 Storage + +S3 service is a general service suited for most of the use cases. S3 service can be used for elementary data storing, automated backups, or various types of data handling applications. + +For more information, see [e-IINFRA CZ S3 Service documentation][a]. + +## Access: + +To access to S3 you must: + +* have an [e-Infra CZ account][3] +* be a [member of an active project][4] + +### Steps: + +1. Fil in application at [https://einfra.cesnet.cz/fed/registrar/?vo=storage][b] to be a member of VO group + * Provide information about the reservation capacity in TB/GB in the **Description of planned activity** field. + +2. Wait for a confirmation email with information about the endpoint (e.g. cl2) and link to [Filesender][e] where your key is stored. + + !!! important "Link Expiration" + Note that the link has an expiration date, so download the information to your local storage. + + **Key Format:** + + ``` + "user": "51de0172a4***************************bc3cf1b211aacc9d5dc7876649a8", + "access_key": "IL2***********JAS", + "secret_key": "WIt******************HRAYvBtHmD" + ``` + Use this information for management tool configuration. + +IT4I offers two tools for object storage management on Karolina: + +* [S3cmd][1] +* [AWS CLI][2] + +## Walkthroughs (Config Connection) + +[https://du.cesnet.cz/en/navody/object_storage/cesnet_s3/start][c] +[https://www.s3express.com/kb/item26.htm][d] + +[1]: s3cmd.md +[2]: awscli.md +[3]: https://docs.e-infra.cz/account/ +[4]: https://docs.it4i.cz/general/access/project-access/ + +[a]: https://docs.e-infra.cz/storage/object-storage/s3-service/ +[b]: https://einfra.cesnet.cz/fed/registrar/?vo=storage +[c]: https://du.cesnet.cz/cs/navody/object_storage/cesnet_s3/start +[d]: https://www.s3express.com/kb/item26.htm +[e]: https://filesender.cesnet.cz/ diff --git a/docs.it4i/storage/s3cmd.md b/docs.it4i/storage/s3cmd.md new file mode 100644 index 0000000000000000000000000000000000000000..dd1a3a7a6d5a1e8a54fa37e59e4f5aa36b4cc241 --- /dev/null +++ b/docs.it4i/storage/s3cmd.md @@ -0,0 +1,74 @@ +# s3cmd + +S3cmd is a command-line tool for managing data in Amazon Simple Storage Service (S3). It is a free and open-source tool that is available for Linux, macOS, and Windows. + +For more information, see [https://docs.e-infra.cz/storage/object-storage/s3cmd/][a] + +We recommend using the S3cmd tool already installed on Karolina. + +## How to Load Modules: + +To see loaded modules, use: + +``` +ml +``` + +To load modules, use: + +``` +ml s3cmd +``` + +To see module info, use: + +``` +ml spider s3cmd +``` + +For more information on how to work with modules, see +[https://docs.it4i.cz/software/modules/lmod/][1]. + +## Configuration + +The s3cmd configuration file is named `.s3cfg` and it is located in the user's home directory, e.g. `/home/username/($HOME)`. + +The configuration file has to be created first: + +``` +s3cmd --configure +``` + +For configuration steps, follow the [e-INFRA CZ guide][b]. + +For the list of basic S3cmd commands, see [https://docs.e-infra.cz/storage/object-storage/s3cmd/#basic-s3cmd-commands][c]. + +## FAQ + +Error when creating bucket: + +``` +ERROR: S3 error: 400 (InvalidLocationConstraint): The specified location-constraint is not valid +``` +Solution: set location: + +``` +s3cmd --bucket-location=":default-placement" mb s3://newbucketname +``` + +## Useful Links + +* [Introduction][d] +* [Usage][e] +* [Common S3cmd Errors][g] +* [Configuration erros][f] + +[1]: https://docs.it4i.cz/software/modules/lmod/ + +[a]: https://docs.e-infra.cz/storage/object-storage/s3cmd/ +[b]: https://docs.e-infra.cz/storage/object-storage/s3cmd/#configuration-of-s3cmd-tool +[c]: https://docs.e-infra.cz/storage/object-storage/s3cmd/#basic-s3cmd-commands +[d]: https://s3tools.org/s3cmd +[e]: https://s3tools.org/usage +[f]: https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html +[g]: https://s3tools.org/kb/category1.htm diff --git a/mkdocs.yml b/mkdocs.yml index d1ee8964ea0de9dbe4c3f8c04d2171bfce0041eb..2fbec8dd0a02749a37a068f4b6df20e8e5de1066 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -128,7 +128,12 @@ nav: - PROJECT: - Project storage: storage/project-storage.md - Proj4 storage - S3: storage/proj4-storage.md - - CESNET: storage/cesnet-storage.md + - CESNET: + - CESNET: storage/cesnet-storage.md + - S3: + - Introduction: storage/cesnet-s3.md + - S3cmd: storage/s3cmd.md + - AWS CLI: storage/awscli.md - Access Control List: - Standard File ACL: storage/standard-file-acl.md - NFSv4 File ACL: storage/nfs4-file-acl.md