Commit bd89c66c authored by David Hrbáč's avatar David Hrbáč

Merge branch 'master' into 'virtual_environment'

# Conflicts:
#   mkdocs.yml
parents be396d05 adc7f60e
# Introduction
Please, use **only** training account (DD-18-36-\*)
## Setting Up for Building
Set up an environment to use your own modules. ([EasyBuild#modulepath](https://docs.it4i.cz/software/tools/easybuild/#modulepath))
### Variant A
For temporary setup.
```console
module use $HOME/.local/easybuild/modules/all/
```
### Variant B
Modify your .bash_profile for permanent setup and reload environment (logout, login)
```console
cat ~/.bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
module use $HOME/.local/easybuild/modules/all/
PATH=$PATH:$HOME/bin
export PATH
```
# Creating a New Easyconfig From Scratch
Create easyconfig for `moon-buggy` software. The manual procedure is listed below.
Software: `moon-buggy`
Homepage: [https://github.com/seehuhn/moon-buggy](https://github.com/seehuhn/moon-buggy)
**Description**
Moon-buggy is a simple character graphics game, where you drive some kind of car across the moon's surface. Unfortunately there are dangerous craters there. Fortunately your car can jump over them!
## Manual Installation
```console
$ wget https://code.it4i.cz/kru0052/moon-buggy/-/archive/1.0/moon-buggy-1.0.tar.gz
$ tar xvf moon-buggy-1.0.tar.gz
$ cd moon-buggy-1.0
$ ./autogen.sh
$ ./configure --prefix=/home/kru0052/game/moon-buggy-build
$ make
$ ls
acinclude.m4 buggy.h config.h.in copying.h error.c highscore.o keyboard.c main.o meteor.o moon-buggy.info pager.o README terminal.c title.eps xmalloc.o
aclocal.m4 buggy.o config.log cursor.c error.o hpath.c keyboard.o Makefile missing moon-buggy.lsm persona.c realname.c terminal.o title.o xstrdup.c
ANNOUNCE car.img config.status cursor.o game.c hpath.o laser.c Makefile.am mode.c moon-buggy.png persona.o realname.o test-score-modes TODO xstrdup.o
AUTHORS ChangeLog config.sub darray.h game.o img.sed laser.o Makefile.in mode.o moon-buggy.texi queue.c signal.c texinfo.tex vclock.c
autogen.sh checklist configure date.c ground.c INSTALL level.c manpage.in moon-buggy moon-buggy.xpm queue.o signal.o text2c.sed vclock.o
autom4te.cache config.guess configure.ac date.o ground.o install-sh level.o mdate-sh moon-buggy.6 NEWS random.c stamp-h1 THANKS version.texi
buggy.c config.h COPYING depcomp highscore.c instcmds main.c meteor.c moon-buggy.h pager.c random.o stamp-vti title.c xmalloc.c
$ ./moon-buggy
```
## Create Easyconfog From Template
* **Task**: *create easyconfig for `moon-buggy` (use template)* ([Download template.eb](template.eb))
```console
$ cp template.eb moon-buggy-1.0.eb
```
* **Task**: *EASYBLOCK* ... choose easyblock (Analyse manual instalation - only step CONFIGURE AND MAKE -> choose easyblock `ConfigureMake`)
```python
easyblock = 'ConfigureMake'
```
* **Task**: *NAME* ... defined name of the software
```python
name = 'moon-buggy'
```
* **Task**: *VERSION* ... defined versions of the software
```python
version = "1.0"
```
* **Task**: *VERSIONSUFFIX* ... add your login
```python
versionsuffix = "-kru0052"
```
* **Task**: *HOMEPAGE* ... homepage url
```python
homepage = 'https://github.com/seehuhn/moon-buggy'
```
* **Task**: *DESCRIPTION* ... basic software information
```python
description = """Moon-buggy is a simple character graphics game, where you drive some
kind of car across the moon's surface. Unfortunately there are
dangerous craters there. Fortunately your car can jump over them!"""
```
* **Task**: *TOOLCHAIN* ... choose toolchain
```python
toolchain = {'name': 'dummy', 'version': ''}
```
* **Task**: *SOURCE_URLS* ... source urls
```python
source_urls = ['https://code.it4i.cz/kru0052/moon-buggy/-/archive/%(version)s/']
```
* **Task**: *SOURCES* ... package name definition
```python
sources = ['%(name)s-%(version)s.tar.gz']
```
* **Task**: *PRECONFIGOPTS* ... autogen.sh
```python
preconfigopts = "./autogen.sh && "
```
* **Task**: *BUILDDEPENDENCY*
```python
('Autoconf', '2.69')
```
* **Task**: *DEPENDENCY*
```python
('ncurses', '6.1'),
```
* **Task**: *SANITY_CHECK_PATH* ... you must check exists binary file
```python
sanity_check_paths = {
'files': ['bin/moon-buggy', 'com/moon-buggy/mbscore'],
'dirs': ['bin', 'com', 'share'],
}
```
* **Task**: *MODULECLASS* ... choose class
```python
moduleclass = 'tools'
```
* **Task**: *install `moon-buggy` from easyconfig*
```console
$ eb moon-buggy-1.0.eb -r
== temporary log file in case of crash /tmp/eb-ctAvZY/easybuild-GQkRPM.log
== resolving dependencies ...
== processing EasyBuild easyconfig /home/kru0052/game/moon-buggy-1.0.eb
== building and installing moon-buggy/1.0-kru0052...
== fetching files...
...
...
== COMPLETED: Installation ended successfully
== Results of the build can be found in the log file(s) /home/kru0052/.local/easybuild/software/moon-buggy/1.0-kru0052/easybuild/easybuild-moon-buggy-1.0-20181016.094918.log
== Build succeeded for 1 out of 1
== Temporary log file(s) /tmp/eb-ctAvZY/easybuild-GQkRPM.log* have been removed.
== Temporary directory /tmp/eb-ctAvZY has been removed.
```
* **Task**: *load module and run `moon-buggy`*
```console
$ ml moon-buggy/1.0-kru0052
$ moon-buggy
```
# Building Your First Software Using Easyconfigs
Install git version 2.19.0 from ready-made easyconfig.
* **task**: *install git/2.19.0* ([Download git-2.19.0.eb](git-2.19.0.eb))
## Load Module EasyBuild
The effectively apply the changes to the environment that are specified by a module, use `ml` and specify the name of the module. For example, to set up your environment to use `ml EasyBuild`.
To get an overview of the currently loaded modules, use module list or ml (without specifying extra arguments).
```console
$ ml EasyBuild
$ ml
Currently Loaded Modules:
1) EasyBuild/3.7.1 (S)
Where:
S: Module is Sticky, requires --force to unload or purge
```
## Show Installed Dependencies
You can do a `dry-run` overview by supplying `-D/--dry-run` (typically combined with `--robot`, in the form of `-Dr`).
```console
$ eb git-2.19.0.eb -Dr
== temporary log file in case of crash /tmp/eb-xk3mg4/easybuild-f90lIR.log
Dry run: printing build status of easyconfigs and dependencies
CFGS=/apps/easybuild/easyconfigs-it4i
* [x] $CFGS/m/M4/M4-1.4.18.eb (module: M4/1.4.18)
* [ ] $CFGS/a/Autoconf/Autoconf-2.68.eb (module: Autoconf/2.68)
* [ ] $CFGS/g/git/git-2.19.0.eb (module: git/2.19.0)
== Temporary log file(s) /tmp/eb-xk3mg4/easybuild-f90lIR.log* have been removed.
== Temporary directory /tmp/eb-xk3mg4 has been removed.
```
## Install Module and All Dependencies
If we try to build git-2.19.0.eb. To enable dependency resolution, use the `--robot` command line option (or `-r` for short).
```console
$ eb git-2.19.0.eb -r
== temporary log file in case of crash /tmp/eb-WEtJ8t/easybuild-dvHmbd.log
== resolving dependencies ...
== processing EasyBuild easyconfig /apps/easybuild/easyconfigs-it4i/a/Autoconf/Autoconf-2.68.eb
== building and installing Autoconf/2.68...
...
...
== COMPLETED: Installation ended successfully
== Results of the build can be found in the log file(s) /apps/all/Autoconf/2.68/easybuild/easybuild-Autoconf-2.68-20181016.085630.log
== processing EasyBuild easyconfig /apps/easybuild/easyconfigs-it4i/g/git/git-2.19.0.eb
== building and installing git/2.19.0...
== fetching files...
...
...
== COMPLETED: Installation ended successfully
== Results of the build can be found in the log file(s) /apps/all/git/2.19.0/easybuild/easybuild-git-2.19.0-20181016.085647.log
== Build succeeded for 2 out of 2
== Temporary log file(s) /tmp/eb-WEtJ8t/easybuild-dvHmbd.log* have been removed.
== Temporary directory /tmp/eb-WEtJ8t has been removed.
```
## Test Installed Module
```console
$ ml git/2.19.0
$ git --version
git version 2.19.0
```
# IT4Innovations 2018
easyblock = 'ConfigureMake'
name = 'git'
version = "2.19.0"
homepage = 'http://git-scm.com/'
description = """Git is a free and open source distributed version control system designed
to handle everything from small to very large projects with speed and efficiency."""
toolchain = {'name': 'dummy', 'version': ''}
source_urls = ['https://www.kernel.org/pub/software/scm/git/']
sources = ['%(name)s-%(version)s.tar.gz']
builddependencies = [
('Autoconf', '2.68', '', True)
]
preconfigopts = 'make configure && '
moduleclass = 'tools'
# IT4Innovations 2018
easyblock = 'ConfigureMake'
name = 'git'
version = "2.19.1"
versionsuffix = "-Py-3.6-foxik"
homepage = 'http://git-scm.com/'
description = """Git is a free and open source distributed version control system designed
to handle everything from small to very large projects with speed and efficiency."""
toolchain = {'name': 'dummy', 'version': ''}
source_urls = ['https://www.kernel.org/pub/software/scm/git/']
sources = ['%(name)s-%(version)s.tar.gz']
builddependencies = [
('Autoconf', '2.69', '', True)
]
dependencies = [
('zlib', '1.2.11', '', True)
]
preconfigopts = 'make configure && '
sanity_check_paths = {
'files': ['bin/git'],
'dirs': ['bin', 'libexec', 'share'],
}
sanity_check_commands = [
('git', '--version')
]
moduleclass = 'tools'
# Modification of an Existing Easyconfig
The goal is to modify the easyconfig for git/2.19.1 so that it meets the requirements below.
## Change Name Easyconfig
* **Task**: *change name* (name-version-Py-version-login)
```console
$ cp git-2.19.0.eb git-2.19.1-Py-3.6-kru0052.eb
```
## Change Software Version
* **Task**: *change git version from 2.19.0 to 2.19.1*
```python
version = "2.19.1"
```
## Add Module Suffix
* **Task**: *add module suffix (all module name git/2.19.1-Py-version-login)*
```python
versionsuffix = "-Py-3.6-kru0052"
```
## Add Dependencies
* **Task**: *cURL/7.56.1 (dummy)*, *expat/2.2.0 (dummy)*, *add Py/3.6 dependency (dummy)*
```python
dependencies = [
('cURL', '7.56.1', '', True),
('expat', '2.2.0', '', True),
('Py', '3.6', '', True)
]
```
## Add Sanity Check Paths
* **Task**: *sanity_check_paths*
```python
sanity_check_paths = {
'files': ['bin/git'],
'dirs': ['bin', 'libexec', 'share'],
}
```
## Add Sanity Check Commands
* **Task**: *sanity_check_commands*
```python
sanity_check_commands = [
('git', '--version')
]
```
## Install Module
```console
$ eb git-2.19.1.eb -r
Couldn't import dot_parser, loading of dot files will not be possible.
== temporary log file in case of crash /tmp/eb-oYDvfO/easybuild-zCM5rM.log
== resolving dependencies ...
== processing EasyBuild easyconfig /home/kru0052/game/git-2.19.1.eb
== building and installing git/2.19.1-Py-3.6-kru0052...
...
...
== COMPLETED: Installation ended successfully
== Results of the build can be found in the log file(s) /home/kru0052/.local/easybuild/software/git/2.19.1-Py-3.6-foxik/easybuild/easybuild-git-2.19.1-20181016.092517.log
== Build succeeded for 1 out of 1
== Temporary log file(s) /tmp/eb-oYDvfO/easybuild-zCM5rM.log* have been removed.
== Temporary directory /tmp/eb-oYDvfO has been removed.
```
## Test New Module
```console
$ ml av git/
-------------------- /home/kru0052/.local/easybuild/modules/all --------------------
git/2.19.0 git/2.19.1-Py-3.6-kru0052 (D)
-------------------- /apps/modules/tools --------------------
git/2.18.0
Where:
D: Default Module
If you need software that is not listed, request it at support@it4i.cz.
$ ml git/2.19.1-Py-3.6-foxik
$ git --version
git version 2.19.1
```
# IT4Innovations 2018
easyblock =
name =
version =
versionsuffix =
homepage =
description = """ """
toolchain = {'name': '', 'version': ''}
source_urls = ['']
sources = ['']
preconfigopts =
builddependencies = []
dependencies = []
sanity_check_paths = {
'files': [],
'dirs': [],
}
moduleclass = ''
# Troubleshooting - Typical Problems
## INSTALL - Already Installed
Install module from easyconfig `git-2.15.0-GCC-7.1.0-2.28-Python-3.6.1.eb`
```console
[kru0052@login2.anselm test-build]$ eb git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb
== temporary log file in case of crash /tmp/eb-KMNHse/easybuild-Jwn_R7.log
== git/2.14.1-GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK is already installed (module found), skipping
== No easyconfigs left to be built.
== Build succeeded for 0 out of 0
== Temporary log file(s) /tmp/eb-KMNHse/easybuild-Jwn_R7.log* have been removed.
== Temporary directory /tmp/eb-KMNHse has been removed.
```
## REINSTALL - Module Already Loaded
Load module `git/2.14.1-GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK` and reinstall module
```console
[kru0052@login2.anselm test-build]$ eb git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb -f
== temporary log file in case of crash /tmp/eb-NDp_Rx/easybuild-XgYOq9.log
WARNING: Found one or more non-allowed loaded (EasyBuild-generated) modules in current environment:
* bzip2/1.0.6
* libreadline/6.3
* SQLite/3.13.0
* Tcl/8.6.5
* Tk/8.6.5
* GMP/6.1.1
* XZ/5.2.2
* zlib/1.2.11
* Python/3.6.1
* cURL/7.53.1
* expat/2.2.0
* ncurses/6.0
* gettext/0.19.8.1
* GCCcore/7.1.0
* GCCcore/7.1.0
* binutils/2.28-GCCcore-7.1.0
* Perl/5.26.0-GCC-7.1.0-2.28-bare
* git/2.14.1-GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK
This is not recommended since it may affect the installation procedure(s) performed by EasyBuild.
To make EasyBuild allow particular loaded modules, use the --allow-loaded-modules configuration option.
To specify action to take when loaded modules are detected, use --detect-loaded-modules={error,ignore,purge,unload,warn}.
See http://easybuild.readthedocs.io/en/latest/Detecting_loaded_modules.html for more information.
== processing EasyBuild easyconfig /home_lustre/kru0052/hands-on/git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb
== building and installing git/2.14.1-GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK...
== fetching files...
== creating build dir, resetting environment...
== FAILED: Installation ended unsuccessfully (build directory: /home/kru0052/.local/easybuild/build/git/2.14.1/GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK): build failed (first 300 chars): Module is already loaded (EBROOTGIT is set), installation cannot continue.
== Results of the build can be found in the log file(s) /tmp/eb-NDp_Rx/easybuild-git-2.14.1-20170911.085415.EaWTJ.log
ERROR: Build of /home_lustre/kru0052/test-build/git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb failed (err: 'build failed (first 300 chars): Module is already loaded (EBROOTGIT is set), installation cannot continue.')
```
## INSTALL/REINSTALL - Irresolvable Dependencies
Change `('Python', '3.6.1', '', True)` to `('Python', '3.6.1')` and reinstall module
```console
[kru0052@login2.anselm ~]$ eb git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb -r
== temporary log file in case of crash /tmp/eb-7toWyv/easybuild-koZYcG.log
== processing EasyBuild easyconfig /home_lustre/kru0052/hands-on/git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb
== building and installing git/2.14.1-GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK...
== fetching files...
== creating build dir, resetting environment...
== FAILED: Installation ended unsuccessfully (build directory: /home/kru0052/.local/easybuild/build/git/2.14.1/GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK): build failed (first 300 chars): Missing modules for one or more dependencies: Python/3.6.1-GCC-7.1.0-2.28
== Results of the build can be found in the log file(s) /tmp/eb-7toWyv/easybuild-git-2.14.1-20170911.084653.qAHAA.log
ERROR: Build of /home_lustre/kru0052/test-build/git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb failed (err: 'build failed (first 300 chars): Missing modules for one or more dependencies: Python/3.6.1-GCC-7.1.0-2.28')
```
## INSTALL/REINSTALL - Can't Find Path
Use bad easyconfig name `eb git-2.14.1-GCC-7.1.0-2.28.eb`
```console
[kru0052@login2.anselm test-build]$ eb git-2.14.1-GCC-7.1.0-2.28.eb -f
== temporary log file in case of crash /tmp/eb-keyVcY/easybuild-AgLBmT.log
ERROR: Can't find path /home_lustre/kru0052/test-build/git-2.14.1-GCC-7.1.0-2.28.eb
```
## INSTALL/REINSTALL - Sanity Check Failed
Change `'files': ['bin/git']` to `'files': ['bin/git2']`
```console
[kru0052@login2.anselm test-build]$ eb git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb -f
== temporary log file in case of crash /tmp/eb-Lv9b1_/easybuild-h2F2uL.log
== processing EasyBuild easyconfig /home_lustre/kru0052/test-build/git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb
== building and installing git/2.14.1-GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK...
== fetching files...
== creating build dir, resetting environment...
== unpacking...
== patching...
== preparing...
== configuring...
== building...
== testing...
== installing...
== taking care of extensions...
== postprocessing...
== sanity checking...
== FAILED: Installation ended unsuccessfully (build directory: /home/kru0052/.local/easybuild/build/git/2.14.1/GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK): build failed (first 300 chars): Sanity check failed: no file of ('bin/git2',) in /home/kru0052/.local/easybuild/software/git/2.14.1-GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK
== Results of the build can be found in the log file(s) /tmp/eb-Lv9b1_/easybuild-git-2.14.1-20170911.085750.VJyvg.log
ERROR: Build of /home_lustre/kru0052/test-build/git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb failed (err: "build failed (first 300 chars): Sanity check failed: no file of ('bin/git2',) in /home/kru0052/.local/easybuild/software/git/2.14.1-GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK")
```
## INSTALL/REINSTALL - Couldn't Find File
Change `sources = ['%(name)s-%(version)s.tar.gz']` to `sources = ['%(name)s-%(version)s-test.tar.gz']`
```console
[kru0052@login2.anselm test-build]$ eb git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb -f
== temporary log file in case of crash /tmp/eb-N64arw/easybuild-lXjC9B.log
== processing EasyBuild easyconfig /home_lustre/kru0052/hands-one/git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb
== building and installing git/2.14.1-GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK...
== fetching files...
== FAILED: Installation ended unsuccessfully (build directory: /home/kru0052/.local/easybuild/build/git/2.14.1/GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK): build failed (first 300 chars): Couldn't find file git-2.14.1-test.tar.gz anywhere, and downloading it didn't work either... Paths attempted (in order): /home_lustre/kru0052/hands-on/g/git/git-2.14.1-test.tar.gz, /home_lustre/kru0052/hands-on/git/git-2.14.1-test.tar.gz, /home_lustre/kru0052/hands-on/git-2.14.1-test.tar.gz, /
== Results of the build can be found in the log file(s) /tmp/eb-N64arw/easybuild-git-2.14.1-20170911.085547.BnfZG.log
ERROR: Build of /home_lustre/kru0052/hands-on/git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb failed (err: "build failed (first 300 chars): Couldn't find file git-2.14.1-test.tar.gz anywhere, and downloading it didn't work either... Paths attempted (in order): /home_lustre/kru0052/hands-on/g/git/git-2.14.1-test.tar.gz, /home_lustre/kru0052/test-build/git/git-2.14.1-test.tar.gz, /home_lustre/kru0052/hands-on/git-2.14.1-test.tar.gz, /")
```
......@@ -373,7 +373,7 @@ exit
In this example, input and executable files are assumed to be preloaded manually in the /scratch/$USER/myjob directory. Note the **mpiprocs** and **ompthreads** qsub options controlling the behavior of the MPI execution. mympiprog.x is executed as one process per node, on all 100 allocated nodes. If mympiprog.x implements OpenMP threads, it will run 16 threads per node.
More information can be found in the [Running OpenMPI](software/mpi/Running_OpenMPI/) and [Running MPICH2](software/mpi/running-mpich2/)
More information can be found in the [Running OpenMPI](../software/mpi/Running_OpenMPI/) and [Running MPICH2](../software/mpi/running-mpich2/)
sections.
### Example Jobscript for Single Node Calculation
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment