2018 EB

EasyBuild/EasyBuild-Hands-on-2018/Introduction.md

+# Introduction
+
+Please, use **only** training account (DD-13-22-*) 
+
+## Setting Up /home For Building
+
+Docs: https://docs.it4i.cz/software/easybuild/
+
+### Variant A
+
+```console
+module use $HOME/.local/easybuild/modules/all/
+```
+
+### Variant B
+
+Modify your .bash_profile and reload enviroment (logout, login)
+
+```console
+cat ~/.bash_profile
+# .bash_profile
+
+# Get the aliases and functions
+if [ -f ~/.bashrc ]; then
+. ~/.bashrc
+fi
+
+# User specific environment and startup programs
+module use $HOME/.local/easybuild/modules/all/
+
+PATH=$PATH:$HOME/bin
+export PATH
+```

EasyBuild/EasyBuild-Hands-on-2018/L01/L01.md

+# L01 - Building Your First Software Using Easyconfigs
+
+Install git version 2.19.0 from ready-made easyconfig.
+
+ * [ ] install git/2.19.0
+
+## Load Module EasyBuild
+
+```console
+$ ml EasyBuild
+$ ml
+
+Currently Loaded Modules:
+  1) EasyBuild/3.7.0 (S)
+
+  Where:
+   S:  Module is Sticky, requires --force to unload or purge
+```
+
+## Show Installed Dependencies
+
+```console
+$ eb git-2.19.1.eb -Dr
+== temporary log file in case of crash /tmp/eb-xk3mg4/easybuild-f90lIR.log
+Dry run: printing build status of easyconfigs and dependencies
+CFGS=/apps/easybuild/easyconfigs-it4i
+ * [x] $CFGS/m/M4/M4-1.4.18.eb (module: M4/1.4.18)
+ * [ ] $CFGS/a/Autoconf/Autoconf-2.68.eb (module: Autoconf/2.68)
+ * [ ] $CFGS/g/git/git-2.19.0.eb (module: git/2.19.0)
+== Temporary log file(s) /tmp/eb-xk3mg4/easybuild-f90lIR.log* have been removed.
+== Temporary directory /tmp/eb-xk3mg4 has been removed.
+```
+
+## Install Module and All Dependencies
+
+```console
+$ eb git-2.19.0.eb -r
+== temporary log file in case of crash /tmp/eb-WEtJ8t/easybuild-dvHmbd.log
+== resolving dependencies ...
+== processing EasyBuild easyconfig /apps/easybuild/easyconfigs-it4i/a/Autoconf/Autoconf-2.68.eb
+== building and installing Autoconf/2.68...
+...
+...
+== COMPLETED: Installation ended successfully
+== Results of the build can be found in the log file(s) /apps/all/Autoconf/2.68/easybuild/easybuild-Autoconf-2.68-20181016.085630.log
+== processing EasyBuild easyconfig /apps/easybuild/easyconfigs-it4i/g/git/git-2.19.0.eb
+== building and installing git/2.19.0...
+== fetching files...
+...
+...
+== COMPLETED: Installation ended successfully
+== Results of the build can be found in the log file(s) /apps/all/git/2.19.0/easybuild/easybuild-git-2.19.0-20181016.085647.log
+== Build succeeded for 2 out of 2
+== Temporary log file(s) /tmp/eb-WEtJ8t/easybuild-dvHmbd.log* have been removed.
+== Temporary directory /tmp/eb-WEtJ8t has been removed.
+```
+
+## Test Installed Module
+
+```console
+$ ml git/2.19.0
+$ git --version
+git version 2.19.0
+```

EasyBuild/EasyBuild-Hands-on-2018/L01/git-2.19.0.eb

+# IT4Innovations 2018
+
+easyblock = 'ConfigureMake'
+
+name = 'git'
+version = "2.19.0"
+
+homepage = 'http://git-scm.com/'
+description = """Git is a free and open source distributed version control system designed
+to handle everything from small to very large projects with speed and efficiency."""
+
+toolchain = {'name': 'dummy', 'version': ''}
+
+source_urls = ['https://www.kernel.org/pub/software/scm/git/']
+sources = ['%(name)s-%(version)s.tar.gz']
+
+builddependencies = [
+    ('Autoconf', '2.68', '', True)
+]
+
+preconfigopts = 'make configure && '
+
+moduleclass = 'tools'

EasyBuild/EasyBuild-Hands-on-2018/L02/L02.md

+# L02 - Modification of an Existing 
+
+The goal is to modify the easyconfig for git/2.19.1 so that it meets the requirements below.
+
+## Change Name Easyconfig
+
+* [ ] change name (name-version-Py-version-login)
+
+```console
+$ cp git-2.19.0.eb git-2.19.1-Py-3.6-kru0052.eb
+```
+
+## Change Software Version
+
+ * [ ] change git version from 2.19.0 to 2.19.1
+
+```python
+version = "2.19.1"
+```
+
+## Add Module Suffix
+
+ * [ ] add module suffix (all module name git/2.19.1-Py-version-login)
+
+```python
+versionsuffix = "-Py-3.6-kru0052"
+```
+
+## Add Dependency
+
+ * [ ] cURL/7.56.1 (dummy)
+ * [ ] expat/2.2.0 (dummy)
+ * [ ] add Py/3.6 dependency (dummy)
+
+```python
+dependencies = [
+    ('cURL', '7.56.1', '', True),
+    ('expat', '2.2.0', '', True),
+    ('Py', '3.6', '', True)
+]
+```
+
+## Add Sanity Check Paths
+
+ * [ ] sanity_check_paths
+
+```python
+sanity_check_paths = {
+    'files': ['bin/git'],
+    'dirs': ['bin', 'libexec', 'share'],
+}
+
+```
+
+## Add Sanity Check Commands
+
+ * [ ] sanity_check_commands
+
+```python
+sanity_check_commands = [
+    True,
+    ('git', '-- version')
+]
+```
+
+## Install Module
+
+```console
+$ eb git-2.19.1.eb -r
+Couldn't import dot_parser, loading of dot files will not be possible.
+== temporary log file in case of crash /tmp/eb-oYDvfO/easybuild-zCM5rM.log
+== resolving dependencies ...
+== processing EasyBuild easyconfig /home/kru0052/game/git-2.19.1.eb
+== building and installing git/2.19.1-Py-3.6-kru0052...
+...
+...
+== COMPLETED: Installation ended successfully
+== Results of the build can be found in the log file(s) /home/kru0052/.local/easybuild/software/git/2.19.1-Py-3.6-foxik/easybuild/easybuild-git-2.19.1-20181016.092517.log
+== Build succeeded for 1 out of 1
+== Temporary log file(s) /tmp/eb-oYDvfO/easybuild-zCM5rM.log* have been removed.
+== Temporary directory /tmp/eb-oYDvfO has been removed.
+```
+
+## Test New Module
+
+``$ ml av git/
+
+-------------------- /home/kru0052/.local/easybuild/modules/all --------------------
+   git/2.19.0    git/2.19.1-Py-3.6-foxik (D)
+
+-------------------- /apps/modules/tools --------------------
+   git/2.18.0
+
+  Where:
+   D:  Default Module
+
+  If you need software that is not listed, request it at support@it4i.cz.
+
+$ ml git/2.19.1-Py-3.6-foxik
+$ git --version
+git version 2.19.1
+```

EasyBuild/EasyBuild-Hands-on-2018/L02/git-2.19.0.eb

+# IT4Innovations 2018
+
+easyblock = 'ConfigureMake'
+
+name = 'git'
+version = "2.19.0"
+
+homepage = 'http://git-scm.com/'
+description = """Git is a free and open source distributed version control system designed
+to handle everything from small to very large projects with speed and efficiency."""
+
+toolchain = {'name': 'dummy', 'version': ''}
+
+source_urls = ['https://www.kernel.org/pub/software/scm/git/']
+sources = ['%(name)s-%(version)s.tar.gz']
+
+builddependencies = [
+    ('Autoconf', '2.68', '', True)
+]
+
+preconfigopts = 'make configure && '
+
+moduleclass = 'tools'

EasyBuild/EasyBuild-Hands-on-2018/L03/L03.md

+# Creating a New Easyconfig From Scratch
+
+Create easyconfig for `moon-buggy` software. The manual procedure is listed below.
+
+Software:     moon-buggy
+Homepage:     'https://github.com/seehuhn/moon-buggy'
+
+Moon-buggy is a simple character graphics game, where you drive some kind of car across the moon's surface. Unfortunately there are dangerous craters there. Fortunately your car can jump over them!
+
+## Manual installation
+
+```console
+$ wget https://code.it4i.cz/kru0052/moon-buggy/-/archive/1.0/moon-buggy-1.0.tar.gz
+$ tar xvf moon-buggy-1.0.tar.gz
+$ cd moon-buggy-1.0
+$ ./autogen.sh
+$ ./configure --prefix=/home/kru0052/game/moon-buggy-build
+$ make
+$ ls
+acinclude.m4    buggy.h       config.h.in    copying.h  error.c      highscore.o  keyboard.c  main.o       meteor.o      moon-buggy.info  pager.o    README      terminal.c        title.eps     xmalloc.o
+aclocal.m4      buggy.o       config.log     cursor.c   error.o      hpath.c      keyboard.o  Makefile     missing       moon-buggy.lsm   persona.c  realname.c  terminal.o        title.o       xstrdup.c
+ANNOUNCE        car.img       config.status  cursor.o   game.c       hpath.o      laser.c     Makefile.am  mode.c        moon-buggy.png   persona.o  realname.o  test-score-modes  TODO          xstrdup.o
+AUTHORS         ChangeLog     config.sub     darray.h   game.o       img.sed      laser.o     Makefile.in  mode.o        moon-buggy.texi  queue.c    signal.c    texinfo.tex       vclock.c
+autogen.sh      checklist     configure      date.c     ground.c     INSTALL      level.c     manpage.in   moon-buggy    moon-buggy.xpm   queue.o    signal.o    text2c.sed        vclock.o
+autom4te.cache  config.guess  configure.ac   date.o     ground.o     install-sh   level.o     mdate-sh     moon-buggy.6  NEWS             random.c   stamp-h1    THANKS            version.texi
+buggy.c         config.h      COPYING        depcomp    highscore.c  instcmds     main.c      meteor.c     moon-buggy.h  pager.c          random.o   stamp-vti   title.c           xmalloc.c
+$ ./moon-buggy
+```
+
+## Create Easyconfog From Template
+
+ * [ ] create easyconfig for `moon-buggy` (use template)
+
+```console
+$ cp template.eb moon-buggy-1.0.eb
+```
+
+ * [ ] EASYBLOCK ... choose easyblock (Analyse manual instalation - only step CONFIGURE AND MAKE -> choose easyblock `ConfigureMake`)
+
+```python
+easyblock = 'ConfigureMake'
+```
+
+ * [ ] NAME ... defined name of the software
+
+```python
+name = 'moon-buggy'
+```
+
+ * [ ] VERSION ... defined versions of the software
+
+```python
+version = "1.0"
+```
+
+ * [ ] VERSIONSUFFIX ... add your login
+
+```python
+versionsuffix = "-kru0052"
+```
+
+ * [ ] HOMEPAGE ... homepage url
+
+```python
+homepage = 'https://github.com/seehuhn/moon-buggy'
+```
+
+ * [ ] DESCRIPTION ... basic software information
+
+```python
+description = """Moon-buggy is a simple character graphics game, where you drive some
+kind of car across the moon's surface.  Unfortunately there are
+dangerous craters there.  Fortunately your car can jump over them!"""
+```
+
+ * [ ] TOOLCHAIN ... choose toolchain
+
+```python
+toolchain = {'name': 'dummy', 'version': ''}
+```
+
+ * [ ] SOURCE_URLS ... source urls
+
+```python
+source_urls = ['https://code.it4i.cz/kru0052/moon-buggy/-/archive/%(version)s/']
+```
+
+ * [ ] SOURCES ... package name definition
+
+```python
+sources = ['%(name)s-%(version)s.tar.gz']
+```
+
+ * [ ] PRECONFIGOPTS ... autogen.sh
+
+```python
+preconfigopts = "./autogen.sh && "
+```
+
+ * [ ] BUILDDEPENDENCY
+
+```python
+('Autoconf', '2.69')
+```
+
+ * [ ] DEPENDENCY
+
+```python
+ ('ncurses', '6.1'),
+```
+
+ * [ ] SANITY_CHECK_PATH ... you must check exists binary file
+
+ * [ ] DEPENDENCY
+
+```python
+sanity_check_paths = {
+    'files': ['bin/moon-buggy', 'com/moon-buggy/mbscore'],
+    'dirs': ['bin', 'com', 'share'],
+}
+```
+
+ * [ ] MODULECLASS ... choose class
+
+```python
+moduleclass = 'tools'
+```
+
+ * [ ] install `moon-buggy` from easyconfig
+
+```console
+$ eb moon-buggy-1.0.eb -r
+== temporary log file in case of crash /tmp/eb-ctAvZY/easybuild-GQkRPM.log
+== resolving dependencies ...
+== processing EasyBuild easyconfig /home/kru0052/game/moon-buggy-1.0.eb
+== building and installing moon-buggy/1.0-kru0052...
+== fetching files...
+...
+...
+== COMPLETED: Installation ended successfully
+== Results of the build can be found in the log file(s) /home/kru0052/.local/easybuild/software/moon-buggy/1.0-kru0052/easybuild/easybuild-moon-buggy-1.0-20181016.094918.log
+== Build succeeded for 1 out of 1
+== Temporary log file(s) /tmp/eb-ctAvZY/easybuild-GQkRPM.log* have been removed.
+== Temporary directory /tmp/eb-ctAvZY has been removed.
+```
+
+ * [ ] load module and run `moon-buggy`
+
+```console
+$ ml moon-buggy/1.0-kru0052
+$ moon-buggy
+```

EasyBuild/EasyBuild-Hands-on-2018/L03/template.eb

+# IT4Innovations 2018
+
+easyblock =
+
+name =
+version =
+versionsuffix =
+
+homepage =
+description = """ """
+
+toolchain = {'name': '', 'version': ''}
+
+source_urls = ['']
+sources = ['']
+
+preconfigopts =
+
+builddependencies = []
+
+dependencies = []
+
+sanity_check_paths = {
+    'files': [],
+    'dirs': [],
+}
+
+moduleclass = ''

EasyBuild/EasyBuild-Hands-on-2018/L04/L04.md

+# EasyBuild And Singularity
+
+[EasyBuild](https://docs.it4i.cz/software/easybuild/)
+[EasyBuild-Singularity](https://docs.it4i.cz/software/tools/easybuild-images/)
+
+
+
+
+

EasyBuild/EasyBuild-Hands-on-2018/L05/L05.md

+# Troubleshooting - typical problems
+
+## INSTALL - Already installed (module found)
+
+Install module from easyconfig `git-2.15.0-GCC-7.1.0-2.28-Python-3.6.1.eb` (your easyconfig from L02)
+
+```console
+[kru0052@login2.anselm test-build]$ eb git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb
+== temporary log file in case of crash /tmp/eb-KMNHse/easybuild-Jwn_R7.log
+== git/2.14.1-GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK is already installed (module found), skipping
+== No easyconfigs left to be built.
+== Build succeeded for 0 out of 0
+== Temporary log file(s) /tmp/eb-KMNHse/easybuild-Jwn_R7.log* have been removed.
+== Temporary directory /tmp/eb-KMNHse has been removed.
+```
+
+## REINSTALL - Module is already loaded
+
+Load module `git/2.14.1-GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK` and reinstall module
+
+```console
+[kru0052@login2.anselm test-build]$ eb git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb -f
+== temporary log file in case of crash /tmp/eb-NDp_Rx/easybuild-XgYOq9.log
+
+WARNING: Found one or more non-allowed loaded (EasyBuild-generated) modules in current environment:
+* bzip2/1.0.6
+* libreadline/6.3
+* SQLite/3.13.0
+* Tcl/8.6.5
+* Tk/8.6.5
+* GMP/6.1.1
+* XZ/5.2.2
+* zlib/1.2.11
+* Python/3.6.1
+* cURL/7.53.1
+* expat/2.2.0
+* ncurses/6.0
+* gettext/0.19.8.1
+* GCCcore/7.1.0
+* GCCcore/7.1.0
+* binutils/2.28-GCCcore-7.1.0
+* Perl/5.26.0-GCC-7.1.0-2.28-bare
+* git/2.14.1-GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK
+
+This is not recommended since it may affect the installation procedure(s) performed by EasyBuild.
+
+To make EasyBuild allow particular loaded modules, use the --allow-loaded-modules configuration option.
+To specify action to take when loaded modules are detected, use --detect-loaded-modules={error,ignore,purge,unload,warn}.
+
+See http://easybuild.readthedocs.io/en/latest/Detecting_loaded_modules.html for more information.
+
+== processing EasyBuild easyconfig /home_lustre/kru0052/hands-on/git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb
+== building and installing git/2.14.1-GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK...
+== fetching files...
+== creating build dir, resetting environment...
+== FAILED: Installation ended unsuccessfully (build directory: /home/kru0052/.local/easybuild/build/git/2.14.1/GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK): build failed (first 300 chars): Module is already loaded (EBROOTGIT is set), installation cannot continue.
+== Results of the build can be found in the log file(s) /tmp/eb-NDp_Rx/easybuild-git-2.14.1-20170911.085415.EaWTJ.log
+ERROR: Build of /home_lustre/kru0052/test-build/git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb failed (err: 'build failed (first 300 chars): Module is already loaded (EBROOTGIT is set), installation cannot continue.')
+```
+
+## INSTALL/REINSTALL - Irresolvable dependencies
+
+Change `('Python', '3.6.1', '', True)` to `('Python', '3.6.1')` and reinstall module
+
+```console
+[kru0052@login2.anselm ~]$ eb git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb -r
+== temporary log file in case of crash /tmp/eb-7toWyv/easybuild-koZYcG.log
+== processing EasyBuild easyconfig /home_lustre/kru0052/hands-on/git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb
+== building and installing git/2.14.1-GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK...
+== fetching files...
+== creating build dir, resetting environment...
+== FAILED: Installation ended unsuccessfully (build directory: /home/kru0052/.local/easybuild/build/git/2.14.1/GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK): build failed (first 300 chars): Missing modules for one or more dependencies: Python/3.6.1-GCC-7.1.0-2.28
+== Results of the build can be found in the log file(s) /tmp/eb-7toWyv/easybuild-git-2.14.1-20170911.084653.qAHAA.log
+ERROR: Build of /home_lustre/kru0052/test-build/git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb failed (err: 'build failed (first 300 chars): Missing modules for one or more dependencies: Python/3.6.1-GCC-7.1.0-2.28')
+```
+
+## INSTALL/REINSTALL - Can't find path
+
+Use bad easyconfig name `eb git-2.14.1-GCC-7.1.0-2.28.eb`
+
+```console
+[kru0052@login2.anselm test-build]$ eb git-2.14.1-GCC-7.1.0-2.28.eb -f
+== temporary log file in case of crash /tmp/eb-keyVcY/easybuild-AgLBmT.log
+ERROR: Can't find path /home_lustre/kru0052/test-build/git-2.14.1-GCC-7.1.0-2.28.eb
+```
+
+## INSTALL/REINSTALL - Sanity check failed
+
+Change `'files': ['bin/git']` to `'files': ['bin/git2']`
+
+```console
+[kru0052@login2.anselm test-build]$ eb git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb -f
+== temporary log file in case of crash /tmp/eb-Lv9b1_/easybuild-h2F2uL.log
+== processing EasyBuild easyconfig /home_lustre/kru0052/test-build/git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb
+== building and installing git/2.14.1-GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK...
+== fetching files...
+== creating build dir, resetting environment...
+== unpacking...
+== patching...
+== preparing...
+== configuring...
+== building...
+== testing...
+== installing...
+== taking care of extensions...
+== postprocessing...
+== sanity checking...
+== FAILED: Installation ended unsuccessfully (build directory: /home/kru0052/.local/easybuild/build/git/2.14.1/GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK): build failed (first 300 chars): Sanity check failed: no file of ('bin/git2',) in /home/kru0052/.local/easybuild/software/git/2.14.1-GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK
+== Results of the build can be found in the log file(s) /tmp/eb-Lv9b1_/easybuild-git-2.14.1-20170911.085750.VJyvg.log
+ERROR: Build of /home_lustre/kru0052/test-build/git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb failed (err: "build failed (first 300 chars): Sanity check failed: no file of ('bin/git2',) in /home/kru0052/.local/easybuild/software/git/2.14.1-GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK")
+```
+
+## INSTALL/REINSTALL - Couldn't find file
+
+Change `sources = ['%(name)s-%(version)s.tar.gz']` to `sources = ['%(name)s-%(version)s-test.tar.gz']`
+
+```console
+[kru0052@login2.anselm test-build]$ eb git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb -f
+== temporary log file in case of crash /tmp/eb-N64arw/easybuild-lXjC9B.log
+== processing EasyBuild easyconfig /home_lustre/kru0052/hands-one/git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb
+== building and installing git/2.14.1-GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK...
+== fetching files...
+== FAILED: Installation ended unsuccessfully (build directory: /home/kru0052/.local/easybuild/build/git/2.14.1/GCC-7.1.0-2.28-Python-3.6.1-by-FoXiK): build failed (first 300 chars): Couldn't find file git-2.14.1-test.tar.gz anywhere, and downloading it didn't work either... Paths attempted (in order): /home_lustre/kru0052/hands-on/g/git/git-2.14.1-test.tar.gz, /home_lustre/kru0052/hands-on/git/git-2.14.1-test.tar.gz, /home_lustre/kru0052/hands-on/git-2.14.1-test.tar.gz, /
+== Results of the build can be found in the log file(s) /tmp/eb-N64arw/easybuild-git-2.14.1-20170911.085547.BnfZG.log
+ERROR: Build of /home_lustre/kru0052/hands-on/git-2.14.1-GCC-7.1.0-2.28-Python-3.6.1.eb failed (err: "build failed (first 300 chars): Couldn't find file git-2.14.1-test.tar.gz anywhere, and downloading it didn't work either... Paths attempted (in order): /home_lustre/kru0052/hands-on/g/git/git-2.14.1-test.tar.gz, /home_lustre/kru0052/test-build/git/git-2.14.1-test.tar.gz, /home_lustre/kru0052/hands-on/git-2.14.1-test.tar.gz, /")
+```