new file: Ubuntu/16.04-OpenFOAM/README

	new file:   Ubuntu/16.04-OpenFOAM/Ubuntu-16.04-Bootstrap
	new file:   Ubuntu/16.04-OpenFOAM/app/check
	new file:   Ubuntu/16.04-OpenFOAM/app/monitor
	new file:   Ubuntu/16.04-OpenFOAM/app/startup
	new file:   Ubuntu/16.04-OpenFOAM/hpc_image
	new file:   Ubuntu/16.04-OpenFOAM/openfoam_hacks/aliases
	new file:   Ubuntu/16.04-OpenFOAM/openfoam_hacks/bashrc
	new file:   Ubuntu/16.04-OpenFOAM/openfoam_hacks/exec
	new file:   Ubuntu/16.04-OpenFOAM/recipe-openfoam.txt
	new file:   Ubuntu/16.04-OpenFOAM/singularity_image.tar
	new file:   install/CENTOS7
	new file:   install/CENTOS7GPU
	new file:   install/CENTOS7MIC
	modified:   install/install.sh
	deleted:    install/CENTOS74
	deleted:    install/CENTOS74GPU
	deleted:    install/CENTOS74MIC

Ubuntu/16.04-OpenFOAM/README

+A few instruction for building an openfoam image to be used on the CFG platform.
+Openfoam in this image can be used directly, by calling the solver, or through a python procedure useful to monitor the calculation on the CFG platform.
+
+Usually I build a sandbox writable image. When it's ok, I build a read-only image to be uploaded on CFG platform.
+
+openfoamdev/ is the sandbox image, to be used for development and debug
+openfoam.simg is a non-writable image deployable on CFG platform
+
+
+BUILDING OF THE IMAGE
+
+To build the writable image, use the recipe-openfoam.txt recipe as follows
+
+   sudo singularity build -s openfoamdev recipe-openfoam.txt
+   
+Files in folders app and openfoam_hacks are needed with this recipe.
+In the app folders there are files needed for checking and monitoring the calculation.
+They can be modified or not added.
+Files in openfoam_hacks are needed to allows openfoam to run in the container.
+
+
+TESTING OF THE IMAGE
+
+To test the container, the two mountpoints requested by the CGF platform are to be creted.
+Of course they can reside in a folder different than ~.
+
+  mkdir ~/scratch
+  mkdir ~/service
+  
+- Serial test (direct call of the solver). Specify the solver and the case folder (-case option)
+
+  singularity -B ~/service:service ~/scratch:/scratch exec --cleanenv openfoamdev ###SOLVER###
+
+- Parallel test (direct call of the solver). Specify the solver, the case folder (-case option) and the number of processors.
+Of course, modify the system/decomposeParDict file accordingly.
+
+  singularity -B ~/service:service ~/scratch:/scratch exec --cleanenv openfoamdev decomposePar
+  mpirun -np ###PROCS#### singularity -B ~/service:service ~/scratch:/scratch exec --cleanenv openfoamdev ###SOLVER### -parallel
+
+NOTE When executing directly a solver in parallel mode, this can be a truly MPI calculation.
+     Instead, when testing the python procedure, only processors on one single node can be used.
+
+- Python procedure test.
+When using the python procedure, the solver, the case folder and the number of processors are specified on command line.
+The decomposePar utility is executed, then the solver is started through an mpirun command and the the check utility is started.
+As stated before, in this way only processors on a single node can be used. This will hopefully change in future.
+A webpage is displayed in "~/service/status.html". The abort button can't work locally, but only on the CFG platform.
+Anyway, the interrupt procedure is implemented. See /app/notification file.
+
+  singularity -B ~/service:service ~/scratch:/scratch --cleanenv exec openfoamdev /app/startup -s SOLVER -f FOLDER -p NUMBER_OF_PROCESSORS
+
+The /app/check file is to be modified in order to display the desided webpage.
+The parsing procedure is taylored for interFoam solver and can fail when parsing output of different solvers, so please modify it and test it locally.
+
+
+DEPLOYMENT ON CFG PLATFORM
+
+To build to a non writable image:
+   sudo singularity build openfoam.simg recipe-openfoam.txt
+
+The hpc_image script can be used to upload and register the new image on the CFG platform, 
+or to unregister and delete. Specify username, password and project in the script before using it.
+To upload and register 
+  
+  ./hpc_image upload openfoam.simg
+
+To unregister and delete
+
+  ./hpc_image delete openfoam.simg
+
+
+
+
+

Ubuntu/16.04-OpenFOAM/Ubuntu-16.04-Bootstrap

+Bootstrap: docker
+From: ubuntu:16.04
+
+%post
+apt -y update && apt -y upgrade
+apt -y install python curl wget gcc g++ make vim mc libncurses-dev rsync tclsh lua5.2 liblua5.2 gfortran ssh
+
+curl -s -O https://bootstrap.pypa.io/get-pip.py
+python get-pip.py
+pip install setuptools python-graph-core python-graph-dot tabulate numpy scipy pandas wheel --upgrade
+
+wget https://github.com/luarocks/luarocks/archive/v2.4.2.tar.gz
+tar xvf v2.4.2.tar.gz
+cd luarocks-2.4.2
+./configure
+make build
+make install
+mkdir ~/.luarocks
+echo "fs_use_modules = false" > ~/.luarocks/config.lua
+luarocks install luaposix
+luarocks install luafilesystem
+LUAROCKS_PREFIX=/usr/local
+export LUA_PATH="$LUAROCKS_PREFIX/share/lua/5.1/?.lua;$LUAROCKS_PREFIX/share/lua/5.1/?/init.lua;;"
+export LUA_CPATH="$LUAROCKS_PREFIX/lib/lua/5.1/?.so;;"
+
+echo "# !/bin/bash" > /bin/logger
+echo "exit 0" >> /bin/logger
+chmod +x /bin/logger
+
+# Lmod
+wget https://github.com/TACC/Lmod/archive/7.7.7.tar.gz
+tar xvf 7.7.7.tar.gz
+cd Lmod-7.7.7
+./configure --prefix=/opt/apps
+make install
+
+# OpenMPI
+wget https://www.open-mpi.org/software/ompi/v2.1/downloads/openmpi-2.1.1.tar.gz
+tar xvf openmpi-2.1.1.tar.gz && cd openmpi-2.1.1
+./configure --prefix=/usr/local
+make -j 16
+make install
+
+# APPS and SCRATCH
+mkdir /apps
+mkdir /scratch
+
+cd /bin
+rm sh && ln -s /bin/bash sh
+
+
+### folders and files ###
+
+mkdir /service  
+mv /opt/startup /app
+mv /opt/check /app
+mv /opt/monitor /app
+
+### set-up python3 ###  
+
+apt install -y python3-pip python3-mpi4py
+pip3 install numpy matplotlib
+  
+### install openfoam ###
+
+apt install -y software-properties-common wget
+add-apt-repository "http://dl.openfoam.org/ubuntu"
+sh -c "wget -O - http://dl.openfoam.org/gpg.key | apt-key add -"
+apt update
+apt install -y openfoam5
+echo '. /opt/openfoam5/etc/bashrc' >>$SINGULARITY_ENVIRONMENT
+
+### openfoam hacks ###
+
+mv -f /opt/bashrc /opt/openfoam5/etc
+mv -f /opt/aliases /opt/openfoam5/etc/config.sh
+mv -f /opt/exec /.singularity.d/actions
+
+%environment
+source /opt/apps/lmod/7.7.7/init/profile
+clearMT
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
+export -f ml
+export -f module
+export -f clearMT
+
+%files
+app/startup opt/
+app/check   opt/
+app/monitor opt/
+openfoam_hacks/bashrc opt/
+openfoam_hacks/aliases opt/
+openfoam_hacks/exec opt/

Ubuntu/16.04-OpenFOAM/app/check

+#!/usr/bin/python3
+
+'''
+   This file parses the solver log file and produces the webpage displayed
+   during the simulation. Also a webpage result is written after the end of the
+   simulation.
+   
+   The webpage displays just the residuals and the abort button. Of course it 
+   can be customized.
+   
+
+
+
+
+'''
+
+import time
+import sys
+from subprocess import check_output
+from os.path import isfile, basename
+from os import getcwd
+from shutil import copy
+import logging
+import base64
+
+import re
+import matplotlib
+matplotlib.use('Agg')
+import matplotlib.pyplot as plt
+
+
+
+PROGRESS_HEAD = '''<html>
+  <head>
+    <title>OPENFOAM job progress</title>
+    <script type="text/javascript">
+      function abort() {
+        notify_running_job("ABORT");
+      }
+    </script>
+  </head>'''
+  
+  
+PROGRESS_BODY='''<body style="margin: 20px; padding: 20px;">
+    <h3 align=center>Simulation {}</h3>
+    <p style="text-align:center;">
+<!--    Images to be embedded encoded in base64-->
+    <img src = "data:image/png;base64,{}" alt ="missing image">
+    </p>
+    <p style="text-align:center;">
+    <input type="button" value="Abort simulation" onclick="abort()">
+    </p>
+  </body>
+</html>
+'''
+
+'''
+  Regular expression needed to parse the log file
+'''
+_rx_dict = {
+    'time': re.compile(r'^Time = (?P<time>(\d+.\d+e[+-]\d+)|(\d+[.\d+]*))'),
+    'timestep': re.compile(r'^deltaT = (?P<timestep>(\d+.\d+e[+-]\d+)|(\d+[.\d+]*))'),
+    'residual': re.compile(r'''
+        ^\w+: \s+                                          # solver name
+        Solving \s for \s (?P<quantity>\w+(.\w+)*), \s     # field
+        Initial \s residual \s = \s                         
+        (?P<residual>(-?\d+.\d+e[+-]\d+)|(-?\d+[.\d+]*))   # residual value
+        ''',
+        re.VERBOSE),
+    'clock': re.compile(r'ExecutionTime = (?P<clock>\d+.\d+) s')
+    }
+    
+def _parse_line(line):
+    '''
+      Check if the line matches one of the regex in _rx_dict
+    '''
+    for key, rx in _rx_dict.items():
+        match = rx.search(line)
+        if match:
+            return key, match       
+    return None, None
+
+def give_nth_word(line,n):
+    try:
+        out = line.split()[n-1].strip(",")
+    except IndexError:
+        return ""
+    return out
+
+def parse_logfile(logfilename,data,offset=0):
+    '''
+      Parses the file LOGFILENAME starting from OFFSET line, putting requested 
+      data in list DATA and updating the OFFSET variable.
+      It is executed every 3 seconds (see main function).
+      Residuals, timestep, simulation time and clock time are parsed and 
+      appended to data, which is a list of list.
+    '''
+    time_data = {}
+    content = check_output(["cat",logfilename])
+    lines = content.decode("utf-8").splitlines(True)
+    for index,line in enumerate(lines):
+        if index > offset:
+            key, match = _parse_line(line)
+            if key == 'time':
+                time_data['time'] = float(match.group('time'))
+            if key == 'residual':
+                time_data[match.group('quantity')] = float(match.group('residual'))
+            if key == 'timestep':
+                time_data['timestep'] = float(match.group('timestep'))
+            if key == 'clock':
+                time_data['clock'] = float(match.group('clock'))
+                offset = index
+                data[0].append(time_data['time'])
+                data[1].append(time_data['timestep'])
+                data[2].append(time_data['Ux'])
+                data[3].append(time_data['Uy'])
+                data[4].append(time_data['Uz'])
+                data[5].append(time_data['p_rgh'])
+                data[6].append(time_data['clock'])
+    return offset
+
+def check_if_finished(logfilename):
+    with open(logfilename,'r') as logfile:
+        for line in logfile:
+            pass
+    if give_nth_word(line,1) == "Finalising":
+        return True
+    else:
+        return False
+
+def plot_images(data):
+    '''
+      Plot residual data and save it in a png file. This file will be embedded
+      in the webpage.
+    '''
+    try:
+        fig = plt.figure()
+        ax1 = fig.add_subplot(1,1,1)
+        ax1.set_xlabel('Time [s]')
+        ax1.set_ylabel('Residuals')
+        ax1.semilogy(data[0],data[2], label='ux')
+        ax1.semilogy(data[0],data[3], label='uy')
+        ax1.semilogy(data[0],data[4], label='uz')
+        ax1.semilogy(data[0],data[5], label='p')
+        leg = ax1.legend()
+        plt.savefig('/service/output.png')
+        plt.close()
+    except UserWarning:
+        return
+    return
+
+def main():
+    '''
+      Main function.
+      Every 3 seconds, che logfile is parsed, it is checked if the simulation is
+      finished, then the status webpage is updated.
+      In the end, the same webpage is written in the result file.
+    
+    '''
+    folder = sys.argv[1]
+    casename = basename((getcwd()))
+    logfilename = folder + '/' + casename + '.log'
+    statusfilename = '/service/status.html'
+    resultfilename = '/service/result.txt'
+    time.sleep(5)
+    finished = False
+    log_offset = 0
+    output_data = [[],[],[],[],[],[],[]]
+###    time, timestep, ux, uy, uz, pres, clock
+    while not finished:
+        time.sleep(3)
+        log_offset = parse_logfile(logfilename,output_data,log_offset)
+        finished = check_if_finished(logfilename)
+        plot_images(output_data)
+        with open('/service/output.png','rb') as imgfile:
+            b64image = base64.b64encode(imgfile.read()).decode()
+        with open(statusfilename,'w') as statfile:
+            statfile.write(str(PROGRESS_HEAD + PROGRESS_BODY.format(casename,b64image)))
+    with open(resultfilename,'w') as resfile:
+        resfile.write(str(PROGRESS_HEAD + PROGRESS_BODY.format(casename,b64image)))
+
+if __name__ == "__main__":
+    main()

Ubuntu/16.04-OpenFOAM/app/monitor

+#!/usr/bin/env python3
+
+
+'''
+  This file checks if something is written in the notification file
+  If the message ABORT is written, the controlDict file is modified in order to
+  stop the simulation.
+  Other answers to other command could be implemented, if needed.
+
+'''
+
+
+import os
+import sys
+import logging
+import time
+from subprocess import check_output
+
+logging.basicConfig(level=logging.DEBUG)
+logger = logging.getLogger("Notifications monitor")
+
+def modify_dict(dictionary,keyword,value):
+  spaces = " " * (16-len(keyword))
+  try:
+    with open(dictionary,"r") as old_filedict:
+      with open(dictionary+"new","w") as new_filedict:
+        for line in old_filedict:
+          keys = line.split()
+          if (len(keys) > 1) and not "//" in keys[0]:
+            if keyword == keys[0]:
+              line = keyword + spaces + str(value) + ";\n"
+          print(line,end='',file=new_filedict) 
+    os.remove(dictionary)
+    os.rename(dictionary+"new",dictionary)
+  except FileNotFoundError:
+    print("File " + dictionary + " not found.\n")
+    exit()
+  return 
+
+def main():
+
+    logger.info("Notification monitor started")
+    folder = sys.argv[1]
+    dictFile = folder + '/system/controlDict'
+    cached_stamp = 0
+
+    while True:
+        if os.path.exists('/service/notifications.txt'):
+            stamp = os.stat('/service/notifications.txt').st_mtime
+            if stamp != cached_stamp:
+                logger.info("New notifications received")
+                cached_stamp = stamp
+                command = check_output(["tail","-1","/service/notifications.txt"]).strip().decode()
+                if command == "ABORT":
+                    logger.info("Received abort command")
+                    modify_dict(dictFile,"stopAt","writeNow")
+                else:
+                    logger.info("Ignoring unknown command {}".format(command))
+        time.sleep(1)
+
+if __name__ == "__main__":
+    main()

Ubuntu/16.04-OpenFOAM/app/startup

+#!/usr/bin/env python3
+
+
+'''
+  This is the principal file.
+  It parses the command line, in which requested openFoam solver, case folder 
+  and number of processors are specified.
+  It decomposes the case, calls the check and monitor scripts, then calls 
+  the solver with the correct parameters and, finally, reconstructs the case.
+  
+  cmdline syntax:
+  
+    startup -s SOLVER [-f FOLDER] -p PROCESSORS
+    
+  The execution is parallel, but since mpi is called inside the container, 
+  processes are spawn only on one node, so there can be at maximum 16 processes
+  on Anselm ann 24 on Salomon.
+ 
+  A truly parallel execution would probably require an mpi4py procedure.
+  
+  (Leonardo) It seems that subprocess commands don't work in conjunction with 
+             an mpi4py script. I have no idea on how to launch an MPI
+             openfoam solver from inside a MPI python script.
+
+'''
+
+from subprocess import Popen, run, DEVNULL
+from os import getcwd, chdir, rename, remove
+from os.path import basename, abspath
+import logging
+import argparse
+
+logging.basicConfig(level=logging.DEBUG)
+logger = logging.getLogger("Openfoam image logger")
+
+def command_line():
+    '''
+      Define and parses the command line arguments.
+      
+    '''
+    parser = argparse.ArgumentParser()
+    parser.add_argument('-solver',action='store',dest='solver')
+    parser.add_argument('-folder', action='store', dest='folder')
+    parser.add_argument('-processor',action='store', type=int, dest='np')
+    args = vars(parser.parse_args())
+    solver = args['solver']
+    folder = args['folder']
+    np = args['np']
+    return solver,folder,np
+    
+def modify_dict(dictionary,keyword,value):
+  '''
+   Modify the VALUE of the desired KEYWORD in the desider DICTIONARY
+  
+  '''    
+  spaces = " " * (16-len(keyword))
+  try:
+    with open(dictionary,"r") as old_filedict:
+      with open(dictionary+"new","w") as new_filedict:
+        for line in old_filedict:
+          keys = line.split()
+          if (len(keys) > 1) and not "//" in keys[0]:
+            if keyword == keys[0]:
+              line = keyword + spaces + str(value) + ";\n"
+          print(line,end='',file=new_filedict) 
+    remove(dictionary)
+    rename(dictionary+"new",dictionary)
+  except FileNotFoundError:
+    print("File " + dictionary + " not found.\n")
+    exit()
+  return 
+
+def main():
+    solver,folder,np = command_line()
+    # folder check
+    if folder is not None:
+        try:
+            chdir(folder)
+        except FileNotFoundError:
+            logger.error("Folder {} not found".format(folder))
+            exit()
+    else:
+        folder = getcwd()
+    folder = abspath(getcwd())
+    casename = basename(getcwd())
+    logfilename = folder + '/' + casename + '.log'
+    statusfilename = '/service/status.html'
+
+    with open(logfilename,'w') as logfile:
+        logger.info("Running decomposePar utility")
+        run(["decomposePar","-force"],stdout=logfile)
+        check_proc = Popen(["/app/check",folder])
+        monitor_proc = Popen(["/app/monitor",folder])
+        logger.info("Running {} on {} processors".format(solver,str(np)))
+        run(["mpirun","-np",str(np),solver,"-parallel","-case",folder],stdout=logfile)
+        logger.info("Running reconstructPar")
+        run(["reconstructPar"],stdout=DEVNULL)
+
+    monitor_proc.terminate()
+### in case of abort command, set the controlDict file in the original state
+    ctrlDict = folder + '/system/controlDict'
+    modify_dict(ctrlDict,"stopAt","endTime")
+    logger.info("Calculation ended")
+
+if __name__ == "__main__":
+    main()

Ubuntu/16.04-OpenFOAM/hpc_image

+#!/usr/bin/python3
+
+"""Ugly but working hard-coded test script for the HPC client"""
+""" 
+    For upload and register, hpc_image upload image_name
+    For unregister and delete, hpc_image delete image_name
+    
+    
+    N.B. Images are uploaded in /home/images folder in anselm cluster
+    N.B. Folder /home/images has to exist on the cluster
+    N.B. Username, password and project should be filled with needed values
+
+"""
+
+
+
+import os
+import filecmp
+import json
+
+import clfpy as cf
+import sys
+
+auth_url = "https://api.hetcomp.org/authManager/AuthManager?wsdl"
+hpc_url = "https://api.hetcomp.org/hpc_anselm/Images?wsdl"
+gss_url = "https://api.hetcomp.org/gss-0.1/FileUtilities?wsdl"
+username = ""
+password = ""
+project = ''
+
+
+print("Obtaining session token ...")
+auth = cf.AuthClient(auth_url)
+session_token = auth.get_session_token(username, project, password)
+
+gss = cf.GssClient(gss_url)
+hpc = cf.HpcImagesClient(hpc_url)
+
+
+
+def upload():
+    print("Uploading and registering a new image")
+    image_filepath = sys.argv[2]
+    gss_ID = "it4i_anselm://home/images/" + sys.argv[2]
+    print(gss.upload(gss_ID, session_token, image_filepath))
+    print(hpc.register_image(session_token, sys.argv[2], gss_ID))
+
+def delete():
+    print("Removing and deleting image")
+    print(hpc.delete_image(session_token, sys.argv[2]))  
+    gss_ID = "it4i_anselm://home/images/" + sys.argv[2]
+    print(gss.delete(gss_ID, session_token))
+
+
+def main():
+    if sys.argv[1] == "upload":
+        upload()
+    elif sys.argv[1] == "delete":
+        delete()
+        
+if __name__ == "__main__":
+    main()
+    
+
+
+
+
+
+
+
+

Ubuntu/16.04-OpenFOAM/openfoam_hacks/aliases

+#----------------------------------*-sh-*--------------------------------------
+# =========                 |
+# \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
+#  \\    /   O peration     |
+#   \\  /    A nd           | Copyright (C) 2011-2016 OpenFOAM Foundation
+#    \\/     M anipulation  |
+#------------------------------------------------------------------------------
+# License
+#     This file is part of OpenFOAM.
+#
+#     OpenFOAM is free software: you can redistribute it and/or modify it
+#     under the terms of the GNU General Public License as published by
+#     the Free Software Foundation, either version 3 of the License, or
+#     (at your option) any later version.
+#
+#     OpenFOAM is distributed in the hope that it will be useful, but WITHOUT
+#     ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+#     FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+#     for more details.
+#
+#     You should have received a copy of the GNU General Public License
+#     along with OpenFOAM.  If not, see <http://www.gnu.org/licenses/>.
+#
+# File
+#     etc/config.sh/aliases
+#
+# Description
+