Commit a64014e4 authored by Lukáš Krupčík's avatar Lukáš Krupčík
Browse files

new file: Ubuntu/16.04-OpenFOAM/README

	new file:   Ubuntu/16.04-OpenFOAM/Ubuntu-16.04-Bootstrap
	new file:   Ubuntu/16.04-OpenFOAM/app/check
	new file:   Ubuntu/16.04-OpenFOAM/app/monitor
	new file:   Ubuntu/16.04-OpenFOAM/app/startup
	new file:   Ubuntu/16.04-OpenFOAM/hpc_image
	new file:   Ubuntu/16.04-OpenFOAM/openfoam_hacks/aliases
	new file:   Ubuntu/16.04-OpenFOAM/openfoam_hacks/bashrc
	new file:   Ubuntu/16.04-OpenFOAM/openfoam_hacks/exec
	new file:   Ubuntu/16.04-OpenFOAM/recipe-openfoam.txt
	new file:   Ubuntu/16.04-OpenFOAM/singularity_image.tar
	new file:   install/CENTOS7
	new file:   install/CENTOS7GPU
	new file:   install/CENTOS7MIC
	modified:   install/
	deleted:    install/CENTOS74
	deleted:    install/CENTOS74GPU
	deleted:    install/CENTOS74MIC
parent 7ce8fe0e
A few instruction for building an openfoam image to be used on the CFG platform.
Openfoam in this image can be used directly, by calling the solver, or through a python procedure useful to monitor the calculation on the CFG platform.
Usually I build a sandbox writable image. When it's ok, I build a read-only image to be uploaded on CFG platform.
openfoamdev/ is the sandbox image, to be used for development and debug
openfoam.simg is a non-writable image deployable on CFG platform
To build the writable image, use the recipe-openfoam.txt recipe as follows
sudo singularity build -s openfoamdev recipe-openfoam.txt
Files in folders app and openfoam_hacks are needed with this recipe.
In the app folders there are files needed for checking and monitoring the calculation.
They can be modified or not added.
Files in openfoam_hacks are needed to allows openfoam to run in the container.
To test the container, the two mountpoints requested by the CGF platform are to be creted.
Of course they can reside in a folder different than ~.
mkdir ~/scratch
mkdir ~/service
- Serial test (direct call of the solver). Specify the solver and the case folder (-case option)
singularity -B ~/service:service ~/scratch:/scratch exec --cleanenv openfoamdev ###SOLVER###
- Parallel test (direct call of the solver). Specify the solver, the case folder (-case option) and the number of processors.
Of course, modify the system/decomposeParDict file accordingly.
singularity -B ~/service:service ~/scratch:/scratch exec --cleanenv openfoamdev decomposePar
mpirun -np ###PROCS#### singularity -B ~/service:service ~/scratch:/scratch exec --cleanenv openfoamdev ###SOLVER### -parallel
NOTE When executing directly a solver in parallel mode, this can be a truly MPI calculation.
Instead, when testing the python procedure, only processors on one single node can be used.
- Python procedure test.
When using the python procedure, the solver, the case folder and the number of processors are specified on command line.
The decomposePar utility is executed, then the solver is started through an mpirun command and the the check utility is started.
As stated before, in this way only processors on a single node can be used. This will hopefully change in future.
A webpage is displayed in "~/service/status.html". The abort button can't work locally, but only on the CFG platform.
Anyway, the interrupt procedure is implemented. See /app/notification file.
singularity -B ~/service:service ~/scratch:/scratch --cleanenv exec openfoamdev /app/startup -s SOLVER -f FOLDER -p NUMBER_OF_PROCESSORS
The /app/check file is to be modified in order to display the desided webpage.
The parsing procedure is taylored for interFoam solver and can fail when parsing output of different solvers, so please modify it and test it locally.
To build to a non writable image:
sudo singularity build openfoam.simg recipe-openfoam.txt
The hpc_image script can be used to upload and register the new image on the CFG platform,
or to unregister and delete. Specify username, password and project in the script before using it.
To upload and register
./hpc_image upload openfoam.simg
To unregister and delete
./hpc_image delete openfoam.simg
Bootstrap: docker
From: ubuntu:16.04
apt -y update && apt -y upgrade
apt -y install python curl wget gcc g++ make vim mc libncurses-dev rsync tclsh lua5.2 liblua5.2 gfortran ssh
curl -s -O
pip install setuptools python-graph-core python-graph-dot tabulate numpy scipy pandas wheel --upgrade
tar xvf v2.4.2.tar.gz
cd luarocks-2.4.2
make build
make install
mkdir ~/.luarocks
echo "fs_use_modules = false" > ~/.luarocks/config.lua
luarocks install luaposix
luarocks install luafilesystem
export LUA_PATH="$LUAROCKS_PREFIX/share/lua/5.1/?.lua;$LUAROCKS_PREFIX/share/lua/5.1/?/init.lua;;"
export LUA_CPATH="$LUAROCKS_PREFIX/lib/lua/5.1/?.so;;"
echo "# !/bin/bash" > /bin/logger
echo "exit 0" >> /bin/logger
chmod +x /bin/logger
# Lmod
tar xvf 7.7.7.tar.gz
cd Lmod-7.7.7
./configure --prefix=/opt/apps
make install
# OpenMPI
tar xvf openmpi-2.1.1.tar.gz && cd openmpi-2.1.1
./configure --prefix=/usr/local
make -j 16
make install
mkdir /apps
mkdir /scratch
cd /bin
rm sh && ln -s /bin/bash sh
### folders and files ###
mkdir /service
mv /opt/startup /app
mv /opt/check /app
mv /opt/monitor /app
### set-up python3 ###
apt install -y python3-pip python3-mpi4py
pip3 install numpy matplotlib
### install openfoam ###
apt install -y software-properties-common wget
add-apt-repository ""
sh -c "wget -O - | apt-key add -"
apt update
apt install -y openfoam5
echo '. /opt/openfoam5/etc/bashrc' >>$SINGULARITY_ENVIRONMENT
### openfoam hacks ###
mv -f /opt/bashrc /opt/openfoam5/etc
mv -f /opt/aliases /opt/openfoam5/etc/
mv -f /opt/exec /.singularity.d/actions
source /opt/apps/lmod/7.7.7/init/profile
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export -f ml
export -f module
export -f clearMT
app/startup opt/
app/check opt/
app/monitor opt/
openfoam_hacks/bashrc opt/
openfoam_hacks/aliases opt/
openfoam_hacks/exec opt/
This file parses the solver log file and produces the webpage displayed
during the simulation. Also a webpage result is written after the end of the
The webpage displays just the residuals and the abort button. Of course it
can be customized.
import time
import sys
from subprocess import check_output
from os.path import isfile, basename
from os import getcwd
from shutil import copy
import logging
import base64
import re
import matplotlib
import matplotlib.pyplot as plt
PROGRESS_HEAD = '''<html>
<title>OPENFOAM job progress</title>
<script type="text/javascript">
function abort() {
PROGRESS_BODY='''<body style="margin: 20px; padding: 20px;">
<h3 align=center>Simulation {}</h3>
<p style="text-align:center;">
<!-- Images to be embedded encoded in base64-->
<img src = "data:image/png;base64,{}" alt ="missing image">
<p style="text-align:center;">
<input type="button" value="Abort simulation" onclick="abort()">
Regular expression needed to parse the log file
_rx_dict = {
'time': re.compile(r'^Time = (?P<time>(\d+.\d+e[+-]\d+)|(\d+[.\d+]*))'),
'timestep': re.compile(r'^deltaT = (?P<timestep>(\d+.\d+e[+-]\d+)|(\d+[.\d+]*))'),
'residual': re.compile(r'''
^\w+: \s+ # solver name
Solving \s for \s (?P<quantity>\w+(.\w+)*), \s # field
Initial \s residual \s = \s
(?P<residual>(-?\d+.\d+e[+-]\d+)|(-?\d+[.\d+]*)) # residual value
'clock': re.compile(r'ExecutionTime = (?P<clock>\d+.\d+) s')
def _parse_line(line):
Check if the line matches one of the regex in _rx_dict
for key, rx in _rx_dict.items():
match =
if match:
return key, match
return None, None
def give_nth_word(line,n):
out = line.split()[n-1].strip(",")
except IndexError:
return ""
return out
def parse_logfile(logfilename,data,offset=0):
Parses the file LOGFILENAME starting from OFFSET line, putting requested
data in list DATA and updating the OFFSET variable.
It is executed every 3 seconds (see main function).
Residuals, timestep, simulation time and clock time are parsed and
appended to data, which is a list of list.
time_data = {}
content = check_output(["cat",logfilename])
lines = content.decode("utf-8").splitlines(True)
for index,line in enumerate(lines):
if index > offset:
key, match = _parse_line(line)
if key == 'time':
time_data['time'] = float('time'))
if key == 'residual':
time_data['quantity')] = float('residual'))
if key == 'timestep':
time_data['timestep'] = float('timestep'))
if key == 'clock':
time_data['clock'] = float('clock'))
offset = index
return offset
def check_if_finished(logfilename):
with open(logfilename,'r') as logfile:
for line in logfile:
if give_nth_word(line,1) == "Finalising":
return True
return False
def plot_images(data):
Plot residual data and save it in a png file. This file will be embedded
in the webpage.
fig = plt.figure()
ax1 = fig.add_subplot(1,1,1)
ax1.set_xlabel('Time [s]')
ax1.semilogy(data[0],data[2], label='ux')
ax1.semilogy(data[0],data[3], label='uy')
ax1.semilogy(data[0],data[4], label='uz')
ax1.semilogy(data[0],data[5], label='p')
leg = ax1.legend()
except UserWarning:
def main():
Main function.
Every 3 seconds, che logfile is parsed, it is checked if the simulation is
finished, then the status webpage is updated.
In the end, the same webpage is written in the result file.
folder = sys.argv[1]
casename = basename((getcwd()))
logfilename = folder + '/' + casename + '.log'
statusfilename = '/service/status.html'
resultfilename = '/service/result.txt'
finished = False
log_offset = 0
output_data = [[],[],[],[],[],[],[]]
### time, timestep, ux, uy, uz, pres, clock
while not finished:
log_offset = parse_logfile(logfilename,output_data,log_offset)
finished = check_if_finished(logfilename)
with open('/service/output.png','rb') as imgfile:
b64image = base64.b64encode(
with open(statusfilename,'w') as statfile:
statfile.write(str(PROGRESS_HEAD + PROGRESS_BODY.format(casename,b64image)))
with open(resultfilename,'w') as resfile:
resfile.write(str(PROGRESS_HEAD + PROGRESS_BODY.format(casename,b64image)))
if __name__ == "__main__":
#!/usr/bin/env python3
This file checks if something is written in the notification file
If the message ABORT is written, the controlDict file is modified in order to
stop the simulation.
Other answers to other command could be implemented, if needed.
import os
import sys
import logging
import time
from subprocess import check_output
logger = logging.getLogger("Notifications monitor")
def modify_dict(dictionary,keyword,value):
spaces = " " * (16-len(keyword))
with open(dictionary,"r") as old_filedict:
with open(dictionary+"new","w") as new_filedict:
for line in old_filedict:
keys = line.split()
if (len(keys) > 1) and not "//" in keys[0]:
if keyword == keys[0]:
line = keyword + spaces + str(value) + ";\n"
except FileNotFoundError:
print("File " + dictionary + " not found.\n")
def main():"Notification monitor started")
folder = sys.argv[1]
dictFile = folder + '/system/controlDict'
cached_stamp = 0
while True:
if os.path.exists('/service/notifications.txt'):
stamp = os.stat('/service/notifications.txt').st_mtime
if stamp != cached_stamp:"New notifications received")
cached_stamp = stamp
command = check_output(["tail","-1","/service/notifications.txt"]).strip().decode()
if command == "ABORT":"Received abort command")
else:"Ignoring unknown command {}".format(command))
if __name__ == "__main__":
#!/usr/bin/env python3
This is the principal file.
It parses the command line, in which requested openFoam solver, case folder
and number of processors are specified.
It decomposes the case, calls the check and monitor scripts, then calls
the solver with the correct parameters and, finally, reconstructs the case.
cmdline syntax:
The execution is parallel, but since mpi is called inside the container,
processes are spawn only on one node, so there can be at maximum 16 processes
on Anselm ann 24 on Salomon.
A truly parallel execution would probably require an mpi4py procedure.
(Leonardo) It seems that subprocess commands don't work in conjunction with
an mpi4py script. I have no idea on how to launch an MPI
openfoam solver from inside a MPI python script.
from subprocess import Popen, run, DEVNULL
from os import getcwd, chdir, rename, remove
from os.path import basename, abspath
import logging
import argparse
logger = logging.getLogger("Openfoam image logger")
def command_line():
Define and parses the command line arguments.
parser = argparse.ArgumentParser()
parser.add_argument('-folder', action='store', dest='folder')
parser.add_argument('-processor',action='store', type=int, dest='np')
args = vars(parser.parse_args())
solver = args['solver']
folder = args['folder']
np = args['np']
return solver,folder,np
def modify_dict(dictionary,keyword,value):
Modify the VALUE of the desired KEYWORD in the desider DICTIONARY
spaces = " " * (16-len(keyword))
with open(dictionary,"r") as old_filedict:
with open(dictionary+"new","w") as new_filedict:
for line in old_filedict:
keys = line.split()
if (len(keys) > 1) and not "//" in keys[0]:
if keyword == keys[0]:
line = keyword + spaces + str(value) + ";\n"
except FileNotFoundError:
print("File " + dictionary + " not found.\n")
def main():
solver,folder,np = command_line()
# folder check
if folder is not None:
except FileNotFoundError:
logger.error("Folder {} not found".format(folder))
folder = getcwd()
folder = abspath(getcwd())
casename = basename(getcwd())
logfilename = folder + '/' + casename + '.log'
statusfilename = '/service/status.html'
with open(logfilename,'w') as logfile:"Running decomposePar utility")
check_proc = Popen(["/app/check",folder])
monitor_proc = Popen(["/app/monitor",folder])"Running {} on {} processors".format(solver,str(np)))
run(["mpirun","-np",str(np),solver,"-parallel","-case",folder],stdout=logfile)"Running reconstructPar")
### in case of abort command, set the controlDict file in the original state
ctrlDict = folder + '/system/controlDict'
modify_dict(ctrlDict,"stopAt","endTime")"Calculation ended")
if __name__ == "__main__":
"""Ugly but working hard-coded test script for the HPC client"""
For upload and register, hpc_image upload image_name
For unregister and delete, hpc_image delete image_name
N.B. Images are uploaded in /home/images folder in anselm cluster
N.B. Folder /home/images has to exist on the cluster
N.B. Username, password and project should be filled with needed values
import os
import filecmp
import json
import clfpy as cf
import sys
auth_url = ""
hpc_url = ""
gss_url = ""
username = ""
password = ""
project = ''
print("Obtaining session token ...")
auth = cf.AuthClient(auth_url)
session_token = auth.get_session_token(username, project, password)
gss = cf.GssClient(gss_url)
hpc = cf.HpcImagesClient(hpc_url)
def upload():
print("Uploading and registering a new image")
image_filepath = sys.argv[2]
gss_ID = "it4i_anselm://home/images/" + sys.argv[2]
print(gss.upload(gss_ID, session_token, image_filepath))
print(hpc.register_image(session_token, sys.argv[2], gss_ID))
def delete():
print("Removing and deleting image")
print(hpc.delete_image(session_token, sys.argv[2]))
gss_ID = "it4i_anselm://home/images/" + sys.argv[2]
print(gss.delete(gss_ID, session_token))
def main():
if sys.argv[1] == "upload":
elif sys.argv[1] == "delete":
if __name__ == "__main__":
# ========= |
# \\ / F ield | OpenFOAM: The Open Source CFD Toolbox
# \\ / O peration |
# \\ / A nd | Copyright (C) 2011-2016 OpenFOAM Foundation
# \\/ M anipulation |
# License
# This file is part of OpenFOAM.
# OpenFOAM is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# OpenFOAM is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
# for more details.
# You should have received a copy of the GNU General Public License
# along with OpenFOAM. If not, see <>.
# File
# etc/
# Description