Skip to content
Snippets Groups Projects
Select Git revision
  • master default protected
1 result
Compare
Name Last commit Last update
OpenFOAM-v1612+
ThirdParty-v1612+
meric
radar
reports
FOAMMODULES-GCC-Salomon.sh
FOAMMODULES-Intel-Salomon.sh
FOAMMODULES-Intel-Taurus.sh
README.md
README.md

1] Repository content

  • OpenFOAM-v1612+
  • ThirdParty-v1612+
  • meric - it is not a copy of meric repository, only wscript - more in section Meric compilation
  • radar - radar config
  • reports - measurement results, also the graph of the measured regions is there in the img directory.

2] Compilation

  • edit etc/bashrc
  • edit Make/options in all derecotories where you want to use Meric - currently edited:
    • OpenFOAM: src/OpenFOAM/Make/options
    • OpenFOAM: src/Pstream/mpi/Make/options
    • icoFoam: applications/solvers/incompressible/icoFoam/Make/options
    • simpleFoam: applications/solvers/incompressible/simpleFoam/Make/options
  • ./Allwmake

Environment

To set the invironment settings source the set_environment.sh file. This script souces the OpenFOAM bashrc, sets paths and loads this list of modules on TUD Taurus:

module load boost/1.61.0-intel2017.0.020-intelmpi2017
module load zlib
module load flex
module load cmake
module load hdeem
module load papi/5.4.1

On IT4I Salomon is possible to load:

module load OpenMPI/1.10.2-GCC-5.3.0-2.26
module load CMake
module load zlib
module load flex
module load Automake

Meric compilation

For the OpenFOAM measurement it is necessary to compile Meric without the OpenMP. It requiers small changes in the Meric wscript file. To avoid the modication of the wscript, an allready modified wscript is placed in this repository in the meric direcory. Replace the Meric wscipt by this file and compile the Meric. How to compile and use meric read its README.

OpenFOAM compilations

Do not compile the OpenFOAM on a login node. It requires all cores to compile faster.

To compile the OpenFOAM, run the Allwmake. If you just did some changes that effects code subset, look for a Make folder. In the direcory whre the Make dir is, you may use wmake command to compile content of this directory.

OpenFOAM compilation with Meric

This OpenFOAM repository allready contains some Meric probes and initializatiom. The initialization is inserted after/before MPI initi/finalize (inside src/Pstream/mpi/UPstream.C), so you don't have to add these other parts of code.

To add probes into your selected code, check if the Init was allredy called. To compile this code edit Make/options file, with paths to Meric and all its libraries.

Changes that were done to compile the code on Taurus

Hopefully this is a complete list of changes:

  • inside etc/bashrc:
    • FOAM_INST_DIR=
      HOME/
      WM_PROJECT
    • export WM_COMPILER=Icc
    • export WM_MPLIB=INTELMPI
  • patchs from the PATCH directory were applied
  • in the /ThirdParty-v1612+/scotch_6.0.3/src/Makefile.inc the flag "-shared-intel" was added to LDFLAGS
  • environment varible $BOOST_ARCH_PATH was set to system boost $BOOST_ROOT

Instrumentation

In the repository the simpleFoam solver, PBiCG, GAMG and some intern OpenFOAM functions were instrumented. The Meric probes are located in this source files:

  • src/OpenFOAM/matrices/lduMatrix/solvers/GAMG.C

  • src/OpenFOAM/matrices/lduMatrix/solvers/PBiCG.C

  • src/OpenFOAM/matrices/lduMatrix/lduMatrixATmul.C

  • applications/solvers/incompressible/simpleFoam/simplFoam.C

  • applications/solvers/incompressible/icoFoam/icoFoam.C

  • src/OpenFOAM/Make/options

  • src/Pstream/mpi/Make/options

  • applications/solvers/incompressible/icoFoam/Make/options

  • applications/solvers/incompressible/simpleFoam/Make/options

The tests were done on simpleFoam motorbike example. The icoFoam solver was not tested. To see the complete list of regions see the yEd graph located in the reports/img directory.

3] Running the tests

The simpleFoam motorBike example is ready to use. For measurement the runFOAM.sh script is present. Edit the system directory:

  • blockMeshDict - block parametres
  • controlDict - simulation timesteps
  • decomposeParDict - system decomposition
  • forceCoeffs - force coefficients
  • fvSolution - solvers selection
  • and other files I didn't change

For the first run of the example Allrun must be called (edit $decompDict variable inside to set path to your decomposeParDict). To run the code once: srun -n 24 simpleFoam -parallel