1] Repository content

• OpenFOAM-v1612+
• ThirdParty-v1612+
• meric - it is not a copy of meric repository, only wscript - more in section Meric compilation
• radar - radar config
• reports - measurement results, also the graph of the measured regions is there in the img directory.

2] Compilation

In general user should edit etc/bashrc, source the file and use Allwmake in root directory to compile. The code has been already compiled on Taurus and Salomon machines, so you don't have to search for modules to compile, all you have to do is source FOAMMODULES-COMPILER-MACHINE.sh file (this script not only loads modules and sets all necessary environment variables, but also edits OpenFOAM compilation paths), switch to OpenFOAM root dir (use foam command) and compile the code using ./Allwmake.

Before the first compilation Meric library should be downloaded to meric directory. Even if one doesn't want to use MERIC instrumentation, the Meric repository is needed to provide readex header file. Edit FOAMMODULES-COMPILER-MACHINE.sh file to swith to USE_SCOREP.

Meric compilation

For the OpenFOAM measurement it is necessary to compile Meric without the OpenMP. It requiers small changes in the Meric wscript file. To avoid the modication of the wscript, an allready modified wscript is placed in this repository in the meric direcory. Replace the Meric wscipt by this file and compile the Meric. How to compile and use Meric read its README.

OpenFOAM compilations notes

• Rename this repository as "OpenFOAM" and place it to you home directory.
• Do not compile the OpenFOAM on a login node. OpenFOAM compilation is parallel and takes long time.
• To compile the OpenFOAM, run the Allwmake. If you just did some changes that effects code subset, look for a Make folder. In the direcory where the Make dir is, you may use wclean and wmake command to clean and compile content of this directory.

OpenFOAM compilation with READEX manual probes

This OpenFOAM repository allready contains some READEX probes and initializatiom. The initialization is inserted after/before MPI initi/finalize (inside src/Pstream/mpi/UPstream.C), so you don't have to add these to other parts of code.

To add probes into your selected code, check if the Init was allredy called. To compile this code edit Make/options file. Currently edited configuration files are listed in following section Instrumentation.

Instrumentation

In the repository the simpleFoam solver, PBiCG, GAMG and some intern OpenFOAM functions were instrumented.

READEX INIT and CLOSE:

• OpenFOAM-v1612+/src/Pstream/mpi/UPstream.C

READEX MANUAL probes are located in this source files:

• src/OpenFOAM/matrices/lduMatrix/solvers/GAMG.C
• src/OpenFOAM/matrices/lduMatrix/solvers/PBiCG.C
• src/OpenFOAM/matrices/lduMatrix/lduMatrixATmul.C
• applications/solvers/incompressible/simpleFoam/simplFoam.C
• applications/solvers/incompressible/icoFoam/icoFoam.C

READEX compilation options:

• wmake/rules/XXX/general
• src/OpenFOAM/Make/options
• src/Pstream/mpi/Make/options
• applications/solvers/incompressible/icoFoam/Make/options
• applications/solvers/incompressible/simpleFoam/Make/options

The tests were done on simpleFoam motorbike example. The icoFoam solver was not tested. To see the complete list of regions see the yEd graph located in the reports/img directory (list of regions in the graph doesn't correspond to current situation in 100% but it should give you some insight). In icoFoam and simpleFoam case the READEX phase region is called "iteration".

Changes that were done to compile the code on Taurus

Hopefully this is a complete list of changes:

• Inside etc/bashrc:
  • FOAM_INST_DIR=
    HOME/
    WM_PROJECT
  • export WM_COMPILER=Icc
  • export WM_MPLIB=INTELMPI
• Patchs from the PATCH directory were applied
• In the /ThirdParty-v1612+/scotch_6.0.3/src/Makefile.inc the flag "-shared-intel" was added to LDFLAGS
• Environment varible $BOOST_ARCH_PATH was set to system boost $BOOST_ROOT
• The compilation on Taurus requires flag -DENDKILL to call abort at the end of the runtime. Everythings works fine, however it stucks at the end of the run if abort is not used.

3] Tests

The simpleFoam motorBike example is ready to use. For measurement the runFOAM.sh script is present. Edit the system directory:

• blockMeshDict - block parameters
• controlDict - simulation timesteps
• decomposeParDict - system decomposition
• forceCoeffs - force coefficients
• fvSolution - solvers selection
• and other files I didn't change

For the first run of the example Allrun must be called (edit $decompDict variable inside to set path to your decomposeParDict). To run the code once: srun -n 24 simpleFoam -parallel

4] READEX DTA Test

NOTE: scorep can handle 4GB RAM only, large domain makes problem. If you need to reduce the domain much use PCG solver (with preconditioner) to compute pressure on 24 processes (GAMG needs larger domain) = edit system/fvSolution.

DIRECTORIES

• compilation scripts OpenFOAM/readex_scripts/
• test directory with simpleFoam test Motorbike OpenFOAM/OpenFOAM-v1612+/tutorials/incompressible/simpleFoam/motorBike

WORKFLOW

1] Autofilter

• compile OpenFOAM using run_compile_1_autofilter.sh
• chage dir to test directory
• run scorep-autofilter using do_scorep_autofilter_loop.sh

2] Dyndetect

• compile OpenFOAM using run_compile_2_dyndetect.sh
• chage dir to test directory
• run scorep-autofilter using do_readex_dyndetect.sh

3] PTF

• run PTF using do_ptf.sh

  ...

4] RRL

• ?

• For manual instrumentation (--nocompiler) use using run_compile_3_dyndetect_manual.sh, however this option doesn't work now.