Skip to content
Snippets Groups Projects
Select Git revision
  • master default protected
1 result
Compare
xvysoc01's avatar
cleaning
Ondrej Vysocky authored
2631ef5a
History
xvysoc01's avatar 2631ef5a
History
Name Last commit Last update
OpenFOAM-v1612+
ThirdParty-v1612+
meric
radar
reports
.gitignore
FOAMMODULES-GCC-Salomon.sh
FOAMMODULES-Intel-Salomon.sh
FOAMMODULES-Intel-Taurus.sh
README.md
README.md

1] Repository content

  • OpenFOAM-v1612+
  • ThirdParty-v1612+
  • meric - it is not a copy of meric repository, only wscript - more in section Meric compilation
  • radar - radar config
  • reports - measurement results, also the graph of the measured regions is there in the img directory.

2] Compilation

In general user should edit etc/bashrc, source the file and use Allwmake in root directory to compile. The code has been already compiled on Taurus and Salomon machines, so you don't have to search for modules to compile, all you have to do is source FOAMMODULES-COMPILER-MACHINE.sh file (this script not only loads modules and sets all necessary environment variables, but also edits OpenFOAM compilation paths), switch to OpenFOAM root dir (use foam command) and compile the code using ./Allwmake.

Before the first compilation Meric library should be downloaded to meric directory. Even if one doesn't want to use MERIC instrumentation, the Meric repository is needed to provide readex header file. Edit FOAMMODULES-COMPILER-MACHINE.sh file to swith to USE_SCOREP.

Meric compilation

For the OpenFOAM measurement it is necessary to compile Meric without the OpenMP. It requiers small changes in the Meric wscript file. To avoid the modication of the wscript, an allready modified wscript is placed in this repository in the meric direcory. Replace the Meric wscipt by this file and compile the Meric. How to compile and use Meric read its README.

OpenFOAM compilations notes

  • Rename this repository as "OpenFOAM" and place it to you home directory.
  • Do not compile the OpenFOAM on a login node. OpenFOAM compilation is parallel and takes long time.
  • To compile the OpenFOAM, run the Allwmake. If you just did some changes that effects code subset, look for a Make folder. In the direcory where the Make dir is, you may use wclean and wmake command to clean and compile content of this directory.

OpenFOAM compilation with READEX manual probes

This OpenFOAM repository allready contains some READEX probes and initializatiom. The initialization is inserted after/before MPI initi/finalize (inside src/Pstream/mpi/UPstream.C), so you don't have to add these to other parts of code.

To add probes into your selected code, check if the Init was allredy called. To compile this code edit Make/options file. Currently edited configuration files are listed in following section Instrumentation.

Instrumentation

In the repository the simpleFoam solver, PBiCG, GAMG and some intern OpenFOAM functions were instrumented.

READEX INIT and CLOSE:

  • OpenFOAM-v1612+/src/Pstream/mpi/UPstream.C

READEX MANUAL probes are located in this source files:

  • src/OpenFOAM/matrices/lduMatrix/solvers/GAMG.C
  • src/OpenFOAM/matrices/lduMatrix/solvers/PBiCG.C
  • src/OpenFOAM/matrices/lduMatrix/lduMatrixATmul.C
  • applications/solvers/incompressible/simpleFoam/simplFoam.C
  • applications/solvers/incompressible/icoFoam/icoFoam.C

READEX compilation options:

  • wmake/rules/XXX/general
  • src/OpenFOAM/Make/options
  • src/Pstream/mpi/Make/options
  • applications/solvers/incompressible/icoFoam/Make/options
  • applications/solvers/incompressible/simpleFoam/Make/options

The tests were done on simpleFoam motorbike example. The icoFoam solver was not tested. To see the complete list of regions see the yEd graph located in the reports/img directory (list of regions in the graph doesn't correspond to current situation in 100% but it should give you some insight). In icoFoam and simpleFoam case the READEX phase region is called "iteration".

Changes that were done to compile the code on Taurus

Hopefully this is a complete list of changes:

  • Inside etc/bashrc:
    • FOAM_INST_DIR=
      HOME/
      WM_PROJECT
    • export WM_COMPILER=Icc
    • export WM_MPLIB=INTELMPI
  • Patchs from the PATCH directory were applied
  • In the /ThirdParty-v1612+/scotch_6.0.3/src/Makefile.inc the flag "-shared-intel" was added to LDFLAGS
  • Environment varible $BOOST_ARCH_PATH was set to system boost $BOOST_ROOT
  • The compilation on Taurus requires flag -DENDKILL to call abort at the end of the runtime. Everythings works fine, however it stucks at the end of the run if abort is not used.

3] Tests

The simpleFoam motorBike example is ready to use. For measurement the runFOAM.sh script is present. Edit the system directory:

  • blockMeshDict - block parameters
  • controlDict - simulation timesteps
  • decomposeParDict - system decomposition
  • forceCoeffs - force coefficients
  • fvSolution - solvers selection
  • and other files I didn't change

For the first run of the example Allrun must be called (edit $decompDict variable inside to set path to your decomposeParDict). To run the code once: srun -n 24 simpleFoam -parallel