docs.it4i/software/viz/gpi2.md

+!!!warning
+    This page has not been updated yet. The page does not reflect the transition from PBS to Slurm.
+
+# GPI-2
+
+## Introduction
+
+Programming Next Generation Supercomputers: GPI-2 is an API library for asynchronous interprocess, cross-node communication. It provides a flexible, scalable, and fault tolerant interface for parallel applications.
+
+The GPI-2 library implements the GASPI specification ([Global Address Space Programming Interface][a]). GASPI is a Partitioned Global Address Space (PGAS) API. It aims at scalable, flexible, and failure-tolerant computing in massively parallel environments.
+
+## Modules
+
+For the current list of installed versions, use:
+
+```console
+$ ml av GPI-2
+```
+
+The module sets up environment variables required for linking and running GPI-2 enabled applications. This particular command loads the default `gpi2/1.0.2` module.
+
+## Linking
+
+!!! note
+    Link with -lGPI2 -libverbs
+
+Load the `gpi2` module. Link using `-lGPI2` and `-libverbs` switches to link your code against GPI-2. The GPI-2 requires the OFED InfiniBand communication library ibverbs.
+
+### Compiling and Linking With Intel Compilers
+
+```console
+$ ml intel
+$ ml gpi2
+$ icc myprog.c -o myprog.x -Wl,-rpath=$LIBRARY_PATH -lGPI2 -libverbs
+```
+
+### Compiling and Linking With GNU Compilers
+
+```console
+$ ml gcc
+$ ml gpi2
+$ gcc myprog.c -o myprog.x -Wl,-rpath=$LIBRARY_PATH -lGPI2 -libverbs
+```
+
+## Running the GPI-2 Codes
+
+!!! note
+    `gaspi_run` starts the GPI-2 application
+
+The `gaspi_run` utility is used to start and run GPI-2 applications:
+
+```console
+$ gaspi_run -m machinefile ./myprog.x
+```
+
+A machine file (** machinefile **) must be provided with the hostnames of nodes where the application will run. The machinefile lists all nodes on which to run, one entry per node per process. This file may be hand created or obtained from standard $PBS_NODEFILE:
+
+```console
+$ cut -f1 -d"." $PBS_NODEFILE > machinefile
+```
+
+machinefile:
+
+```console
+    cn79
+    cn80
+```
+
+This machinefile will run 2 GPI-2 processes, one on node cn79 and one on node cn80.
+
+machinefle:
+
+```console
+    cn79
+    cn79
+    cn80
+    cn80
+```
+
+This machinefile will run 4 GPI-2 processes, two on node cn79 and two on node cn80.
+
+!!! note
+    Use `mpiprocs` to control how many GPI-2 processes will run per node.
+
+Example:
+
+```console
+$ qsub -A OPEN-0-0 -q qexp -l select=2:ncpus=16:mpiprocs=16 -I
+```
+
+This example will produce $PBS_NODEFILE with 16 entries per node.
+
+### Gaspi_logger
+
+!!! note
+    `gaspi_logger` views the output from GPI-2 application ranks.
+
+The `gaspi_logger` utility is used to view the output from all nodes except the master node (rank 0). `gaspi_logger` is started, on another session, on the master node - the node where the `gaspi_run` is executed. The output of the application, when called with `gaspi_printf()`, will be redirected to the `gaspi_logger`. Other I/O routines (e.g. `printf`) will not.
+
+## Example
+
+Following is an example of GPI-2 enabled code:
+
+```cpp
+#include <GASPI.h>
+#include <stdlib.h>
+
+void success_or_exit ( const char* file, const int line, const int ec)
+{
+  if (ec != GASPI_SUCCESS)
+    {
+      gaspi_printf ("Assertion failed in %s[%i]:%dn", file, line, ec);
+      exit (1);
+    }
+}
+
+#define ASSERT(ec) success_or_exit (__FILE__, __LINE__, ec);
+
+int main(int argc, char *argv[])
+{
+  gaspi_rank_t rank, num;
+  gaspi_return_t ret;
+
+  /* Initialize GPI-2 */
+  ASSERT( gaspi_proc_init(GASPI_BLOCK) );
+
+  /* Get ranks information */
+  ASSERT( gaspi_proc_rank(&rank) );
+  ASSERT( gaspi_proc_num(&num) );
+
+  gaspi_printf("Hello from rank %d of %dn",
+           rank, num);
+
+  /* Terminate */
+  ASSERT( gaspi_proc_term(GASPI_BLOCK) );
+
+  return 0;
+}
+```
+
+Load the modules and compile:
+
+```console
+$ ml gcc gpi2
+$ gcc helloworld_gpi.c -o helloworld_gpi.x -Wl,-rpath=$LIBRARY_PATH -lGPI2 -libverbs
+```
+
+Submit the job and run the GPI-2 application:
+
+```console
+$ qsub -q qexp -l select=2:ncpus=1:mpiprocs=1,place=scatter,walltime=00:05:00 -I
+    qsub: waiting for job 171247.dm2 to start
+    qsub: job 171247.dm2 ready
+cn79 $ ml gpi2
+cn79 $ cut -f1 -d"." $PBS_NODEFILE > machinefile
+cn79 $ gaspi_run -m machinefile ./helloworld_gpi.x
+    Hello from rank 0 of 2
+```
+
+At the same time, in another session, you may start the GASPI logger:
+
+```console
+$ ssh cn79
+cn79 $ gaspi_logger
+    GASPI Logger (v1.1)
+    [cn80:0] Hello from rank 1 of 2
+```
+
+In this example, we compile the helloworld_gpi.c code using the **gnu compiler**(gcc) and link it to the GPI-2 and the ibverbs library. The library search path is compiled in. For execution, we use the qexp queue, 2 nodes 1 core each. The GPI module must be loaded on the master compute node (in this example cn79), `gaspi_logger` is used from a different session to view the output of the second process.
+
+[a]: http://www.gaspi.de/en/project.html

docs.it4i/software/viz/insitu.md

+# In Situ Visualization
+
+## Introduction
+
+In situ visualization allows you to visualize your data while your computation is progressing on multiple nodes of a cluster. It is a visualization pipeline based on the [ParaView Catalyst][a] library.
+
+To leverage the possibilities of the in situ visualization by Catalyst library, you have to write an adaptor code that will use the actual data from your simulation and process them in the way they can be passed to ParaView for visualization. We provide a simple example of such simulator/adaptor code that binds together to provide the in situ visualization.
+
+Detailed description of the Catalyst API can be found [here][b]. We restrict ourselves to provide more of an overall description of the code together with specifications for building, and explanation about how to run the code on the cluster.
+
+## Installed Version
+
+For the current list of installed versions, use:
+
+```console
+$ ml av CUDA
+```
+
+## Usage
+
+All code concerning the simulator/adaptor is available to download from [here][code]. It is a package with the following files: [CMakeLists.txt][cmakelist_txt], [FEAdaptor.h][feadaptor_h], [FEAdaptor.cxx][feadaptor_cxx], [FEDataStructures.h][fedatastructures_h], [FEDataStructures.cxx][fedatastructures_cxx], [FEDriver.cxx][fedriver_cxx], and [feslicescript.py][feslicescript].
+
+After the download, unpack the code:
+
+```console
+$ tar xvf package_name
+```
+
+Then use CMake to manage the build process, but before that, load the appropriate modules (`CMake`, `ParaView`):
+
+```console
+$ ml CMake ParaView/5.6.0-intel-2017a-mpi
+```
+
+This module set also loads a necessary Intel compiler within the dependencies:
+
+```console
+$ mkdir build
+$ cd build
+$ cmake ../
+```
+
+Now you can build the simulator/adaptor code by using `make`:
+
+```console
+$ make
+```
+
+It will generate the CxxFullExampleAdaptor executable file. This will be later run together with ParaView and it will provide the in situ visualization example.
+
+## Code Explanation
+
+The provided example is a simple MPI program. Main executing part is written in FEDriver.cxx. It is simulator code that creates computational grid and performs simulator/adaptor interaction (see below).
+
+Dimensions of the computational grid in terms of number of points in x, y, z directions are supplied as input parameters to the `main` function (see lines 22-24 in the code below). Based on the number of MPI ranks, each MPI process creates a different part of the overall grid. This is done by grid initialization (see line 30). The respective code for this is in FEDataStructures.cxx. The parameter nr. 4 in `main` is for the name of the Python script (we use feslicescript.py). It sets up the ParaView-Catalyst pipeline (see line 36). The simulation starts by linearly progressing the `timeStep` value in the `for` loop. Each iteration of the loop upates the grid attributes (Velocity and Pressure) by calling the `UpdateFields` method from the `Attributes` class. Next in the loop, the adaptor's `CoProcess` function is called by using actual parameters (`grid`, `time`). To provide some information about the state of the simulation, the actual time step is printed together with the name of the processor that handles the computation inside the allocated MPI rank. Before the loop ends, appropriate sleep time is used to give some time for visualization to update. After the simulation loop ends, clean up is done by calling the `Finalize` function on adaptor and `MPI_Finalize` on MPI processes.
+
+![](insitu/img/FEDriver.png "FEDriver.cxx")
+
+Adaptor's initialization performs several necessary steps, see the code below. It creates vtkCPProcessor using the Catalyst library and adds a pipeline to it. The pipeline is initialized by the reffered Python script:
+
+![](insitu/img/Initialize.png "Initialization of the adaptor, within FEAdaptor.cxx")
+
+To initialize the Catalyst pipeline, we use the feslicescript.py Python script. You enable the live visualization in here and set the proper connection port. You can also use another commands and functions to configure it for saving the data during the visualization or another tasks that are available from the ParaView environment. For more details, see the [Catalyst guide][catalyst_guide].
+
+![](insitu/img/feslicescript.png "Catalyst pipeline setup by Python script")
+
+The `UpdateFields` method from the `Attributes` class updates the `velocity` value in respect to the value of `time` and the value of `setting` which depends on the actual MPI rank, see the code below. This way, different processes can be visually distinguished during the simulation.
+
+![](insitu/img/UpdateFields.png "UpdateFields method of the Attributes class from FEDataStructures.cxx")
+
+As mentioned before, further in the simulation loop, the adaptor's `CoProcess` function is called by using actual parameters of the `grid`, `time`, and `timeStep`. In the function, proper representation and description of the data is created. Such data is then passed to the Processor that has been created during the adaptor's initialization. The code of the `CoProcess` function is shown below.
+
+![](insitu/img/CoProcess.png "CoProcess function of the adaptor, within FEAdaptor.cxx")
+
+### Launching the Simulator/Adaptor With ParaView
+
+There are two standard options to launch ParaView,. You can run ParaView from your local machine in client-server mode by following the information [here][2] or you can connect to the cluster using VNC and the graphical environment by following the information on [VNC][3]. In both cases, we will use ParaView version 5.6.0 and its respective module.
+
+Whether you use the client-server mode or VNC for ParaView, you have to allocate some computing resources. This is done by the console commands below (supply valid Project ID).
+
+For the client-server mode of ParaView we allocate the resources by
+
+```console
+$ salloc -p qcpu -A PROJECT_ID --nodes=2
+```
+
+In the case of VNC connection, we use X11 forwarding by the `--x11` option to allow the graphical environment on the interactive session:
+
+```console
+$ salloc -p qcpu -A PROJECT_ID --nodes=2 --x11
+```
+
+The issued console commands launch the interactive session on 2 nodes. This is the minimal setup to test that the simulator/adaptor code runs on multiple nodes.
+
+After the interactive session is opened, load the `ParaView` module:
+
+```console
+$ ml ParaView/5.6.0-intel-2017a-mpi
+```
+
+When the module is loaded and you run the client-server mode, launch the mpirun command for pvserver as used in the description for [ParaView client-server][2] but also use the `&` sign at the end of the command. Then you can use the console later for running the simulator/adaptor code. If you run the VNC session, after loading the ParaView module, set up the environmental parameter for correct keyboard input and then run the ParaView in the background using the `&` sign.
+
+```console
+$ export QT_XKB_CONFIG_ROOT=/usr/share/X11/xkb
+$ paraview &
+```
+
+If you have proceeded to the end in the description for the ParaView client-server mode and run ParaView localy or you have launched ParaView remotely using VNC the further steps are the same for both options. In the ParaView, go to the *Catalyst* -> *Connect* and hit *OK* in the pop up window asking for the connection port. After that, ParaView starts listening for incomming data to the in situ visualization.
+
+![](insitu/img/Catalyst_connect.png "Starting Catalyst connection in ParaView")
+
+Go to your build directory and run the built simulator/adaptor code from the console:
+
+```console
+mpirun -n 2 ./CxxFullExample 30 30 30 ../feslicescript.py
+```
+
+The programs starts to compute on the allocated nodes and prints out the response:
+
+![](insitu/img/Simulator_response.png "Simulator/adaptor response")
+
+In ParaView, you should see a new pipeline called *input* in the *Pipeline Browser*:
+
+![](insitu/img/Input_pipeline.png "Input pipeline")
+
+By clicking on it, another item called *Extract: input* is added:
+
+![](insitu/img/Extract_input.png "Extract: input")
+
+If you click on the eye icon to the left of the *Extract: input* item, the data will appear:
+
+![](insitu/img/Data_shown.png "Sent data")
+
+To visualize the velocity property on the geometry, go to the *Properties* tab and choose *velocity* instead of *Solid Color* in the *Coloring* menu:
+
+![](insitu/img/Show_velocity.png "Show velocity data")
+
+The result will look like in the image below, where different domains dependent on the number of allocated resources can be seen and they will progress through the time:
+
+![](insitu/img/Result.png "Velocity results")
+
+### Cleanup
+
+After you finish the in situ visualization, you should provide a proper cleanup.
+
+Terminate the simulation if it is still running.
+
+In the VNC session, close ParaView and the interactive job. Close the VNC client. Kill the VNC Server as described [here][3].
+
+In the client-server mode of ParaView, disconnect from the server in ParaView and close the interactive job.
+
+[1]: ../../../salomon/introduction/
+[2]: ../paraview/
+[3]: ../../../general/accessing-the-clusters/graphical-user-interface/vnc/
+
+[a]: https://www.paraview.org/in-situ/
+[b]: https://www.paraview.org/files/catalyst/docs/ParaViewCatalystUsersGuide_v2.pdf
+[c]: http://www.paraview.org/
+
+[code]: insitu/insitu.tar.gz
+[cmakelist_txt]: insitu/CMakeLists.txt
+[feadaptor_h]: insitu/FEAdaptor.h
+[feadaptor_cxx]: insitu/FEAdaptor.cxx
+[fedatastructures_h]: insitu/FEDataStructures.h
+[fedatastructures_cxx]: insitu/FEDataStructures.cxx
+[fedriver_cxx]: insitu/FEDriver.cxx
+[feslicescript]: insitu/feslicescript.py

docs.it4i/software/viz/insitu/CMakeLists.txt

+cmake_minimum_required(VERSION 3.3)
+project(CatalystCxxFullExample)
+
+set(USE_CATALYST ON CACHE BOOL "Link the simulator with Catalyst")
+if(USE_CATALYST)
+  find_package(ParaView 5.6 REQUIRED COMPONENTS vtkPVPythonCatalyst)
+  include("${PARAVIEW_USE_FILE}")
+  set(Adaptor_SRCS
+    FEAdaptor.cxx
+    )
+  add_library(CxxFullExampleAdaptor ${Adaptor_SRCS})
+  target_link_libraries(CxxFullExampleAdaptor vtkPVPythonCatalyst vtkParallelMPI)
+  add_definitions("-DUSE_CATALYST")
+  if(NOT PARAVIEW_USE_MPI)
+    message(SEND_ERROR "ParaView must be built with MPI enabled")
+  endif()
+else()
+  find_package(MPI REQUIRED)
+  include_directories(${MPI_C_INCLUDE_PATH})
+endif()
+
+add_executable(CxxFullExample FEDriver.cxx FEDataStructures.cxx)
+if(USE_CATALYST)
+  target_link_libraries(CxxFullExample LINK_PRIVATE CxxFullExampleAdaptor)
+  include(vtkModuleMacros)
+  include(vtkMPI)
+  vtk_mpi_link(CxxFullExample)
+else()
+  target_link_libraries(CxxFullExample LINK_PRIVATE ${MPI_LIBRARIES})
+endif()
+
+option(BUILD_TESTING "Build Testing" OFF)
+# Setup testing.
+if (BUILD_TESTING)
+  include(CTest)
+  add_test(NAME CxxFullExampleTest COMMAND CxxFullExample ${CMAKE_CURRENT_SOURCE_DIR}/feslicescript.py)
+  set_tests_properties(CxxFullExampleTest PROPERTIES LABELS "PARAVIEW;CATALYST")
+endif()

docs.it4i/software/viz/insitu/FEAdaptor.cxx

+#include "FEAdaptor.h"
+#include "FEDataStructures.h"
+#include <iostream>
+
+#include <vtkCPDataDescription.h>
+#include <vtkCPInputDataDescription.h>
+#include <vtkCPProcessor.h>
+#include <vtkCPPythonScriptPipeline.h>
+#include <vtkCellData.h>
+#include <vtkCellType.h>
+#include <vtkDoubleArray.h>
+#include <vtkFloatArray.h>
+#include <vtkNew.h>
+#include <vtkPointData.h>
+#include <vtkPoints.h>
+#include <vtkUnstructuredGrid.h>
+
+namespace
+{
+vtkCPProcessor* Processor = NULL;
+vtkUnstructuredGrid* VTKGrid;
+
+void BuildVTKGrid(Grid& grid)
+{
+  // create the points information
+  vtkNew<vtkDoubleArray> pointArray;
+  pointArray->SetNumberOfComponents(3);
+  pointArray->SetArray(
+    grid.GetPointsArray(), static_cast<vtkIdType>(grid.GetNumberOfPoints() * 3), 1);
+  vtkNew<vtkPoints> points;
+  points->SetData(pointArray.GetPointer());
+  VTKGrid->SetPoints(points.GetPointer());
+
+  // create the cells
+  size_t numCells = grid.GetNumberOfCells();
+  VTKGrid->Allocate(static_cast<vtkIdType>(numCells * 9));
+  for (size_t cell = 0; cell < numCells; cell++)
+  {
+    unsigned int* cellPoints = grid.GetCellPoints(cell);
+    vtkIdType tmp[8] = { cellPoints[0], cellPoints[1], cellPoints[2], cellPoints[3], cellPoints[4],
+      cellPoints[5], cellPoints[6], cellPoints[7] };
+    VTKGrid->InsertNextCell(VTK_HEXAHEDRON, 8, tmp);
+  }
+}
+
+void UpdateVTKAttributes(Grid& grid, Attributes& attributes, vtkCPInputDataDescription* idd)
+{
+  if (idd->IsFieldNeeded("velocity", vtkDataObject::POINT) == true)
+  {
+    if (VTKGrid->GetPointData()->GetNumberOfArrays() == 0)
+    {
+      // velocity array
+      vtkNew<vtkDoubleArray> velocity;
+      velocity->SetName("velocity");
+      velocity->SetNumberOfComponents(3);
+      velocity->SetNumberOfTuples(static_cast<vtkIdType>(grid.GetNumberOfPoints()));
+      VTKGrid->GetPointData()->AddArray(velocity.GetPointer());
+    }
+    vtkDoubleArray* velocity =
+      vtkDoubleArray::SafeDownCast(VTKGrid->GetPointData()->GetArray("velocity"));
+    // The velocity array is ordered as vx0,vx1,vx2,..,vy0,vy1,vy2,..,vz0,vz1,vz2,..
+    // so we need to create a full copy of it with VTK's ordering of
+    // vx0,vy0,vz0,vx1,vy1,vz1,..
+    double* velocityData = attributes.GetVelocityArray();
+    vtkIdType numTuples = velocity->GetNumberOfTuples();
+    for (vtkIdType i = 0; i < numTuples; i++)
+    {
+      double values[3] = { velocityData[i], velocityData[i + numTuples],
+        velocityData[i + 2 * numTuples] };
+      velocity->SetTypedTuple(i, values);
+    }
+  }
+  if (idd->IsFieldNeeded("pressure", vtkDataObject::CELL) == true)
+  {
+    if (VTKGrid->GetCellData()->GetNumberOfArrays() == 0)
+    {
+      // pressure array
+      vtkNew<vtkFloatArray> pressure;
+      pressure->SetName("pressure");
+      pressure->SetNumberOfComponents(1);
+      VTKGrid->GetCellData()->AddArray(pressure.GetPointer());
+    }
+    vtkFloatArray* pressure =
+      vtkFloatArray::SafeDownCast(VTKGrid->GetCellData()->GetArray("pressure"));
+    // The pressure array is a scalar array so we can reuse
+    // memory as long as we ordered the points properly.
+    float* pressureData = attributes.GetPressureArray();
+    pressure->SetArray(pressureData, static_cast<vtkIdType>(grid.GetNumberOfCells()), 1);
+  }
+}
+
+void BuildVTKDataStructures(Grid& grid, Attributes& attributes, vtkCPInputDataDescription* idd)
+{
+  if (VTKGrid == NULL)
+  {
+    // The grid structure isn't changing so we only build it
+    // the first time it's needed. If we needed the memory
+    // we could delete it and rebuild as necessary.
+    VTKGrid = vtkUnstructuredGrid::New();
+    BuildVTKGrid(grid);
+  }
+  UpdateVTKAttributes(grid, attributes, idd);
+}
+}
+
+namespace FEAdaptor
+{
+
+void Initialize(char* script)
+{
+  if (Processor == NULL)
+  {
+    Processor = vtkCPProcessor::New();
+    Processor->Initialize();
+  }
+  else
+  {
+    Processor->RemoveAllPipelines();
+  }
+  
+  vtkNew<vtkCPPythonScriptPipeline> pipeline;
+  pipeline->Initialize(script);
+  Processor->AddPipeline(pipeline.GetPointer()); 
+}
+
+void Finalize()
+{
+  if (Processor)
+  {
+    Processor->Delete();
+    Processor = NULL;
+  }
+  if (VTKGrid)
+  {
+    VTKGrid->Delete();
+    VTKGrid = NULL;
+  }
+}
+
+void CoProcess(
+  Grid& grid, Attributes& attributes, double time, unsigned int timeStep, bool lastTimeStep)
+{
+  vtkNew<vtkCPDataDescription> dataDescription;
+  dataDescription->AddInput("input");
+  dataDescription->SetTimeData(time, timeStep);
+  if (lastTimeStep == true)
+  {
+    // assume that we want to execute all the pipelines if it
+    // is the last time step.
+    dataDescription->ForceOutputOn();
+  }
+  if (Processor->RequestDataDescription(dataDescription.GetPointer()) != 0)
+  {
+    vtkCPInputDataDescription* idd = dataDescription->GetInputDescriptionByName("input");
+    BuildVTKDataStructures(grid, attributes, idd);
+    idd->SetGrid(VTKGrid);
+    Processor->CoProcess(dataDescription.GetPointer());
+  }
+}
+} // end of Catalyst namespace

docs.it4i/software/viz/insitu/FEAdaptor.h

+#ifndef FEADAPTOR_HEADER
+#define FEADAPTOR_HEADER
+
+class Attributes;
+class Grid;
+
+namespace FEAdaptor
+{
+void Initialize(char* script);
+
+void Finalize();
+
+void CoProcess(
+  Grid& grid, Attributes& attributes, double time, unsigned int timeStep, bool lastTimeStep);
+}
+
+#endif

docs.it4i/software/viz/insitu/FEDataStructures.cxx

+#include "FEDataStructures.h"
+
+#include <iostream>
+#include <mpi.h>
+
+Grid::Grid()
+{
+}
+
+void Grid::Initialize(const unsigned int numPoints[3], const double spacing[3])
+{
+  if (numPoints[0] == 0 || numPoints[1] == 0 || numPoints[2] == 0)
+  {
+    std::cerr << "Must have a non-zero amount of points in each direction.\n";
+  }
+  // in parallel, we do a simple partitioning in the x-direction.
+  int mpiSize = 1;
+  int mpiRank = 0;
+  MPI_Comm_rank(MPI_COMM_WORLD, &mpiRank);
+  MPI_Comm_size(MPI_COMM_WORLD, &mpiSize);
+
+  unsigned int startXPoint = mpiRank * numPoints[0] / mpiSize;
+  unsigned int endXPoint = (mpiRank + 1) * numPoints[0] / mpiSize;
+  if (mpiSize != mpiRank + 1)
+  {
+    endXPoint++;
+  }
+
+  // create the points -- slowest in the x and fastest in the z directions
+  double coord[3] = { 0, 0, 0 };
+  for (unsigned int i = startXPoint; i < endXPoint; i++)
+  {
+    coord[0] = i * spacing[0];
+    for (unsigned int j = 0; j < numPoints[1]; j++)
+    {
+      coord[1] = j * spacing[1];
+      for (unsigned int k = 0; k < numPoints[2]; k++)
+      {
+        coord[2] = k * spacing[2];
+        // add the coordinate to the end of the vector
+        std::copy(coord, coord + 3, std::back_inserter(this->Points));
+      }
+    }
+  }
+  // create the hex cells
+  unsigned int cellPoints[8];
+  unsigned int numXPoints = endXPoint - startXPoint;
+  for (unsigned int i = 0; i < numXPoints - 1; i++)
+  {
+    for (unsigned int j = 0; j < numPoints[1] - 1; j++)
+    {
+      for (unsigned int k = 0; k < numPoints[2] - 1; k++)
+      {
+        cellPoints[0] = i * numPoints[1] * numPoints[2] + j * numPoints[2] + k;
+        cellPoints[1] = (i + 1) * numPoints[1] * numPoints[2] + j * numPoints[2] + k;
+        cellPoints[2] = (i + 1) * numPoints[1] * numPoints[2] + (j + 1) * numPoints[2] + k;
+        cellPoints[3] = i * numPoints[1] * numPoints[2] + (j + 1) * numPoints[2] + k;
+        cellPoints[4] = i * numPoints[1] * numPoints[2] + j * numPoints[2] + k + 1;
+        cellPoints[5] = (i + 1) * numPoints[1] * numPoints[2] + j * numPoints[2] + k + 1;
+        cellPoints[6] = (i + 1) * numPoints[1] * numPoints[2] + (j + 1) * numPoints[2] + k + 1;
+        cellPoints[7] = i * numPoints[1] * numPoints[2] + (j + 1) * numPoints[2] + k + 1;
+        std::copy(cellPoints, cellPoints + 8, std::back_inserter(this->Cells));
+      }
+    }
+  }
+}
+
+size_t Grid::GetNumberOfPoints()
+{
+  return this->Points.size() / 3;
+}
+
+size_t Grid::GetNumberOfCells()
+{
+  return this->Cells.size() / 8;
+}
+
+double* Grid::GetPointsArray()
+{
+  if (this->Points.empty())
+  {
+    return NULL;
+  }
+  return &(this->Points[0]);
+}
+
+double* Grid::GetPoint(size_t pointId)
+{
+  if (pointId >= this->Points.size())
+  {
+    return NULL;
+  }
+  return &(this->Points[pointId * 3]);
+}
+
+unsigned int* Grid::GetCellPoints(size_t cellId)
+{
+  if (cellId >= this->Cells.size())
+  {
+    return NULL;
+  }
+  return &(this->Cells[cellId * 8]);
+}
+
+Attributes::Attributes()
+{
+  this->GridPtr = NULL;
+}
+
+void Attributes::Initialize(Grid* grid)
+{
+  this->GridPtr = grid;
+}
+
+void Attributes::UpdateFields(double time)
+{
+  size_t  numPoints = this->GridPtr->GetNumberOfPoints();
+  this->Velocity.resize(numPoints * 3);
+  
+  // provide different update setting for different parallel process
+  int mpiSize = 1;
+  int mpiRank = 0;
+  MPI_Comm_rank(MPI_COMM_WORLD, &mpiRank);
+  MPI_Comm_size(MPI_COMM_WORLD, &mpiSize);
+  double setting = 1.0 + (double) mpiRank / (double) mpiSize;
+  
+  for (size_t pt = 0; pt < numPoints; pt++)
+  {
+    double* coord = this->GridPtr->GetPoint(pt);
+    this->Velocity[pt] = coord[1] * time * setting;
+  }
+  std::fill(this->Velocity.begin() + numPoints, this->Velocity.end(), 0.0);
+
+  size_t numCells = this->GridPtr->GetNumberOfCells();
+  this->Pressure.resize(numCells);
+  std::fill(this->Pressure.begin(), this->Pressure.end(), setting);
+}
+
+double* Attributes::GetVelocityArray()
+{
+  if (this->Velocity.empty())
+  {
+    return NULL;
+  }
+  return &this->Velocity[0];
+}
+
+float* Attributes::GetPressureArray()
+{
+  if (this->Pressure.empty())
+  {
+    return NULL;
+  }
+  return &this->Pressure[0];
+}

docs.it4i/software/viz/insitu/FEDataStructures.h

+#ifndef FEDATASTRUCTURES_HEADER
+#define FEDATASTRUCTURES_HEADER
+
+#include <cstddef>
+#include <vector>
+
+class Grid
+{
+public:
+  Grid();
+  void Initialize(const unsigned int numPoints[3], const double spacing[3]);
+  size_t GetNumberOfPoints();
+  size_t GetNumberOfCells();
+  double* GetPointsArray();
+  double* GetPoint(size_t pointId);
+  unsigned int* GetCellPoints(size_t cellId);
+
+private:
+  std::vector<double> Points;
+  std::vector<unsigned int> Cells;
+};
+
+class Attributes
+{
+  // A class for generating and storing point and cell fields.
+  // Velocity is stored at the points and pressure is stored
+  // for the cells.
+
+public:
+  Attributes();
+  void Initialize(Grid* grid);
+  void UpdateFields(double time);
+  double* GetVelocityArray();
+  float* GetPressureArray();
+
+private:
+  std::vector<double> Velocity;
+  std::vector<float> Pressure;
+  Grid* GridPtr;
+};
+#endif

docs.it4i/software/viz/insitu/FEDriver.cxx

+#include "FEDataStructures.h"
+#include <mpi.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <iostream>
+#include <stdlib.h>
+
+#ifdef USE_CATALYST
+#include "FEAdaptor.h"
+#endif
+
+int main(int argc, char** argv)
+{
+  // Check the input arguments
+  if (argc < 5) {
+	printf("Not all arguments supplied (grid definition, Python script name)\n");
+	return 0;
+  }
+
+  unsigned int pointsX = abs(std::stoi(argv[1])); 
+  unsigned int pointsY = abs(std::stoi(argv[2]));
+  unsigned int pointsZ = abs(std::stoi(argv[3]));
+  
+  // MPI_Init(&argc, &argv);
+  MPI_Init(NULL, NULL);
+  Grid grid;
+
+  unsigned int numPoints[3] = { pointsX, pointsY, pointsZ };
+  double spacing[3] = { 1, 1.1, 1.3 };
+  grid.Initialize(numPoints, spacing);
+  Attributes attributes;
+  attributes.Initialize(&grid);
+
+#ifdef USE_CATALYST
+  // The argument nr. 4 is the Python script name
+  FEAdaptor::Initialize(argv[4]);
+#endif
+  unsigned int numberOfTimeSteps = 1000;
+  for (unsigned int timeStep = 0; timeStep < numberOfTimeSteps; timeStep++)
+  {
+    // Use a time step of length 0.1
+    double time = timeStep * 0.1;
+    attributes.UpdateFields(time);
+#ifdef USE_CATALYST
+    FEAdaptor::CoProcess(grid, attributes, time, timeStep, timeStep == numberOfTimeSteps - 1);
+#endif
+    
+    // Get the name of the processor
+    char processor_name[MPI_MAX_PROCESSOR_NAME];
+    int name_len;
+    MPI_Get_processor_name(processor_name, &name_len);
+
+    // Print actual time step and processor name that handles the calculation
+    printf("This is processor %s, time step: %0.3f\n", processor_name, time);
+    usleep(500000);
+  }
+
+#ifdef USE_CATALYST
+  FEAdaptor::Finalize();
+#endif
+  MPI_Finalize();
+
+  return 0;
+}

docs.it4i/software/viz/insitu/feslicescript.py

+from paraview.simple import *
+from paraview import coprocessing
+
+#--------------------------------------------------------------
+# Code generated from cpstate.py to create the CoProcessor.
+
+
+# ----------------------- CoProcessor definition -----------------------
+
+def CreateCoProcessor():
+  def _CreatePipeline(coprocessor, datadescription):
+    class Pipeline:
+      coprocessor.CreateProducer( datadescription, "input" )
+
+    return Pipeline()
+
+  class CoProcessor(coprocessing.CoProcessor):
+    def CreatePipeline(self, datadescription):
+      self.Pipeline = _CreatePipeline(self, datadescription)
+
+  coprocessor = CoProcessor()
+  freqs = {'input': []}
+  coprocessor.SetUpdateFrequencies(freqs)
+  return coprocessor
+
+#--------------------------------------------------------------
+# Global variables that will hold the pipeline for each timestep
+# Creating the CoProcessor object, doesn't actually create the ParaView pipeline.
+# It will be automatically setup when coprocessor.UpdateProducers() is called the
+# first time.
+coprocessor = CreateCoProcessor()
+
+#--------------------------------------------------------------
+# Enable Live-Visualizaton with ParaView
+coprocessor.EnableLiveVisualization(True)
+
+
+# ---------------------- Data Selection method ----------------------
+
+def RequestDataDescription(datadescription):
+    "Callback to populate the request for current timestep"
+    global coprocessor
+    if datadescription.GetForceOutput() == True:
+        # We are just going to request all fields and meshes from the simulation
+        # code/adaptor.
+        for i in range(datadescription.GetNumberOfInputDescriptions()):
+            datadescription.GetInputDescription(i).AllFieldsOn()
+            datadescription.GetInputDescription(i).GenerateMeshOn()
+        return
+
+    # setup requests for all inputs based on the requirements of the
+    # pipeline.
+    coprocessor.LoadRequestedData(datadescription)
+
+# ------------------------ Processing method ------------------------
+
+def DoCoProcessing(datadescription):
+    "Callback to do co-processing for current timestep"
+    global coprocessor
+
+    # Update the coprocessor by providing it the newly generated simulation data.
+    # If the pipeline hasn't been setup yet, this will setup the pipeline.
+    coprocessor.UpdateProducers(datadescription)
+
+    # Write output data, if appropriate.
+    coprocessor.WriteData(datadescription);
+
+    # Write image capture (Last arg: rescale lookup table), if appropriate.
+    coprocessor.WriteImages(datadescription, rescale_lookuptable=False)
+
+    # Live Visualization, if enabled.
+    coprocessor.DoLiveVisualization(datadescription, "localhost", 22222)