README.md

# Instrument API #

Shared C API for HPC applications performance analysis and auto-tuning.

Instrument.h header file is based on preprocessor define directives, which allows with single manual application instrumentation insert annotation for several libraries. User selects at the compilation time which tool's instrumentation should be inserted, as well as it is possible remove the instrumentation at all. 

The functionality of a selected library, that is not covered by these functions, must be inserted in its original way, the shared API will remove them (the functions declaration are defined with no definition).

--------------------------------------------------------------------------------
#     List of supported libraries                                              #
--------------------------------------------------------------------------------
 * [MERIC](https://code.it4i.cz/energy-efficiency/instrumentapi#meric-api), compile with `-DUSE_MERIC`
 * [TIMEPROF](https://code.it4i.cz/energy-efficiency/instrumentapi#timeprof-api), compile with `-DUSE_TIMEPROF`
 * [Score-P](https://code.it4i.cz/energy-efficiency/instrumentapi#score-p-api), compile with `-DUSE_SCOREP_MANUAL` or `-DUSE_SCOREP` (phase region inserted only)
 * [Scalasca](https://code.it4i.cz/energy-efficiency/instrumentapi#score-p-api), compile with `-DUSE_SCOREP_MANUAL`, uses Score-P instrumentation
 * [TAU](https://code.it4i.cz/energy-efficiency/instrumentapi#tau-api), compile with `-DUSE_TAU`
 * [GEOPM](https://code.it4i.cz/energy-efficiency/instrumentapi#geopm-api), compile with `-DUSE_GEOPM`
 * [mpiP](https://code.it4i.cz/energy-efficiency/instrumentapi#mpip-api), compile with `-DUSE_MPIP`
 * [Extrae](https://code.it4i.cz/energy-efficiency/instrumentapi#extrae-api), compile with `-DUSE_EXTRAE`
 * [LIKWID](https://code.it4i.cz/energy-efficiency/instrumentapi#likwid-api), compile with `-DUSE_LIKWID`, in case of OpenMP applications, include the header after omp.h
 * Alternatively, it is possible to compile with `-DUSE_TIME_MEASUREMENT` to get just runtime measurement of the instrumented regions. C applications compile with `-DINSTRUMENT_STACK_SIZE=20` to specify nesting region level (default 20).

When none of mentioned compiler flags is used, no instrumentation is inserted. List of supported libraries is based on [OpenHPC](https://openhpc.community/) and [READEX](https://www.readex.eu/) projects. The list can be easily exteded for any library, just by providing your definition of the shared API functions.

--------------------------------------------------------------------------------
#     List of functions                                                        #
--------------------------------------------------------------------------------
* `void INSTRUMENT_INIT(nodeId)`
	* library initialization
* `void INSTRUMENT_CLOSE()`
	* end of use of the library 
* `void INSTRUMENT_REGION_DEFINE(handle, name, GEOPMhint)`
	* region definition before it can start
* `void INSTRUMENT_PHASE_DEFINE(handle, name, GEOPMhint)`
	* definition of a region inside a loop
* `void INSTRUMENT_REGION_START(handle, name, SCOREPtype)`
	* region measurement start
* `double INSTRUMENT_REGION_STOP(handle, name)`
	* region measurement stop
	* returns runtime of the stopped region if the information is available, otherwise 0.0
* `double INSTRUMENT_REGION_STOP_START(handle_stop, name_stop, handle_start, name_start, SCOREPtype)`
	* region measurement stop and at the same time start of another region
	* returns runtime of the stopped region if the information is available, otherwise 0.0
* `void INSTRUMENT_PHASE_START(handle, name, SCOREPtype)`
	* start of a region inside a loop
* `double INSTRUMENT_PHASE_STOP(handle, name)`
	* end of a region inside a loop
	* returns runtime of the stopped region if the information is available, otherwise 0.0
* `void INSTRUMENT_IGNORE_START()`
	* turn off the measurement 
* `void INSTRUMENT_IGNORE_STOP()`
	* turn the measurement on after it was stopped

## Input parameters data types:
* int `nodeId` = compute node identifier, required by TAU only
* `handle` = variable name (it means string without quotation marks) for internal identification of a region in the application code
* `handle_start`, `handle_stop` = handle of a previously defined region that should start/stop
* const char * `name` = string identifying a region in the application code
* const char * `name_start`, `name_stop` = string identifying a region that should start/stop
* geopm_region_hint_e (uint64_t) `GEOPMhint` = initial control settings of GEOPM runtime. Supported values are:
	* `GEOPM_REGION_HINT_UNKNOWN` = Default value, provides no hint to the runtime.
	* `GEOPM_REGION_HINT_COMPUTE` = Compute limited region.
	* `GEOPM_REGION_HINT_MEMORY` = Memory bandwidth bound region.
	* `GEOPM_REGION_HINT_NETWORK` = Inter-node network dominated region, default for unnested MPI calls. User defined regions that have this hint will have the MPI time spent within this region attributed to the region as whole for the periods of time when all ranks are within an MPI function.
	* `GEOPM_REGION_HINT_IO` = Disk input/output dominated region.
	* `GEOPM_REGION_HINT_SERIAL` = Calculation that is not executed by a multi-threaded process (may be multi-process).
	* `GEOPM_REGION_HINT_PARALLEL` = Calculation that is executed by a multi-threaded process in a hybrid thread/process parallelism (e.g. MPI + OpenMP).
	* `GEOPM_REGION_HINT_IGNORE` = Region that control algorithms should ignore, and/or apply default policies. This hint should be applied to application start up, shutdown, and events that do not happen on every trip through the outer loop.
* SCOREP_User_RegionType (uint32_t) `SCOREPtype` = Specifies the type of the region. Parameter used by Score-P only. Possible values are:
	* `SCOREP_USER_REGION_TYPE_COMMON`
	* `SCOREP_USER_REGION_TYPE_FUNCTION`
	* `SCOREP_USER_REGION_TYPE_LOOP`
	* `SCOREP_USER_REGION_TYPE_DYNAMIC`
	* `SCOREP_USER_REGION_TYPE_PHASE`
	* combination of these

--------------------------------------------------------------------------------
#     MERIC API                                                                #
--------------------------------------------------------------------------------
If [MERIC](https://code.it4i.cz/vys0053/meric) library selected, the instrument.h includes `meric.h`.

| Shared API                                    | MERIC API                 |
| --------------------------------------------- | ------------------------- |
| INSTRUMENT_INIT(nodeID)                       | MERIC_Init();             |
| INSTRUMENT_CLOSE()                            | MERIC_Close();            |
| INSTRUMENT_PHASE_DEFINE(handle, name, hint)   |                           |
| INSTRUMENT_REGION_DEFINE(handle, name, hint)  |                           |
| INSTRUMENT_REGION_START(handle, name, type)   | MERIC_MeasureStart(name); |
| INSTRUMENT_REGION_STOP(handle, name)          | MERIC_MeasureStop(name);  |
| INSTRUMENT_REGION_STOP_START(handle_stop, name_stop, handle_start, name_start, type) | MERIC_MeasureStopStart (name_stop, name_start); |
| INSTRUMENT_PHASE_START(handle, name, type)    | MERIC_MeasureStart(name); |
| INSTRUMENT_PHASE_STOP(handle, name)           | MERIC_MeasureStop(name);  |
| INSTRUMENT_IGNORE_START()                     | MERIC_IgnoreStart();      |
| INSTRUMENT_IGNORE_STOP()                      | MERIC_IgnoreStop();       |


--------------------------------------------------------------------------------
#     TIMEPROF API                                                             #
--------------------------------------------------------------------------------
If [TIMEPROF](https://code.it4i.cz/vys0053/meric) library selected, the instrument.h includes `timeprof.h`.


| Shared API                                    | TIMEPROF API                |
| --------------------------------------------- | --------------------------- |
| INSTRUMENT_INIT(nodeID)                       |                             |
| INSTRUMENT_CLOSE()                            | TIMEPROF_evaluate();        |
| INSTRUMENT_PHASE_DEFINE(handle, name, hint)   |                             |
| INSTRUMENT_REGION_DEFINE(handle, name, hint)  |                             |
| INSTRUMENT_REGION_START(handle, name, type)   | TIMEPROF_regionStart(name); |
| INSTRUMENT_REGION_STOP(handle, name)          | TIMEPROF_regionStop(name);  |
| INSTRUMENT_REGION_STOP_START(handle_stop, name_stop, handle_start, name_start, SCOREPtype) | TIMEPROF_regionStop(name_stop); TIMEPROF_regionStart(name_start); |
| INSTRUMENT_PHASE_START(handle, name, type)    |                             |
| INSTRUMENT_PHASE_STOP(handle, name)           |                             |
| INSTRUMENT_IGNORE_START()                     |                             |
| INSTRUMENT_IGNORE_STOP()                      |                             |

--------------------------------------------------------------------------------
#     Score-P API                                                                #
--------------------------------------------------------------------------------
[Score-P](https://www.vi-hps.org/projects/score-p/) instrumentation also used for [Scalasca](http://www.scalasca.org/).
The instrument.h includes `scorep/SCOREP_User.h` and `scorep/SCOREP_User_Types.h`.


| Shared API                                   | Score-P API                                     |
| -------------------------------------------- | ----------------------------------------------- |
| INSTRUMENT_INIT(nodeID)                      |                                                 |
| INSTRUMENT_CLOSE()                           |                                                 |
| INSTRUMENT_PHASE_DEFINE(handle, name, hint)  | SCOREP_USER_REGION_DEFINE(handle);              |
| INSTRUMENT_REGION_DEFINE(handle, name, hint) | SCOREP_USER_REGION_DEFINE(handle);              |
| INSTRUMENT_REGION_START(handle, name, type)  | SCOREP_USER_REGION_BEGIN(handle, name, type);   |
| INSTRUMENT_REGION_STOP(handle, name)         | SCOREP_USER_REGION_END(handle);                 |
| INSTRUMENT_REGION_STOP_START(handle_stop, name_stop, handle_start, name_start, SCOREPtype) | SCOREP_USER_REGION_END(handle_stop); SCOREP_USER_REGION_BEGIN (handle_start, name_start, SCOREPtype); |
| INSTRUMENT_PHASE_START(handle, name, type)   | SCOREP_USER_OA_PHASE_BEGIN(handle, name, type); |
| INSTRUMENT_PHASE_STOP(handle, name)          | SCOREP_USER_OA_PHASE_END(handle);               |
| INSTRUMENT_IGNORE_START()                    | SCOREP_RECORDING_OFF();                         |
| INSTRUMENT_IGNORE_STOP()                     | SCOREP_RECORDING_ON();                          |

--------------------------------------------------------------------------------
#     TAU API                                                                  #
--------------------------------------------------------------------------------
If [TAU](https://www.cs.uoregon.edu/research/tau/home.php) library selected, the instrument.h includes `TAU.h`.


| Shared API                                   | TAU API                                               |
| -------------------------------------------- | ----------------------------------------------------- |
| INSTRUMENT_INIT(nodeID)                      | TAU_INIT(&argc, &argv); TAU_PROFILE_SET_NODE(nodeId); |
| INSTRUMENT_CLOSE()                           | TAU_PROFILE_EXIT("");                                 |
| INSTRUMENT_PHASE_DEFINE(handle, name, hint)  | TAU_DYNAMIC_PHASE(handle, name, "", TAU_USER1);       |
| INSTRUMENT_REGION_DEFINE(handle, name, hint) | TAU_PROFILE_TIMER(handle, name, "", TAU_USER);        |
| INSTRUMENT_REGION_START(handle, name, type)  | TAU_PROFILE_START(handle);                            |
| INSTRUMENT_REGION_STOP(handle, name)         | TAU_PROFILE_STOP(handle);                             |
| INSTRUMENT_REGION_STOP_START(handle_stop, name_stop, handle_start, name_start, SCOREPtype) | TAU_PROFILE_STOP(handle_stop); TAU_PROFILE_START(handler_start); |
| INSTRUMENT_PHASE_START(handle, name, type)   | TAU_PHASE_START(handle);                              |
| INSTRUMENT_PHASE_STOP(handle, name)          | TAU_PHASE_STOP(handle);                               |
| INSTRUMENT_IGNORE_START()                    | TAU_DISABLE_ALL_GROUPS();                             |
| INSTRUMENT_IGNORE_STOP()                     | TAU_ENABLE_ALL_GROUPS();                              |


--------------------------------------------------------------------------------
#     GEOPM API                                                                #
--------------------------------------------------------------------------------
If [GEOPM](https://geopm.github.io/) library selected, the instrument.h includes `geopm_prof.h` and `geopm_hint.h`.

| Shared API                                   |  GEOPM API                |
| -------------------------------------------- | ------------------------- |
| INSTRUMENT_INIT(nodeID)                      |                           |
| INSTRUMENT_CLOSE()                           | geopm_prof_shutdown();    |
| INSTRUMENT_PHASE_DEFINE(handle, name, hint)  |                           |
| INSTRUMENT_REGION_DEFINE(handle, name, hint) | uint64_t handle; geopm_prof_region(name, hint, & handle); |
| INSTRUMENT_REGION_START(handle, name, type)  | geopm_prof_enter(handle); |
| INSTRUMENT_REGION_STOP(handle, name)         | geopm_prof_exit(handle);  |
| INSTRUMENT_REGION_STOP_START(handle_stop, name_stop, handle_start, name_start, SCOREPtype) | geopm_prof_exit(handle_stop); geopm_prof_enter(handle_start); |
| INSTRUMENT_PHASE_START(handle, name, type)   | geopm_prof_epoch();       |
| INSTRUMENT_PHASE_STOP(handle, name)          |                           |
| INSTRUMENT_IGNORE_START()                    |                           |
| INSTRUMENT_IGNORE_STOP()                     |                           |


--------------------------------------------------------------------------------
#     mpiP API                                                                 #
--------------------------------------------------------------------------------
If [mpiP](http://mpip.sourceforge.net/) library selected, the instrument.h includes `mpiP-API.h`.

| Shared API                                   | mpiP API         |
| -------------------------------------------- | ---------------- |
| INSTRUMENT_INIT(nodeID)                      | mpiP_init_api(); |
| INSTRUMENT_CLOSE()                           |                  |
| INSTRUMENT_PHASE_DEFINE(handle, name, hint)  |                  |
| INSTRUMENT_REGION_DEFINE(handle, name, hint) |                  |
| INSTRUMENT_REGION_START(handle, name, type)  |                  |
| INSTRUMENT_REGION_STOP(handle, name)         |                  |
| INSTRUMENT_REGION_STOP_START(handle_stop, name_stop, handle_start, name_start, SCOREPtype) |  |
| INSTRUMENT_PHASE_START(handle, name, type)   |                  |
| INSTRUMENT_PHASE_STOP(handle, name)          |                  |
| INSTRUMENT_IGNORE_START()                    | MPI_Pcontrol(0); |
| INSTRUMENT_IGNORE_STOP()                     | MPI_Pcontrol(1); |

--------------------------------------------------------------------------------
#     Extrae API                                                               #
--------------------------------------------------------------------------------
If [Extrae](https://tools.bsc.es/extrae) library selected, the instrument.h includes `extrae.h`.

| Shared API                                   |  Extrae API                     |
| -------------------------------------------- | ------------------------------- |
| INSTRUMENT_INIT(nodeID)                      | Extrae_init();                  |
| INSTRUMENT_CLOSE()                           | Extrae_fini();                  |
| INSTRUMENT_PHASE_DEFINE(handle, name, hint)  | unsigned int handle = 0;        |
| INSTRUMENT_REGION_DEFINE(handle, name, hint) | Extrae_define_event_type(...);* |
| INSTRUMENT_REGION_START(handle, name, type)  | Extrae_event(...);**            |
| INSTRUMENT_REGION_STOP(handle, name)         | Extrae_event(...);***           |
| INSTRUMENT_REGION_STOP_START(handle_stop, name_stop, handle_start, name_start, SCOREPtype) | Extrae_event(...);*** Extrae_event(...)** |
| INSTRUMENT_PHASE_START(handle, name, type)   | Extrae_event(...);****          |
| INSTRUMENT_PHASE_STOP(handle, name)          | Extrae_event(...);***           |
| INSTRUMENT_IGNORE_START()                    | Extrae_shutdown();              |
| INSTRUMENT_IGNORE_STOP()                     | Extrae_restart();               |

```
	* Definition of the INSTRUMENT_REGION_DEFINE(handle, name, hint):
{
	unsigned INSTRUMENT_EXTRAE_type = INSTRUMENT_GET_HASH_ID(name);
	int INSTRUMENT_EXTRAE_nvalues = 2;
	extrae_value_t INSTRUMENT_EXTRAE_values[2] = {0, 1};
	char * INSTRUMENT_EXTRAE_description_values[2] = {"End", "Begin" };
	Extrae_define_event_type (&INSTRUMENT_EXTRAE_type, name, &INSTRUMENT_EXTRAE_nvalues, INSTRUMENT_EXTRAE_values, INSTRUMENT_EXTRAE_description_values);
}
```

```
	** Definition of the INSTRUMENT_REGION_START(handle, name, type):
{
	unsigned INSTRUMENT_EXTRAE_type = INSTRUMENT_GET_HASH_ID(name);
	Extrae_event( INSTRUMENT_EXTRAE_type, 1 );
}
```

```
	*** Definition of the INSTRUMENT_REGION_STOP(handle, name):
{
	unsigned INSTRUMENT_EXTRAE_type = INSTRUMENT_GET_HASH_ID(name);
	Extrae_event( INSTRUMENT_EXTRAE_type, 0 );
}
```

```
	**** Definition of the INSTRUMENT_PHASE_START(handle, name, type):
handle++;
{
	unsigned INSTRUMENT_EXTRAE_type = INSTRUMENT_GET_HASH_ID(name);
	Extrae_event( INSTRUMENT_EXTRAE_type, handle );
}
```

When using Extrae regions (events) must be defined first to assign a unique ID. In default the ID is generated by DJB2 hash function. If the hash function would generate the same identifier for two different regions the Extrae produces following error message `Warning! Ignoring duplicate definition`. In such a case, please recompile the application with `-DINSTRUMENT_HASH=SDBM` to swith to alternative hash method.


--------------------------------------------------------------------------------
#     LIKWID API                                                                #
--------------------------------------------------------------------------------
If [LIKWID](https://github.com/RRZE-HPC/likwid/wiki) library selected, the instrument.h includes `likwid.h`. For OpenMP applications `omp.h` must be included before `instrument.h`.

| Shared API                                   |  LIKWID API                        |
| -------------------------------------------- | ---------------------------------- |
| INSTRUMENT_INIT(nodeID)                      | likwid_markerInit();*              |
| INSTRUMENT_CLOSE()                           |                                    |
| INSTRUMENT_PHASE_DEFINE(handle, name, hint)  | likwid_markerRegisterRegion(name); |
| INSTRUMENT_REGION_DEFINE(handle, name, hint) | likwid_markerRegisterRegion(name); |
| INSTRUMENT_REGION_START(handle, name, type)  | likwid_markerStartRegion(name);    |
| INSTRUMENT_REGION_STOP(handle, name)         | likwid_markerStopRegion(name);     |
| INSTRUMENT_REGION_STOP_START(handle_stop, name_stop, handle_start, name_start, SCOREPtype) | likwid_markerStopRegion(name_stop); likwid_markerStartRegion(name_start); |
| INSTRUMENT_PHASE_START(handle, name, type)   | likwid_markerStartRegion(name);    |
| INSTRUMENT_PHASE_STOP(handle, name)          | likwid_markerStopRegion(name);     |
| INSTRUMENT_IGNORE_START()                    |                                    |
| INSTRUMENT_IGNORE_STOP()                     |                                    |
	*  In case of OpenMP applications also calls `likwid_markerThreadInit()`

--------------------------------------------------------------------------------
#     Acknowledgement                                                          #
--------------------------------------------------------------------------------
InstrumentAPI is being developed at [IT4Innovations National Supercomputing Center](https://www.it4i.cz/) under [BSD-3 license](https://code.it4i.cz/energy-efficiency/instrumentapi/-/blob/master/LICENSE).
Please, open an issue, if you meet any problem.