Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
Total View
==========
TotalView is a GUI-based source code multi-process, multi-thread
debugger.
License and Limitations for cluster Users
-----------------------------------------
On the cluster users can debug OpenMP or MPI code that runs up to 64
parallel processes. These limitation means that:
1 user can debug up 64 processes, or
32 users can debug 2 processes, etc.
Debugging of GPU accelerated codes is also supported.
You can check the status of the licenses
[here](https://extranet.it4i.cz/rsweb/anselm/license/totalview).
Compiling Code to run with TotalView
------------------------------------
### Modules
Load all necessary modules to compile the code. For example:
module load intel
module load impi ... or ... module load OpenMPI/X.X.X-icc
Load the TotalView module:
module load TotalView/8.15.4-6-linux-x86-64
Compile the code:
mpicc -g -O0 -o test_debug test.c
mpif90 -g -O0 -o test_debug test.f
### Compiler flags
Before debugging, you need to compile your code with theses flags:
-g** : Generates extra debugging information usable by GDB. -g3**
includes even more debugging information. This option is available for
GNU and INTEL C/C++ and Fortran compilers.
-O0** : Suppress all optimizations.**
Starting a Job with TotalView
-----------------------------
Be sure to log in with an X window forwarding enabled. This could mean
using the -X in the ssh:
ssh -X username@salomon.it4i.cz
Other options is to access login node using VNC. Please see the detailed
information on how to use graphic user interface on Anselm
[here](https://docs.it4i.cz/salomon/software/debuggers/resolveuid/11e53ad0d2fd4c5187537f4baeedff33#VNC).
From the login node an interactive session with X windows forwarding (-X
option) can be started by following command:
qsub -I -X -A NONE-0-0 -q qexp -lselect=1:ncpus=24:mpiprocs=24,walltime=01:00:00
Then launch the debugger with the totalview command followed by the name
of the executable to debug.
### Debugging a serial code
To debug a serial code use:
totalview test_debug
### Debugging a parallel code - option 1
To debug a parallel code compiled with >**OpenMPI** you need
to setup your TotalView environment:
Please note:** To be able to run parallel debugging procedure from the
command line without stopping the debugger in the mpiexec source code
you have to add the following function to your **~/.tvdrc** file:
proc mpi_auto_run_starter {loaded_id} {
set starter_programs {mpirun mpiexec orterun}
set executable_name [TV::symbol get $loaded_id full_pathname]
set file_component [file tail $executable_name]
if {[lsearch -exact $starter_programs $file_component] != -1} {
puts "*************************************"
puts "Automatically starting $file_component"
puts "*************************************"
dgo
}
}
# Append this function to TotalView's image load callbacks so that
# TotalView run this program automatically.
dlappend TV::image_load_callbacks mpi_auto_run_starter
The source code of this function can be also found in
/apps/all/OpenMPI/1.10.1-GNU-4.9.3-2.25/etc/openmpi-totalview.tcl
You can also add only following line to you ~/.tvdrc file instead of
the entire function:
source /apps/all/OpenMPI/1.10.1-GNU-4.9.3-2.25/etc/openmpi-totalview.tcl**
You need to do this step only once. See also [OpenMPI FAQ
entry](https://www.open-mpi.org/faq/?category=running#run-with-tv)
Now you can run the parallel debugger using:
mpirun -tv -n 5 ./test_debug
When following dialog appears click on "Yes"
At this point the main TotalView GUI window will appear and you can
insert the breakpoints and start debugging:
### Debugging a parallel code - option 2
Other option to start new parallel debugging session from a command line
is to let TotalView to execute mpirun by itself. In this case user has
to specify a MPI implementation used to compile the source code.
The following example shows how to start debugging session with Intel
MPI:
module load intel/2015b-intel-2015b impi/5.0.3.048-iccifort-2015.3.187-GNU-5.1.0-2.25 TotalView/8.15.4-6-linux-x86-64
totalview -mpi "Intel MPI-Hydra" -np 8 ./hello_debug_impi
After running previous command you will see the same window as shown in
the screenshot above.
More information regarding the command line parameters of the TotalView
can be found TotalView Reference Guide, Chapter 7: TotalView Command
Syntax.
Documentation
-------------
[1] The [TotalView
documentation](http://www.roguewave.com/support/product-documentation/totalview-family.aspx#totalview)
web page is a good resource for learning more about some of the advanced
TotalView features.