job-submission-and-execution.md 15.5 KB
Newer Older
Lukáš Krupčík's avatar
Lukáš Krupčík committed
1 2 3
# Job submission and execution

## Job Submission
Lukáš Krupčík's avatar
Lukáš Krupčík committed
4 5 6

When allocating computational resources for the job, please specify

David Hrbáč's avatar
David Hrbáč committed
7 8 9 10 11 12
1.  suitable queue for your job (default is qprod)
2.  number of computational nodes required
3.  number of cores per node required
4.  maximum wall time allocated to your calculation, note that jobs exceeding maximum wall time will be killed
5.  Project ID
6.  Jobscript or interactive switch
Lukáš Krupčík's avatar
Lukáš Krupčík committed
13

14
!!! Note "Note"
Lukáš Krupčík's avatar
Lukáš Krupčík committed
15
    Use the **qsub** command to submit your job to a queue for allocation of the computational resources.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
16 17 18

Submit the job using the qsub command:

Lukáš Krupčík's avatar
Lukáš Krupčík committed
19
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
20
$ qsub -A Project_ID -q queue -l select=x:ncpus=y,walltime=[[hh:]mm:]ss[.ms] jobscript
Lukáš Krupčík's avatar
Lukáš Krupčík committed
21
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
22

Lukáš Krupčík's avatar
Lukáš Krupčík committed
23
The qsub submits the job into the queue, in another words the qsub command creates a request to the PBS Job manager for allocation of specified resources. The resources will be allocated when available, subject to above described policies and constraints. **After the resources are allocated the jobscript or interactive shell is executed on first of the allocated nodes.**
Lukáš Krupčík's avatar
Lukáš Krupčík committed
24 25 26

### Job Submission Examples

Lukáš Krupčík's avatar
Lukáš Krupčík committed
27
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
28
$ qsub -A OPEN-0-0 -q qprod -l select=64:ncpus=16,walltime=03:00:00 ./myjob
Lukáš Krupčík's avatar
Lukáš Krupčík committed
29
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
30

Lukáš Krupčík's avatar
Lukáš Krupčík committed
31
In this example, we allocate 64 nodes, 16 cores per node, for 3 hours. We allocate these resources via the qprod queue, consumed resources will be accounted to the Project identified by Project ID OPEN-0-0. Jobscript myjob will be executed on the first node in the allocation.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
32

Lukáš Krupčík's avatar
Lukáš Krupčík committed
33
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
34
$ qsub -q qexp -l select=4:ncpus=16 -I
Lukáš Krupčík's avatar
Lukáš Krupčík committed
35
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
36

Lukáš Krupčík's avatar
Lukáš Krupčík committed
37
In this example, we allocate 4 nodes, 16 cores per node, for 1 hour. We allocate these resources via the qexp queue. The resources will be available interactively
Lukáš Krupčík's avatar
Lukáš Krupčík committed
38

Lukáš Krupčík's avatar
Lukáš Krupčík committed
39
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
40
$ qsub -A OPEN-0-0 -q qnvidia -l select=10:ncpus=16 ./myjob
Lukáš Krupčík's avatar
Lukáš Krupčík committed
41
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
42

David Hrbáč's avatar
David Hrbáč committed
43
In this example, we allocate 10 nvidia accelerated nodes, 16 cores per node, for  24 hours. We allocate these resources via the qnvidia queue. Jobscript myjob will be executed on the first node in the allocation.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
44

Lukáš Krupčík's avatar
Lukáš Krupčík committed
45
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
46
$ qsub -A OPEN-0-0 -q qfree -l select=10:ncpus=16 ./myjob
Lukáš Krupčík's avatar
Lukáš Krupčík committed
47
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
48

David Hrbáč's avatar
David Hrbáč committed
49
In this example, we allocate 10  nodes, 16 cores per node, for 12 hours. We allocate these resources via the qfree queue. It is not required that the project OPEN-0-0 has any available resources left. Consumed resources are still accounted for. Jobscript myjob will be executed on the first node in the allocation.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
50

51
All qsub options may be [saved directly into the jobscript](job-submission-and-execution/#PBSsaved). In such a case, no options to qsub are needed.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
52

Lukáš Krupčík's avatar
Lukáš Krupčík committed
53
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
54
$ qsub ./myjob
Lukáš Krupčík's avatar
Lukáš Krupčík committed
55
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
56

Lukáš Krupčík's avatar
Lukáš Krupčík committed
57
By default, the PBS batch system sends an e-mail only when the job is aborted. Disabling mail events completely can be done like this:
Lukáš Krupčík's avatar
Lukáš Krupčík committed
58

Lukáš Krupčík's avatar
Lukáš Krupčík committed
59
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
60
$ qsub -m n
Lukáš Krupčík's avatar
Lukáš Krupčík committed
61
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
62

Lukáš Krupčík's avatar
Lukáš Krupčík committed
63
## Advanced job placement
Lukáš Krupčík's avatar
Lukáš Krupčík committed
64 65 66 67 68

### Placement by name

Specific nodes may be allocated via the PBS

Lukáš Krupčík's avatar
Lukáš Krupčík committed
69
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
70
qsub -A OPEN-0-0 -q qprod -l select=1:ncpus=16:host=cn171+1:ncpus=16:host=cn172 -I
Lukáš Krupčík's avatar
Lukáš Krupčík committed
71
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
72

David Hrbáč's avatar
David Hrbáč committed
73
In this example, we allocate nodes cn171 and cn172, all 16 cores per node, for 24 hours.  Consumed resources will be accounted to the Project identified by Project ID OPEN-0-0. The resources will be available interactively.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
74 75 76

### Placement by CPU type

David Hrbáč's avatar
David Hrbáč committed
77
Nodes equipped with Intel Xeon E5-2665 CPU have base clock frequency 2.4GHz, nodes equipped with Intel Xeon E5-2470 CPU have base frequency 2.3 GHz (see section Compute Nodes for details).  Nodes may be selected via the PBS resource attribute cpu_freq .
Lukáš Krupčík's avatar
Lukáš Krupčík committed
78

David Hrbáč's avatar
David Hrbáč committed
79 80 81 82
| CPU Type           | base freq. | Nodes                  | cpu_freq attribute |
| ------------------ | ---------- | ---------------------- | ------------------ |
| Intel Xeon E5-2665 | 2.4GHz     | cn[1-180], cn[208-209] | 24                 |
| Intel Xeon E5-2470 | 2.3GHz     | cn[181-207]            | 23                 |
Lukáš Krupčík's avatar
Lukáš Krupčík committed
83

Lukáš Krupčík's avatar
Lukáš Krupčík committed
84
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
85
$ qsub -A OPEN-0-0 -q qprod -l select=4:ncpus=16:cpu_freq=24 -I
Lukáš Krupčík's avatar
Lukáš Krupčík committed
86
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
87

Lukáš Krupčík's avatar
Lukáš Krupčík committed
88
In this example, we allocate 4 nodes, 16 cores, selecting only the nodes with Intel Xeon E5-2665 CPU.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
89 90 91

### Placement by IB switch

David Hrbáč's avatar
David Hrbáč committed
92
Groups of computational nodes are connected to chassis integrated Infiniband switches. These switches form the leaf switch layer of the [Infiniband network](../network/) fat tree topology. Nodes sharing the leaf switch can communicate most efficiently. Sharing the same switch prevents hops in the network and provides for unbiased, most efficient network communication.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
93

94
Nodes sharing the same switch may be selected via the PBS resource attribute ibswitch. Values of this attribute are iswXX, where XX is the switch number. The node-switch mapping can be seen at [Hardware Overview](../hardware-overview/) section.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
95

Lukáš Krupčík's avatar
Lukáš Krupčík committed
96
We recommend allocating compute nodes of a single switch when best possible computational network performance is required to run the job efficiently:
Lukáš Krupčík's avatar
Lukáš Krupčík committed
97

Lukáš Krupčík's avatar
Lukáš Krupčík committed
98
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
99
    qsub -A OPEN-0-0 -q qprod -l select=18:ncpus=16:ibswitch=isw11 ./myjob
Lukáš Krupčík's avatar
Lukáš Krupčík committed
100
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
101

Lukáš Krupčík's avatar
Lukáš Krupčík committed
102
In this example, we request all the 18 nodes sharing the isw11 switch for 24 hours. Full chassis will be allocated.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
103

Lukáš Krupčík's avatar
Lukáš Krupčík committed
104
## Advanced job handling
Lukáš Krupčík's avatar
Lukáš Krupčík committed
105 106 107

### Selecting Turbo Boost off

Lukáš Krupčík's avatar
Lukáš Krupčík committed
108
Intel Turbo Boost Technology is on by default. We strongly recommend keeping the default.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
109

Lukáš Krupčík's avatar
Lukáš Krupčík committed
110
If necessary (such as in case of benchmarking) you can disable the Turbo for all nodes of the job by using the PBS resource attribute cpu_turbo_boost
Lukáš Krupčík's avatar
Lukáš Krupčík committed
111

Lukáš Krupčík's avatar
Lukáš Krupčík committed
112
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
113
    $ qsub -A OPEN-0-0 -q qprod -l select=4:ncpus=16 -l cpu_turbo_boost=0 -I
Lukáš Krupčík's avatar
Lukáš Krupčík committed
114
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
115 116 117 118 119

More about the Intel Turbo Boost in the TurboBoost section

### Advanced examples

Lukáš Krupčík's avatar
Lukáš Krupčík committed
120
In the following example, we select an allocation for benchmarking a very special and demanding MPI program. We request Turbo off, 2 full chassis of compute nodes (nodes sharing the same IB switches) for 30 minutes:
Lukáš Krupčík's avatar
Lukáš Krupčík committed
121

Lukáš Krupčík's avatar
Lukáš Krupčík committed
122 123 124 125
```bash
    $ qsub -A OPEN-0-0 -q qprod
    -l select=18:ncpus=16:ibswitch=isw10:mpiprocs=1:ompthreads=16+18:ncpus=16:ibswitch=isw20:mpiprocs=16:ompthreads=1
    -l cpu_turbo_boost=0,walltime=00:30:00
Lukáš Krupčík's avatar
Lukáš Krupčík committed
126
    -N Benchmark ./mybenchmark
Lukáš Krupčík's avatar
Lukáš Krupčík committed
127
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
128

David Hrbáč's avatar
David Hrbáč committed
129
The MPI processes will be distributed differently on the nodes connected to the two switches. On the isw10 nodes, we will run 1 MPI process per node 16 threads per process, on isw20  nodes we will run 16 plain MPI processes.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
130

Lukáš Krupčík's avatar
Lukáš Krupčík committed
131
Although this example is somewhat artificial, it demonstrates the flexibility of the qsub command options.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
132

Lukáš Krupčík's avatar
Lukáš Krupčík committed
133 134
## Job Management

135
!!! Note "Note"
Lukáš Krupčík's avatar
Lukáš Krupčík committed
136
    Check status of your jobs using the **qstat** and **check-pbs-jobs** commands
Lukáš Krupčík's avatar
Lukáš Krupčík committed
137

Lukáš Krupčík's avatar
Lukáš Krupčík committed
138
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
139 140 141 142
$ qstat -a
$ qstat -a -u username
$ qstat -an -u username
$ qstat -f 12345.srv11
Lukáš Krupčík's avatar
Lukáš Krupčík committed
143
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
144 145 146

Example:

Lukáš Krupčík's avatar
Lukáš Krupčík committed
147
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
148 149 150 151 152 153 154 155 156
$ qstat -a

srv11:
                                                            Req'd  Req'd   Elap
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time
--------------- -------- --  |---|---| ------ --- --- ------ ----- - -----
16287.srv11     user1    qlong    job1         6183   4  64    --  144:0 R 38:25
16468.srv11     user1    qlong    job2         8060   4  64    --  144:0 R 17:44
16547.srv11     user2    qprod    job3x       13516   2  32    --  48:00 R 00:58
Lukáš Krupčík's avatar
Lukáš Krupčík committed
157 158
```

David Hrbáč's avatar
David Hrbáč committed
159
In this example user1 and user2 are running jobs named job1, job2 and job3x. The jobs job1 and job2 are using 4 nodes, 16 cores per node each. The job1 already runs for 38 hours and 25 minutes, job2 for 17 hours 44 minutes. The job1 already consumed 64_38.41 = 2458.6 core hours. The job3x already consumed 0.96_32 = 30.93 core hours. These consumed core hours will be accounted on the respective project accounts, regardless of whether the allocated cores were actually used for computations.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
160 161 162 163

Check status of your jobs using check-pbs-jobs command. Check presence of user's PBS jobs' processes on execution hosts. Display load, processes. Display job standard and error output. Continuously display (tail -f) job standard or error output.

```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
164 165 166 167 168 169 170
$ check-pbs-jobs --check-all
$ check-pbs-jobs --print-load --print-processes
$ check-pbs-jobs --print-job-out --print-job-err

$ check-pbs-jobs --jobid JOBID --check-all --print-all

$ check-pbs-jobs --jobid JOBID --tailf-job-out
Lukáš Krupčík's avatar
Lukáš Krupčík committed
171
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
172 173 174

Examples:

Lukáš Krupčík's avatar
Lukáš Krupčík committed
175
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
176 177 178 179 180 181
$ check-pbs-jobs --check-all
JOB 35141.dm2, session_id 71995, user user2, nodes cn164,cn165
Check session id: OK
Check processes
cn164: OK
cn165: No process
Lukáš Krupčík's avatar
Lukáš Krupčík committed
182
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
183

Lukáš Krupčík's avatar
Lukáš Krupčík committed
184
In this example we see that job 35141.dm2 currently runs no process on allocated node cn165, which may indicate an execution error.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
185

Lukáš Krupčík's avatar
Lukáš Krupčík committed
186
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
187 188 189 190 191 192 193 194 195 196 197
$ check-pbs-jobs --print-load --print-processes
JOB 35141.dm2, session_id 71995, user user2, nodes cn164,cn165
Print load
cn164: LOAD: 16.01, 16.01, 16.00
cn165: LOAD:  0.01,  0.00,  0.01
Print processes
       %CPU CMD
cn164:  0.0 -bash
cn164:  0.0 /bin/bash /var/spool/PBS/mom_priv/jobs/35141.dm2.SC
cn164: 99.7 run-task
...
Lukáš Krupčík's avatar
Lukáš Krupčík committed
198
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
199

Lukáš Krupčík's avatar
Lukáš Krupčík committed
200
In this example we see that job 35141.dm2 currently runs process run-task on node cn164, using one thread only, while node cn165 is empty, which may indicate an execution error.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
201

Lukáš Krupčík's avatar
Lukáš Krupčík committed
202
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
203 204 205 206
$ check-pbs-jobs --jobid 35141.dm2 --print-job-out
JOB 35141.dm2, session_id 71995, user user2, nodes cn164,cn165
Print job standard output:
======================== Job start  ==========================
David Hrbáč's avatar
David Hrbáč committed
207 208
Started at    : Fri Aug 30 02:47:53 CEST 2013
Script name   : script
Lukáš Krupčík's avatar
Lukáš Krupčík committed
209 210 211
Run loop 1
Run loop 2
Run loop 3
Lukáš Krupčík's avatar
Lukáš Krupčík committed
212
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
213

Lukáš Krupčík's avatar
Lukáš Krupčík committed
214
In this example, we see actual output (some iteration loops) of the job 35141.dm2
Lukáš Krupčík's avatar
Lukáš Krupčík committed
215

216
!!! Note "Note"
Lukáš Krupčík's avatar
Lukáš Krupčík committed
217
    Manage your queued or running jobs, using the **qhold**, **qrls**, **qdel**, **qsig** or **qalter** commands
Lukáš Krupčík's avatar
Lukáš Krupčík committed
218 219 220

You may release your allocation at any time, using qdel command

Lukáš Krupčík's avatar
Lukáš Krupčík committed
221
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
222
$ qdel 12345.srv11
Lukáš Krupčík's avatar
Lukáš Krupčík committed
223
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
224 225 226

You may kill a running job by force, using qsig command

Lukáš Krupčík's avatar
Lukáš Krupčík committed
227
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
228
$ qsig -s 9 12345.srv11
Lukáš Krupčík's avatar
Lukáš Krupčík committed
229
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
230 231 232

Learn more by reading the pbs man page

Lukáš Krupčík's avatar
Lukáš Krupčík committed
233
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
234
$ man pbs_professional
Lukáš Krupčík's avatar
Lukáš Krupčík committed
235
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
236

Lukáš Krupčík's avatar
Lukáš Krupčík committed
237
## Job Execution
Lukáš Krupčík's avatar
Lukáš Krupčík committed
238 239 240

### Jobscript

241
!!! Note "Note"
Lukáš Krupčík's avatar
Lukáš Krupčík committed
242
    Prepare the jobscript to run batch jobs in the PBS queue system
Lukáš Krupčík's avatar
Lukáš Krupčík committed
243

Lukáš Krupčík's avatar
Lukáš Krupčík committed
244
The Jobscript is a user made script, controlling sequence of commands for executing the calculation. It is often written in bash, other scripts may be used as well. The jobscript is supplied to PBS **qsub** command as an argument and executed by the PBS Professional workload manager.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
245

246
!!! Note "Note"
Lukáš Krupčík's avatar
Lukáš Krupčík committed
247
    The jobscript or interactive shell is executed on first of the allocated nodes.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
248

Lukáš Krupčík's avatar
Lukáš Krupčík committed
249
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
250 251 252 253 254 255 256 257 258
$ qsub -q qexp -l select=4:ncpus=16 -N Name0 ./myjob
$ qstat -n -u username

srv11:
                                                            Req'd  Req'd   Elap
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time
--------------- -------- --  |---|---| ------ --- --- ------ ----- - -----
15209.srv11     username qexp     Name0        5530   4  64    --  01:00 R 00:00
   cn17/0*16+cn108/0*16+cn109/0*16+cn110/0*16
Lukáš Krupčík's avatar
Lukáš Krupčík committed
259
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
260

Lukáš Krupčík's avatar
Lukáš Krupčík committed
261
In this example, the nodes cn17, cn108, cn109 and cn110 were allocated for 1 hour via the qexp queue. The jobscript myjob will be executed on the node cn17, while the nodes cn108, cn109 and cn110 are available for use as well.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
262

Lukáš Krupčík's avatar
Lukáš Krupčík committed
263
The jobscript or interactive shell is by default executed in home directory
Lukáš Krupčík's avatar
Lukáš Krupčík committed
264

Lukáš Krupčík's avatar
Lukáš Krupčík committed
265
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
266 267 268 269 270 271
$ qsub -q qexp -l select=4:ncpus=16 -I
qsub: waiting for job 15210.srv11 to start
qsub: job 15210.srv11 ready

$ pwd
/home/username
Lukáš Krupčík's avatar
Lukáš Krupčík committed
272
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
273

Lukáš Krupčík's avatar
Lukáš Krupčík committed
274
In this example, 4 nodes were allocated interactively for 1 hour via the qexp queue. The interactive shell is executed in the home directory.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
275

276
!!! Note "Note"
Lukáš Krupčík's avatar
Lukáš Krupčík committed
277
    All nodes within the allocation may be accessed via ssh.  Unallocated nodes are not accessible to user.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
278

Lukáš Krupčík's avatar
Lukáš Krupčík committed
279
The allocated nodes are accessible via ssh from login nodes. The nodes may access each other via ssh as well.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
280

Lukáš Krupčík's avatar
Lukáš Krupčík committed
281
Calculations on allocated nodes may be executed remotely via the MPI, ssh, pdsh or clush. You may find out which nodes belong to the allocation by reading the $PBS_NODEFILE file
Lukáš Krupčík's avatar
Lukáš Krupčík committed
282

Lukáš Krupčík's avatar
Lukáš Krupčík committed
283
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
284 285 286 287 288 289 290 291 292 293 294 295
qsub -q qexp -l select=4:ncpus=16 -I
qsub: waiting for job 15210.srv11 to start
qsub: job 15210.srv11 ready

$ pwd
/home/username

$ sort -u $PBS_NODEFILE
cn17.bullx
cn108.bullx
cn109.bullx
cn110.bullx
Lukáš Krupčík's avatar
Lukáš Krupčík committed
296

Lukáš Krupčík's avatar
Lukáš Krupčík committed
297 298 299 300 301
$ pdsh -w cn17,cn[108-110] hostname
cn17: cn17
cn108: cn108
cn109: cn109
cn110: cn110
Lukáš Krupčík's avatar
Lukáš Krupčík committed
302
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
303

Lukáš Krupčík's avatar
Lukáš Krupčík committed
304
In this example, the hostname program is executed via pdsh from the interactive shell. The execution runs on all four allocated nodes. The same result would be achieved if the pdsh is called from any of the allocated nodes or from the login nodes.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
305 306 307

### Example Jobscript for MPI Calculation

308
!!! Note "Note"
Lukáš Krupčík's avatar
Lukáš Krupčík committed
309
    Production jobs must use the /scratch directory for I/O
Lukáš Krupčík's avatar
Lukáš Krupčík committed
310

Lukáš Krupčík's avatar
Lukáš Krupčík committed
311
The recommended way to run production jobs is to change to /scratch directory early in the jobscript, copy all inputs to /scratch, execute the calculations and copy outputs to home directory.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
312

Lukáš Krupčík's avatar
Lukáš Krupčík committed
313
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
314 315 316 317 318 319 320
#!/bin/bash

# change to scratch directory, exit on failure
SCRDIR=/scratch/$USER/myjob
mkdir -p $SCRDIR
cd $SCRDIR || exit

Lukáš Krupčík's avatar
Lukáš Krupčík committed
321
# copy input file to scratch
Lukáš Krupčík's avatar
Lukáš Krupčík committed
322 323 324 325 326 327 328 329 330 331 332 333 334 335
cp $PBS_O_WORKDIR/input .
cp $PBS_O_WORKDIR/mympiprog.x .

# load the mpi module
module load openmpi

# execute the calculation
mpiexec -pernode ./mympiprog.x

# copy output file to home
cp output $PBS_O_WORKDIR/.

#exit
exit
Lukáš Krupčík's avatar
Lukáš Krupčík committed
336 337 338 339
```

In this example, some directory on the /home holds the input file input and executable mympiprog.x . We create a directory myjob on the /scratch filesystem, copy input and executable files from the /home directory where the qsub was invoked ($PBS_O_WORKDIR) to /scratch, execute the MPI programm mympiprog.x and copy the output file back to the /home directory. The mympiprog.x is executed as one process per node, on all allocated nodes.

340
!!! Note "Note"
Lukáš Krupčík's avatar
Lukáš Krupčík committed
341
    Consider preloading inputs and executables onto [shared scratch](storage/) before the calculation starts.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
342 343 344

In some cases, it may be impractical to copy the inputs to scratch and outputs to home. This is especially true when very large input and output files are expected, or when the files should be reused by a subsequent calculation. In such a case, it is users responsibility to preload the input files on shared /scratch before the job submission and retrieve the outputs manually, after all calculations are finished.

345
!!! Note "Note"
Lukáš Krupčík's avatar
Lukáš Krupčík committed
346
    Store the qsub options within the jobscript. Use **mpiprocs** and **ompthreads** qsub options to control the MPI job execution.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
347 348 349 350

Example jobscript for an MPI job with preloaded inputs and executables, options for qsub are stored within the script :

```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368
#!/bin/bash
#PBS -q qprod
#PBS -N MYJOB
#PBS -l select=100:ncpus=16:mpiprocs=1:ompthreads=16
#PBS -A OPEN-0-0

# change to scratch directory, exit on failure
SCRDIR=/scratch/$USER/myjob
cd $SCRDIR || exit

# load the mpi module
module load openmpi

# execute the calculation
mpiexec ./mympiprog.x

#exit
exit
Lukáš Krupčík's avatar
Lukáš Krupčík committed
369 370 371 372
```

In this example, input and executable files are assumed preloaded manually in /scratch/$USER/myjob directory. Note the **mpiprocs** and **ompthreads** qsub options, controlling behavior of the MPI execution. The mympiprog.x is executed as one process per node, on all 100 allocated nodes. If mympiprog.x implements OpenMP threads, it will run 16 threads per node.

373
More information is found in the [Running OpenMPI](../software/mpi/Running_OpenMPI/) and [Running MPICH2](../software/mpi/running-mpich2/)
Lukáš Krupčík's avatar
Lukáš Krupčík committed
374 375 376 377
sections.

### Example Jobscript for Single Node Calculation

378
!!! Note "Note"
Lukáš Krupčík's avatar
Lukáš Krupčík committed
379
    Local scratch directory is often useful for single node jobs. Local scratch will be deleted immediately after the job ends.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
380

Pavel Jirásek's avatar
Pavel Jirásek committed
381
Example jobscript for single node calculation, using [local scratch](storage/) on the node:
Lukáš Krupčík's avatar
Lukáš Krupčík committed
382

Lukáš Krupčík's avatar
Lukáš Krupčík committed
383
```bash
Lukáš Krupčík's avatar
Lukáš Krupčík committed
384 385 386 387 388
#!/bin/bash

# change to local scratch directory
cd /lscratch/$PBS_JOBID || exit

Lukáš Krupčík's avatar
Lukáš Krupčík committed
389
# copy input file to scratch
Lukáš Krupčík's avatar
Lukáš Krupčík committed
390 391 392 393 394 395 396 397 398 399 400
cp $PBS_O_WORKDIR/input .
cp $PBS_O_WORKDIR/myprog.x .

# execute the calculation
./myprog.x

# copy output file to home
cp output $PBS_O_WORKDIR/.

#exit
exit
Lukáš Krupčík's avatar
Lukáš Krupčík committed
401
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
402

Lukáš Krupčík's avatar
Lukáš Krupčík committed
403
In this example, some directory on the home holds the input file input and executable myprog.x . We copy input and executable files from the home directory where the qsub was invoked ($PBS_O_WORKDIR) to local scratch /lscratch/$PBS_JOBID, execute the myprog.x and copy the output file back to the /home directory. The myprog.x runs on one node only and may use threads.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
404 405 406

### Other Jobscript Examples

407
Further jobscript examples may be found in the software section and the [Capacity computing](capacity-computing/) section.