virtualization.md 15.9 KB
Newer Older
David Hrbáč's avatar
David Hrbáč committed
1
# Virtualization
Lukáš Krupčík's avatar
Lukáš Krupčík committed
2

David Hrbáč's avatar
Typo    
David Hrbáč committed
3
Running virtual machines on compute nodes
David Hrbáč's avatar
David Hrbáč committed
4
5

## Introduction
Lukáš Krupčík's avatar
Lukáš Krupčík committed
6

Lukáš Krupčík's avatar
Lukáš Krupčík committed
7
There are situations when Anselm's environment is not suitable for user needs.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
8

Lukáš Krupčík's avatar
* -> *    
Lukáš Krupčík committed
9
10
11
12
13
* Application requires different operating system (e.g Windows), application is not available for Linux
* Application requires different versions of base system libraries and tools
* Application requires specific setup (installation, configuration) of complex software stack
* Application requires privileged access to operating system
* ... and combinations of above cases
Lukáš Krupčík's avatar
Lukáš Krupčík committed
14

Josef Hrabal's avatar
Josef Hrabal committed
15
We offer solution for these cases - **virtualization**. Anselm's environment gives the possibility to run virtual machines on compute nodes. Users can create their own images of operating system with specific software stack and run instances of these images as virtual machines on compute nodes. Run of virtual machines is provided by standard mechanism of [Resource Allocation and Job Execution](/salomon/job-submission-and-execution/).
Lukáš Krupčík's avatar
Lukáš Krupčík committed
16

Lukáš Krupčík's avatar
Lukáš Krupčík committed
17
Solution is based on QEMU-KVM software stack and provides hardware-assisted x86 virtualization.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
18

David Hrbáč's avatar
David Hrbáč committed
19
## Limitations
Lukáš Krupčík's avatar
Lukáš Krupčík committed
20

Lukáš Krupčík's avatar
Lukáš Krupčík committed
21
Anselm's infrastructure was not designed for virtualization. Anselm's environment is not intended primary for virtualization, compute nodes, storages and all infrastructure of Anselm is intended and optimized for running HPC jobs, this implies suboptimal configuration of virtualization and limitations.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
22

Lukáš Krupčík's avatar
Lukáš Krupčík committed
23
Anselm's virtualization does not provide performance and all features of native environment. There is significant performance hit (degradation) in I/O performance (storage, network). Anselm's virtualization is not suitable for I/O (disk, network) intensive workloads.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
24

Lukáš Krupčík's avatar
Lukáš Krupčík committed
25
Virtualization has also some drawbacks, it is not so easy to setup efficient solution.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
26

27
Solution described in chapter [HOWTO](virtualization/#howto)  is suitable for single node tasks, does not introduce virtual machine clustering.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
28

David Hrbáč's avatar
David Hrbáč committed
29
!!! note
Lukáš Krupčík's avatar
Lukáš Krupčík committed
30
    Please consider virtualization as last resort solution for your needs.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
31

David Hrbáč's avatar
David Hrbáč committed
32
!!! warning
David Hrbáč's avatar
David Hrbáč committed
33
    Please consult use of virtualization with IT4Innovation's support.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
34

David Hrbáč's avatar
Typo    
David Hrbáč committed
35
For running Windows application (when source code and Linux native application are not available) consider use of Wine, Windows compatibility layer. Many Windows applications can be run using Wine with less effort and better performance than when using virtualization.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
36

David Hrbáč's avatar
David Hrbáč committed
37
## Licensing
Lukáš Krupčík's avatar
Lukáš Krupčík committed
38

Lukáš Krupčík's avatar
Lukáš Krupčík committed
39
IT4Innovations does not provide any licenses for operating systems and software of virtual machines. Users are ( in accordance with [Acceptable use policy document](http://www.it4i.cz/acceptable-use-policy.pdf)) fully responsible for licensing all software running in virtual machines on Anselm. Be aware of complex conditions of licensing software in virtual environments.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
40

David Hrbáč's avatar
David Hrbáč committed
41
!!! note
Lukáš Krupčík's avatar
Lukáš Krupčík committed
42
    Users are responsible for licensing OS e.g. MS Windows and all software running in their virtual machines.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
43

David Hrbáč's avatar
David Hrbáč committed
44
## Howto
Lukáš Krupčík's avatar
Lukáš Krupčík committed
45
46
47
48
49

### Virtual Machine Job Workflow

We propose this job workflow:

50
![Workflow](../../img/virtualization-job-workflow.png)
Lukáš Krupčík's avatar
Lukáš Krupčík committed
51

Lukáš Krupčík's avatar
Lukáš Krupčík committed
52
Our recommended solution is that job script creates distinct shared job directory, which makes a central point for data exchange between Anselm's environment, compute node (host) (e.g. HOME, SCRATCH, local scratch and other local or cluster file systems) and virtual machine (guest). Job script links or copies input data and instructions what to do (run script) for virtual machine to job directory and virtual machine process input data according instructions in job directory and store output back to job directory. We recommend, that virtual machine is running in so called [snapshot mode](virtualization/#snapshot-mode), image is immutable - image does not change, so one image can be used for many concurrent jobs.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
53
54
55

### Procedure

Lukáš Krupčík's avatar
Lukáš Krupčík committed
56
57
58
59
60
1. Prepare image of your virtual machine
1. Optimize image of your virtual machine for Anselm's virtualization
1. Modify your image for running jobs
1. Create job script for executing virtual machine
1. Run jobs
Lukáš Krupčík's avatar
Lukáš Krupčík committed
61

David Hrbáč's avatar
David Hrbáč committed
62
### Prepare Image of Your Virtual Machine
Lukáš Krupčík's avatar
Lukáš Krupčík committed
63
64
65
66
67

You can either use your existing image or create new image from scratch.

QEMU currently supports these image types or formats:

Lukáš Krupčík's avatar
* -> *    
Lukáš Krupčík committed
68
69
70
71
72
73
74
* raw
* cloop
* cow
* qcow
* qcow2
* vmdk - VMware 3 & 4, or 6 image format, for exchanging images with that product
* vdi - VirtualBox 1.1 compatible image format, for exchanging images with VirtualBox.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
75
76

You can convert your existing image using qemu-img convert command. Supported formats of this command are: blkdebug blkverify bochs cloop cow dmg file ftp ftps host_cdrom host_device host_floppy http https nbd parallels qcow qcow2 qed raw sheepdog tftp vdi vhdx vmdk vpc vvfat.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
77
78
79

We recommend using advanced QEMU native image format qcow2.

Lukáš Krupčík's avatar
Lukáš Krupčík committed
80
[More about QEMU Images](http://en.wikibooks.org/wiki/QEMU/Images)
Lukáš Krupčík's avatar
Lukáš Krupčík committed
81

David Hrbáč's avatar
David Hrbáč committed
82
### Optimize Image of Your Virtual Machine
Lukáš Krupčík's avatar
Lukáš Krupčík committed
83

Lukáš Krupčík's avatar
Lukáš Krupčík committed
84
Use virtio devices (for disk/drive and network adapter) and install virtio drivers (paravirtualized drivers) into virtual machine. There is significant performance gain when using virtio drivers. For more information see [Virtio Linux](http://www.linux-kvm.org/page/Virtio) and [Virtio Windows](http://www.linux-kvm.org/page/WindowsGuestDrivers/Download_Drivers).
Lukáš Krupčík's avatar
Lukáš Krupčík committed
85

Lukáš Krupčík's avatar
Lukáš Krupčík committed
86
Disable all unnecessary services and tasks. Restrict all unnecessary operating system operations.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
87

Lukáš Krupčík's avatar
Lukáš Krupčík committed
88
Remove all unnecessary software and files.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
89

Lukáš Krupčík's avatar
Lukáš Krupčík committed
90
Remove all paging space, swap files, partitions, etc.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
91

Lukáš Krupčík's avatar
Lukáš Krupčík committed
92
Shrink your image. (It is recommended to zero all free space and reconvert image using qemu-img.)
Lukáš Krupčík's avatar
Lukáš Krupčík committed
93

David Hrbáč's avatar
David Hrbáč committed
94
### Modify Your Image for Running Jobs
Lukáš Krupčík's avatar
Lukáš Krupčík committed
95

Lukáš Krupčík's avatar
Lukáš Krupčík committed
96
Your image should run some kind of operating system startup script. Startup script should run application and when application exits run shutdown or quit virtual machine.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
97
98
99

We recommend, that startup script

Lukáš Krupčík's avatar
* -> *    
Lukáš Krupčík committed
100
101
* maps Job Directory from host (from compute node)
* runs script (we call it "run script") from Job Directory and waits for application's exit
Lukáš Krupčík's avatar
Lukáš Krupčík committed
102
  * for management purposes if run script does not exist wait for some time period (few minutes)
Lukáš Krupčík's avatar
* -> *    
Lukáš Krupčík committed
103
* shutdowns/quits OS
Lukáš Krupčík's avatar
Lukáš Krupčík committed
104

Lukáš Krupčík's avatar
Lukáš Krupčík committed
105
For Windows operating systems we suggest using Local Group Policy Startup script, for Linux operating systems rc.local, runlevel init script or similar service.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
106
107
108

Example startup script for Windows virtual machine:

David Hrbáč's avatar
David Hrbáč committed
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
```bat
@echo off
set LOG=c:\startup.log
set MAPDRIVE=z:
set SCRIPT=%MAPDRIVE%\run.bat
set TIMEOUT=300

echo %DATE% %TIME% Running startup script>%LOG%

rem Mount share
echo %DATE% %TIME% Mounting shared drive>>%LOG%
net use z: \\10.0.2.4\qemu >>%LOG% 2>&1
dir z:\ >>%LOG% 2>&1
echo. >>%LOG%


if exist %MAPDRIVE%\ (
  echo %DATE% %TIME% The drive "%MAPDRIVE%" exists>>%LOG%

  if exist %SCRIPT% (
    echo %DATE% %TIME% The script file "%SCRIPT%"exists>>%LOG%
    echo %DATE% %TIME% Running script %SCRIPT%>>%LOG%
    set TIMEOUT=0
    call %SCRIPT%
  ) else (
    echo %DATE% %TIME% The script file "%SCRIPT%"does not exist>>%LOG%
  )

) else (
  echo %DATE% %TIME% The drive "%MAPDRIVE%" does not exist>>%LOG%
)
echo. >>%LOG%

timeout /T %TIMEOUT%

echo %DATE% %TIME% Shut down>>%LOG%
shutdown /s /t 0
Lukáš Krupčík's avatar
Lukáš Krupčík committed
146
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
147

Lukáš Krupčík's avatar
Lukáš Krupčík committed
148
Example startup script maps shared job script as drive z: and looks for run script called run.bat. If run script is found it is run else wait for 5 minutes, then shutdown virtual machine.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
149

David Hrbáč's avatar
David Hrbáč committed
150
### Create Job Script for Executing Virtual Machine
Lukáš Krupčík's avatar
Lukáš Krupčík committed
151

Lukáš Krupčík's avatar
Lukáš Krupčík committed
152
Create job script according recommended
Lukáš Krupčík's avatar
Lukáš Krupčík committed
153

Josef Hrabal's avatar
Josef Hrabal committed
154
[Virtual Machine Job Workflow](virtualization#virtual-machine-job-workflow).
Lukáš Krupčík's avatar
Lukáš Krupčík committed
155
156
157

Example job for Windows virtual machine:

David Hrbáč's avatar
David Hrbáč committed
158
```bat
Lukáš Krupčík's avatar
Lukáš Krupčík committed
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
    #/bin/sh

    JOB_DIR=/scratch/$USER/win/${PBS_JOBID}

    #Virtual machine settings
    VM_IMAGE=~/work/img/win.img
    VM_MEMORY=49152
    VM_SMP=16

    # Prepare job dir
    mkdir -p ${JOB_DIR} && cd ${JOB_DIR} || exit 1
    ln -s ~/work/win .
    ln -s /scratch/$USER/data .
    ln -s ~/work/win/script/run/run-appl.bat run.bat

    # Run virtual machine
    export TMPDIR=/lscratch/${PBS_JOBID}
    module add qemu
Lukáš Krupčík's avatar
Lukáš Krupčík committed
177
    qemu-system-x86_64
David Hrbáč's avatar
David Hrbáč committed
178
179
180
181
182
183
184
185
186
187
188
189
      -enable-kvm
      -cpu host
      -smp ${VM_SMP}
      -m ${VM_MEMORY}
      -vga std
      -localtime
      -usb -usbdevice tablet
      -device virtio-net-pci,netdev=net0
      -netdev user,id=net0,smb=${JOB_DIR},hostfwd=tcp::3389-:3389
      -drive file=${VM_IMAGE},media=disk,if=virtio
      -snapshot
      -nographic
Lukáš Krupčík's avatar
Lukáš Krupčík committed
190
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
191

Lukáš Krupčík's avatar
Lukáš Krupčík committed
192
Job script links application data (win), input data (data) and run script (run.bat) into job directory and runs virtual machine.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
193
194
195

Example run script (run.bat) for Windows virtual machine:

David Hrbáč's avatar
David Hrbáč committed
196
```doscon
Lukáš Krupčík's avatar
Lukáš Krupčík committed
197
198
199
    z:
    cd winappl
    call application.bat z:data z:output
Lukáš Krupčík's avatar
Lukáš Krupčík committed
200
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
201

Lukáš Krupčík's avatar
->    
Lukáš Krupčík committed
202
Run script runs application from shared job directory (mapped as drive z:), process input data (z:data) from job directory and store output to job directory (z:output).
Lukáš Krupčík's avatar
Lukáš Krupčík committed
203

David Hrbáč's avatar
David Hrbáč committed
204
### Run Jobs
Lukáš Krupčík's avatar
Lukáš Krupčík committed
205

Josef Hrabal's avatar
Josef Hrabal committed
206
Run jobs as usual, see  [Resource Allocation and Job Execution](/salomon/job-submission-and-execution/). Use only full node allocation for virtualization jobs.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
207
208
209

### Running Virtual Machines

Lukáš Krupčík's avatar
Lukáš Krupčík committed
210
Virtualization is enabled only on compute nodes, virtualization does not work on login nodes.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
211
212
213

Load QEMU environment module:

Lukáš Krupčík's avatar
Lukáš Krupčík committed
214
215
```console
$ module add qemu
Lukáš Krupčík's avatar
Lukáš Krupčík committed
216
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
217
218
219

Get help

Lukáš Krupčík's avatar
Lukáš Krupčík committed
220
221
```console
$ man qemu
Lukáš Krupčík's avatar
Lukáš Krupčík committed
222
```
David Hrbáč's avatar
David Hrbáč committed
223

Lukáš Krupčík's avatar
Lukáš Krupčík committed
224
225
Run virtual machine (simple)

Lukáš Krupčík's avatar
Lukáš Krupčík committed
226
227
228
```console
$ qemu-system-x86_64 -hda linux.img -enable-kvm -cpu host -smp 16 -m 32768 -vga std -vnc :0
$ qemu-system-x86_64 -hda win.img   -enable-kvm -cpu host -smp 16 -m 32768 -vga std -localtime -usb -usbdevice tablet -vnc :0
Lukáš Krupčík's avatar
Lukáš Krupčík committed
229
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
230

Pavel Jirásek's avatar
Pavel Jirásek committed
231
You can access virtual machine by VNC viewer (option -vnc) connecting to IP address of compute node. For VNC you must use VPN network.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
232

David Hrbáč's avatar
David Hrbáč committed
233
Install virtual machine from ISO file
Lukáš Krupčík's avatar
Lukáš Krupčík committed
234

Lukáš Krupčík's avatar
Lukáš Krupčík committed
235
236
237
```console
$ qemu-system-x86_64 -hda linux.img -enable-kvm -cpu host -smp 16 -m 32768 -vga std -cdrom linux-install.iso -boot d -vnc :0
$ qemu-system-x86_64 -hda win.img   -enable-kvm -cpu host -smp 16 -m 32768 -vga std -localtime -usb -usbdevice tablet -cdrom win-install.iso -boot d -vnc :0
Lukáš Krupčík's avatar
Lukáš Krupčík committed
238
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
239

David Hrbáč's avatar
David Hrbáč committed
240
Run virtual machine using optimized devices, user network back-end with sharing and port forwarding, in snapshot mode
Lukáš Krupčík's avatar
Lukáš Krupčík committed
241

Lukáš Krupčík's avatar
Lukáš Krupčík committed
242
243
244
```console
$ qemu-system-x86_64 -drive file=linux.img,media=disk,if=virtio -enable-kvm -cpu host -smp 16 -m 32768 -vga std -device virtio-net-pci,netdev=net0 -netdev user,id=net0,smb=/scratch/$USER/tmp,hostfwd=tcp::2222-:22 -vnc :0 -snapshot
$ qemu-system-x86_64 -drive file=win.img,media=disk,if=virtio -enable-kvm -cpu host -smp 16 -m 32768 -vga std -localtime -usb -usbdevice tablet -device virtio-net-pci,netdev=net0 -netdev user,id=net0,smb=/scratch/$USER/tmp,hostfwd=tcp::3389-:3389 -vnc :0 -snapshot
Lukáš Krupčík's avatar
Lukáš Krupčík committed
245
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
246

Pavel Jirásek's avatar
Pavel Jirásek committed
247
Thanks to port forwarding you can access virtual machine via SSH (Linux) or RDP (Windows) connecting to IP address of compute node (and port 2222 for SSH). You must use VPN network).
Lukáš Krupčík's avatar
Lukáš Krupčík committed
248

David Hrbáč's avatar
David Hrbáč committed
249
!!! note
Lukáš Krupčík's avatar
Lukáš Krupčík committed
250
    Keep in mind, that if you use virtio devices, you must have virtio drivers installed on your virtual machine.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
251

David Hrbáč's avatar
David Hrbáč committed
252
### Networking and Data Sharing
Lukáš Krupčík's avatar
Lukáš Krupčík committed
253

David Hrbáč's avatar
David Hrbáč committed
254
For networking virtual machine we suggest to use (default) user network back-end (sometimes called slirp). This network back-end NATs virtual machines and provides useful services for virtual machines as DHCP, DNS, SMB sharing, port forwarding.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
255

Lukáš Krupčík's avatar
Lukáš Krupčík committed
256
In default configuration IP network 10.0.2.0/24 is used, host has IP address 10.0.2.2, DNS server 10.0.2.3, SMB server 10.0.2.4 and virtual machines obtain address from range 10.0.2.15-10.0.2.31. Virtual machines have access to Anselm's network via NAT on compute node (host).
Lukáš Krupčík's avatar
Lukáš Krupčík committed
257
258
259

Simple network setup

Lukáš Krupčík's avatar
Lukáš Krupčík committed
260
261
```console
$ qemu-system-x86_64 ... -net nic -net user
Lukáš Krupčík's avatar
Lukáš Krupčík committed
262
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
263
264
265

(It is default when no -net options are given.)

Lukáš Krupčík's avatar
Lukáš Krupčík committed
266
Simple network setup with sharing and port forwarding (obsolete but simpler syntax, lower performance)
Lukáš Krupčík's avatar
Lukáš Krupčík committed
267

Lukáš Krupčík's avatar
Lukáš Krupčík committed
268
269
```console
$ qemu-system-x86_64 ... -net nic -net user,smb=/scratch/$USER/tmp,hostfwd=tcp::3389-:3389
Lukáš Krupčík's avatar
Lukáš Krupčík committed
270
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
271
272
273

Optimized network setup with sharing and port forwarding

Lukáš Krupčík's avatar
Lukáš Krupčík committed
274
275
```console
$ qemu-system-x86_64 ... -device virtio-net-pci,netdev=net0 -netdev user,id=net0,smb=/scratch/$USER/tmp,hostfwd=tcp::2222-:22
Lukáš Krupčík's avatar
Lukáš Krupčík committed
276
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
277

David Hrbáč's avatar
David Hrbáč committed
278
### Advanced Networking
Lukáš Krupčík's avatar
Lukáš Krupčík committed
279

David Hrbáč's avatar
David Hrbáč committed
280
#### Internet Access
Lukáš Krupčík's avatar
Lukáš Krupčík committed
281

David Hrbáč's avatar
David Hrbáč committed
282
Sometime your virtual machine needs access to internet (install software, updates, software activation, etc). We suggest solution using Virtual Distributed Ethernet (VDE) enabled QEMU with SLIRP running on login node tunneled to compute node. Be aware, this setup has very low performance, the worst performance of all described solutions.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
283

Lukáš Krupčík's avatar
Lukáš Krupčík committed
284
Load VDE enabled QEMU environment module (unload standard QEMU module first if necessary).
Lukáš Krupčík's avatar
Lukáš Krupčík committed
285

Lukáš Krupčík's avatar
Lukáš Krupčík committed
286
287
```console
$ module add qemu/2.1.2-vde2
Lukáš Krupčík's avatar
Lukáš Krupčík committed
288
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
289
290
291

Create virtual network switch.

Lukáš Krupčík's avatar
Lukáš Krupčík committed
292
293
```console
$ vde_switch -sock /tmp/sw0 -mgmt /tmp/sw0.mgmt -daemon
Lukáš Krupčík's avatar
Lukáš Krupčík committed
294
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
295

Lukáš Krupčík's avatar
Lukáš Krupčík committed
296
Run SLIRP daemon over SSH tunnel on login node and connect it to virtual network switch.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
297

Lukáš Krupčík's avatar
Lukáš Krupčík committed
298
299
```console
$ dpipe vde_plug /tmp/sw0 = ssh login1 $VDE2_DIR/bin/slirpvde -s - --dhcp &
Lukáš Krupčík's avatar
Lukáš Krupčík committed
300
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
301

David Hrbáč's avatar
David Hrbáč committed
302
Run qemu using vde network back-end, connect to created virtual switch.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
303
304
305

Basic setup (obsolete syntax)

Lukáš Krupčík's avatar
Lukáš Krupčík committed
306
307
```console
$ qemu-system-x86_64 ... -net nic -net vde,sock=/tmp/sw0
Lukáš Krupčík's avatar
Lukáš Krupčík committed
308
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
309
310
311

Setup using virtio device (obsolete syntax)

Lukáš Krupčík's avatar
Lukáš Krupčík committed
312
313
```console
$ qemu-system-x86_64 ... -net nic,model=virtio -net vde,sock=/tmp/sw0
Lukáš Krupčík's avatar
Lukáš Krupčík committed
314
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
315
316
317

Optimized setup

Lukáš Krupčík's avatar
Lukáš Krupčík committed
318
319
```console
$ qemu-system-x86_64 ... -device virtio-net-pci,netdev=net0 -netdev vde,id=net0,sock=/tmp/sw0
Lukáš Krupčík's avatar
Lukáš Krupčík committed
320
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
321

David Hrbáč's avatar
David Hrbáč committed
322
#### TAP Interconnect
Lukáš Krupčík's avatar
Lukáš Krupčík committed
323

David Hrbáč's avatar
David Hrbáč committed
324
Both user and vde network back-end have low performance. For fast interconnect (10 Gbit/s and more) of compute node (host) and virtual machine (guest) we suggest using Linux kernel TAP device.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
325

Lukáš Krupčík's avatar
Lukáš Krupčík committed
326
Cluster Anselm provides TAP device tap0 for your job. TAP interconnect does not provide any services (like NAT, DHCP, DNS, SMB, etc.) just raw networking, so you should provide your services if you need them.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
327

328
329
330
331
332
333
To enable TAP interconect feature you need to specify virt_network=True PBS resource at job submit.

```console
$ qsub ... -l virt_network=True
```

David Hrbáč's avatar
David Hrbáč committed
334
Run qemu with TAP network back-end:
Lukáš Krupčík's avatar
Lukáš Krupčík committed
335

Lukáš Krupčík's avatar
Lukáš Krupčík committed
336
337
```console
$ qemu-system-x86_64 ... -device virtio-net-pci,netdev=net1 -netdev tap,id=net1,ifname=tap0,script=no,downscript=no
Lukáš Krupčík's avatar
Lukáš Krupčík committed
338
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
339

Lukáš Krupčík's avatar
Lukáš Krupčík committed
340
Interface tap0 has IP address 192.168.1.1 and network mask 255.255.255.0 (/24). In virtual machine use IP address from range 192.168.1.2-192.168.1.254. For your convenience some ports on tap0 interface are redirected to higher numbered ports, so you as non-privileged user can provide services on these ports.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
341
342
343

Redirected ports:

Lukáš Krupčík's avatar
* -> *    
Lukáš Krupčík committed
344
345
346
* DNS udp/53->udp/3053, tcp/53->tcp3053
* DHCP udp/67->udp3067
* SMB tcp/139->tcp3139, tcp/445->tcp3445).
Lukáš Krupčík's avatar
Lukáš Krupčík committed
347

Lukáš Krupčík's avatar
Lukáš Krupčík committed
348
You can configure IP address of virtual machine statically or dynamically. For dynamic addressing provide your DHCP server on port 3067 of tap0 interface, you can also provide your DNS server on port 3053 of tap0 interface for example:
Lukáš Krupčík's avatar
Lukáš Krupčík committed
349

Lukáš Krupčík's avatar
Lukáš Krupčík committed
350
351
```console
$ dnsmasq --interface tap0 --bind-interfaces -p 3053 --dhcp-alternate-port=3067,68 --dhcp-range=192.168.1.15,192.168.1.32 --dhcp-leasefile=/tmp/dhcp.leasefile
Lukáš Krupčík's avatar
Lukáš Krupčík committed
352
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
353

Lukáš Krupčík's avatar
Lukáš Krupčík committed
354
You can also provide your SMB services (on ports 3139, 3445) to obtain high performance data sharing.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
355
356
357

Example smb.conf (not optimized)

Lukáš Krupčík's avatar
Lukáš Krupčík committed
358
359
360
```console
$ cat smb.conf

Lukáš Krupčík's avatar
Lukáš Krupčík committed
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
    [global]
    socket address=192.168.1.1
    smb ports = 3445 3139

    private dir=/tmp/qemu-smb
    pid directory=/tmp/qemu-smb
    lock directory=/tmp/qemu-smb
    state directory=/tmp/qemu-smb
    ncalrpc dir=/tmp/qemu-smb/ncalrpc
    log file=/tmp/qemu-smb/log.smbd
    smb passwd file=/tmp/qemu-smb/smbpasswd
    security = user
    map to guest = Bad User
    unix extensions = no
    load printers = no
    printing = bsd
    printcap name = /dev/null
    disable spoolss = yes
    log level = 1
    guest account = USER
    [qemu]
    path=/scratch/USER/tmp
    read only=no
    guest ok=yes
    writable=yes
    follow symlinks=yes
    wide links=yes
    force user=USER
Lukáš Krupčík's avatar
Lukáš Krupčík committed
389
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
390
391
392
393
394

(Replace USER with your login name.)

Run SMB services

Lukáš Krupčík's avatar
Lukáš Krupčík committed
395
396
```console
$ smbd -s /tmp/qemu-smb/smb.conf
Lukáš Krupčík's avatar
Lukáš Krupčík committed
397
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
398

David Hrbáč's avatar
David Hrbáč committed
399
Virtual machine can of course have more than one network interface controller, virtual machine can use more than one network back-end. So, you can combine for example use network back-end and TAP interconnect.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
400

David Hrbáč's avatar
David Hrbáč committed
401
### Snapshot Mode
Lukáš Krupčík's avatar
Lukáš Krupčík committed
402

Lukáš Krupčík's avatar
Lukáš Krupčík committed
403
In snapshot mode image is not written, changes are written to temporary file (and discarded after virtual machine exits). **It is strongly recommended mode for running your jobs.** Set TMPDIR environment variable to local scratch directory for placement temporary files.
Lukáš Krupčík's avatar
Lukáš Krupčík committed
404

Lukáš Krupčík's avatar
Lukáš Krupčík committed
405
406
407
```console
$ export TMPDIR=/lscratch/${PBS_JOBID}
$ qemu-system-x86_64 ... -snapshot
Lukáš Krupčík's avatar
Lukáš Krupčík committed
408
```
Lukáš Krupčík's avatar
Lukáš Krupčík committed
409

David Hrbáč's avatar
David Hrbáč committed
410
### Windows Guests
Lukáš Krupčík's avatar
Lukáš Krupčík committed
411
412
413

For Windows guests we recommend these options, life will be easier:

Lukáš Krupčík's avatar
Lukáš Krupčík committed
414
415
```console
$ qemu-system-x86_64 ... -localtime -usb -usbdevice tablet
Lukáš Krupčík's avatar
Lukáš Krupčík committed
416
```