virtualization.md

Virtualization

Running virtual machines on compute nodes.

Introduction

There are situations when our clusters' environment is not suitable for user's needs:

• Application requires a different operating system (e.g. Windows) or it is not available for Linux;
• Application requires a different versions of base system libraries and tools;
• Application requires specific setup (installation, configuration) of a complex software stack;
• Application requires privileged access to the operating system;
• ... and combinations of the above cases.

The solution for these cases is virtualization. Clusters' environment allows to run virtual machines on compute nodes. Users can create their own images of the operating system with a specific software stack and run instances of these images as virtual machines on compute nodes. Run of virtual machines is provided by the standard mechanism of Resource Allocation and Job Execution.

Solution is based on QEMU-KVM software stack and provides hardware-assisted x86 virtualization.

Limitations

Clusters' infrastructure and environment were s not designed for virtualization. Compute nodes, storages, and infrastructure is intended and optimized for running HPC jobs. This implies suboptimal configuration of virtualization and limitations.

Clusters' virtualization does not provide the performance and all features of a native environment. There is significant a performance hit (degradation) in I/O performance (storage, network). The virtualization is not suitable for I/O (disk, network) intensive workloads.

Virtualization has some drawbacks, as well. It is not so easy to set up an efficient solution.

The solution described in the HOWTO section is suitable for single node tasks; it does not introduce virtual machine clustering.

!!! note Consider virtualization as a last resort solution for your needs.

!!! warning Consult use of virtualization with IT4Innovations' support.

For running a Windows application (when the source code and Linux native application are not available), consider use of Wine, Windows compatibility layer. Many Windows applications can be run using Wine with less effort and better performance than when using virtualization.

Licensing

IT4Innovations does not provide any licenses for operating systems and software of virtual machines. Users are (in accordance with Acceptable use policy document) fully responsible for licensing all software running on virtual machines on clusters. Be aware of complex conditions of licensing software in virtual environments.

!!! note Users are responsible for licensing OS (e.g. MS Windows) and all software running on their virtual machines.

Howto

Virtual Machine Job Workflow

We propose this job workflow:

Our recommended solution is that the job script creates a distinct shared job directory, which makes a central point for data exchange between cluster's environment, compute node (host) (e.g. HOME, SCRATCH, local scratch, and other local or cluster file systems), and virtual machine (guest). The job script links or copies input data and instructions on what to do (run script) for the virtual machine to the job directory and the virtual machine process input data according to the instructions in the job directory and store output back to the job directory. We recommend that the virtual machine is running in a so called snapshot mode, the image is immutable - it does not change, so one image can be used for many concurrent jobs.

Procedure

1. Prepare the image of your virtual machine
2. Optimize the image of your virtual machine for cluster's virtualization
3. Modify your image for running jobs
4. Create a job script for executing the virtual machine
5. Run jobs

Prepare Image of Your Virtual Machine

You can either use your existing image or create a new image from scratch.

QEMU currently supports these image types or formats:

• raw
• cloop
• cow
• qcow
• qcow2
• vmdk - VMware 3 & 4, or 6 image format, for exchanging images with that product
• vdi - VirtualBox 1.1 compatible image format, for exchanging images with VirtualBox.

You can convert your existing image using the qemu-img convert command. Supported formats of this command are: blkdebug blkverify bochs cloop cow dmg file ftp ftps host_cdrom host_device host_floppy http https nbd parallels qcow qcow2 qed raw sheepdog tftp vdi vhdx vmdk vpc vvfat.

We recommend using an advanced QEMU native image format qcow2.

More about QEMU images here.

Optimize Image of Your Virtual Machine

Use virtio devices (for the disk/drive and the network adapter) and install virtio drivers (paravirtualized drivers) into the virtual machine. There is a significant performance gain when using virtio drivers. For more information, see Virtio Linux and Virtio Windows.

Disable all unnecessary services and tasks. Restrict all unnecessary operating system operations.

Remove all unnecessary software and files.

Remove all paging space, swap files, partitions, etc.

Shrink your image. (It is recommended to zero all free space and reconvert the image using qemu-img.)

Modify Your Image for Running Jobs

Your image should contian an operating system startup script. The startup script should run the application and when the application exits, run shutdown or quit the virtual machine.

We recommend that the startup script:

• maps Job Directory from host (from compute node);
• runs script (we call it "run script") from Job Directory and waits for application's exit;
  • for management purposes if the run script does not exist wait for some time period (few minutes);
• shutdowns/quits OS.

For Windows operating systems, we suggest using a Local Group Policy Startup script; for Linux operating systems, use the rc.local, runlevel init script, or similar service.

Example startup script for the Windows virtual machine:

@echo off
set LOG=c:\startup.log
set MAPDRIVE=z:
set SCRIPT=%MAPDRIVE%\run.bat
set TIMEOUT=300

echo %DATE% %TIME% Running startup script>%LOG%

rem Mount share
echo %DATE% %TIME% Mounting shared drive>>%LOG%
net use z: \\10.0.2.4\qemu >>%LOG% 2>&1
dir z:\ >>%LOG% 2>&1
echo. >>%LOG%


if exist %MAPDRIVE%\ (
  echo %DATE% %TIME% The drive "%MAPDRIVE%" exists>>%LOG%

  if exist %SCRIPT% (
    echo %DATE% %TIME% The script file "%SCRIPT%"exists>>%LOG%
    echo %DATE% %TIME% Running script %SCRIPT%>>%LOG%
    set TIMEOUT=0
    call %SCRIPT%
  ) else (
    echo %DATE% %TIME% The script file "%SCRIPT%"does not exist>>%LOG%
  )

) else (
  echo %DATE% %TIME% The drive "%MAPDRIVE%" does not exist>>%LOG%
)
echo. >>%LOG%

timeout /T %TIMEOUT%

echo %DATE% %TIME% Shut down>>%LOG%
shutdown /s /t 0

The example startup script maps a shared job script as a drive z: and looks for a run script called run.bat. If the run script is found, it is run, otherwise it waits for 5 minutes, then shuts down the virtual machine.

Create Job Script for Executing Virtual Machine

Create the job script according to the recommended Virtual Machine Job Workflow.

Example job for the Windows virtual machine:

    #/bin/sh

    JOB_DIR=/scratch/$USER/win/${PBS_JOBID}

    #Virtual machine settings
    VM_IMAGE=~/work/img/win.img
    VM_MEMORY=49152
    VM_SMP=16

    # Prepare job dir
    mkdir -p ${JOB_DIR} && cd ${JOB_DIR} || exit 1
    ln -s ~/work/win .
    ln -s /scratch/$USER/data .
    ln -s ~/work/win/script/run/run-appl.bat run.bat

    # Run virtual machine
    export TMPDIR=/lscratch/${PBS_JOBID}
    module add qemu
    qemu-system-x86_64
      -enable-kvm
      -cpu host
      -smp ${VM_SMP}
      -m ${VM_MEMORY}
      -vga std
      -localtime
      -usb -usbdevice tablet
      -device virtio-net-pci,netdev=net0
      -netdev user,id=net0,smb=${JOB_DIR},hostfwd=tcp::3389-:3389
      -drive file=${VM_IMAGE},media=disk,if=virtio
      -snapshot
      -nographic

The job script links application data (win), input data (data), and run script (run.bat) into the job directory and runs the virtual machine.

Example run script (run.bat) for the Windows virtual machine:

    z:
    cd winappl
    call application.bat z:data z:output

The run script runs the application from the shared job directory (mapped as drive z:), processes the input data (z:data) from the job directory, and stores output to the job directory (z:output).

Run Jobs

Run jobs as usual, see Resource Allocation and Job Execution. Use only full node allocation for virtualization jobs.

Running Virtual Machines

Virtualization is enabled only on compute nodes, virtualization does not work on login nodes.

Load the QEMU environment module:

$ module add qemu

Get help:

$ man qemu

Run the virtual machine (simple):

$ qemu-system-x86_64 -hda linux.img -enable-kvm -cpu host -smp 16 -m 32768 -vga std -vnc :0
$ qemu-system-x86_64 -hda win.img   -enable-kvm -cpu host -smp 16 -m 32768 -vga std -localtime -usb -usbdevice tablet -vnc :0

You can access the virtual machine via the VNC viewer (the -vnc option) connecting to the IP address of the compute node. For VNC, you must use a VPN network.

Install the virtual machine from the ISO file:

$ qemu-system-x86_64 -hda linux.img -enable-kvm -cpu host -smp 16 -m 32768 -vga std -cdrom linux-install.iso -boot d -vnc :0
$ qemu-system-x86_64 -hda win.img   -enable-kvm -cpu host -smp 16 -m 32768 -vga std -localtime -usb -usbdevice tablet -cdrom win-install.iso -boot d -vnc :0

Run the virtual machine using optimized devices, user network back-end with sharing and port forwarding, in snapshot mode

$ qemu-system-x86_64 -drive file=linux.img,media=disk,if=virtio -enable-kvm -cpu host -smp 16 -m 32768 -vga std -device virtio-net-pci,netdev=net0 -netdev user,id=net0,smb=/scratch/$USER/tmp,hostfwd=tcp::2222-:22 -vnc :0 -snapshot
$ qemu-system-x86_64 -drive file=win.img,media=disk,if=virtio -enable-kvm -cpu host -smp 16 -m 32768 -vga std -localtime -usb -usbdevice tablet -device virtio-net-pci,netdev=net0 -netdev user,id=net0,smb=/scratch/$USER/tmp,hostfwd=tcp::3389-:3389 -vnc :0 -snapshot

Port forwarding allows you to access the virtual machine via SSH (Linux) or RDP (Windows) connecting to the IP address of the compute node (and port 2222 for SSH). You must use a VPN network).

!!! note Keep in mind, that if you use virtio devices, you must have virtio drivers installed on your virtual machine.

Networking and Data Sharing

For a networking virtual machine, we suggest using a (default) user network back-end (sometimes called slirp). This network back-end NATs virtual machines and provides useful services for virtual machines as DHCP, DNS, SMB sharing, port forwarding.

In default configuration, IP network 10.0.2.0/24 is used, host has IP address 10.0.2.2, DNS server 10.0.2.3, SMB server 10.0.2.4 and virtual machines obtain address from range 10.0.2.15-10.0.2.31. Virtual machines have access to cluster's network via NAT on the compute node (host).

Simple network setup:

$ qemu-system-x86_64 ... -net nic -net user

(It is default when no -net options are given.)

Simple network setup with sharing and port forwarding (obsolete but simpler syntax, lower performance):

$ qemu-system-x86_64 ... -net nic -net user,smb=/scratch/$USER/tmp,hostfwd=tcp::3389-:3389

Optimized network setup with sharing and port forwarding: