User virtual machines on VLAB servers

Draft/test user-defined virtual machine setup for VLAB servers. Operational on vx01.

Virtual machine and virtual network configurations created by vmexec

Implement and run user-defined virtual machines (VM). Link to examples.

TL;DR

• VMs run as x86_64 Kernel Virtual Machines (KVM) under qemu.
• Can install VMs from scratch using ISO files as boot media.
• Virtual hard disk files and and CD/ISO image files *must* be in user subdirectories located somewhere under the directory listed below.
• Virtual hard disk files must be in qcow2 format.
• Three network configurations (see diagram):
  • Standalone VM using NAT to communicate with the outside world (-z0),
  • Gateway VM with one network interface using NAT to communicate with the outside world (as above), plus second network interface connected to a dedicated logical bridge creating a private virtual network (-z1), and
  • Internal/private VM connecting to the dedicated logical bridge (see previous) (-z2).
  • The standalone and gateway configurations support using port forwarding to allow connections from the outside world to the VM's (such as to SSH servers or web servers running on the VM's).
• qemu-img is used to create virtual hard disk files.
• vmexecmkdir is used to create a user subdirectory on a host if one doesn't already exist.
• vmexec runs VMs on a host.

CD/ISO files and virtual hard disk files location

The CD/ISO files and virtual hard disk files used by the user-created virtual machines must actually be located somewhere under:

• /localstorage

on the host where the VMs will be run. If not, vmexec will give an error when trying to run the VM. Symlink-ing to locations outside of this directory will not work and will cause the same error.

Typically, users will have their own subdirectories under this directory.

Using the local storage location is enforced to ensure that virtual machine disk images are on local, high-sopeed disk storage.

Note that the local storage is NOT BACKED UP.

Programs

Program name	Description
qemu-img	Create a virtual hard disk (file). Can use a pre-existing disk file, say from a class account, as a starting point.
vmexecmkdir	Create a user subdirectory for VM disk images and ISO files.
vmexec	Run a virtual machine, booting either from a supplied CD/ISO image or from a given virtual hard disk.

qemu-img

See the man page, but...

qemu-img -f qcow2 <disk_file_path> 20G

will create an empty 20G qcow2 disk.

And:

qemu-img -f qcow2 -b <base_file_path> <disk_file_path>

creates a disk whose initial contents are based on the given base disk image (such as in a class account). The size of the created disk will be that specified when the base file was created.

vmexecmkdir

Create a user subdirectory on a host for ISO files and virtual hard disk files (see location above).

Usage: vmexecmkdir [options]

-h  Display this help
-c  Create user directory in <see above>

vmexec

Usage: vmexec [options] [<disk0> [<disk1> ...]]

-h               Display this help
-m <n>           RAM megabytes (128/2048 - default 1024)
-n <n>           Number of virtual CPUs (1/2 - default 1)
-k <kernelpath>  Path to kernel image
-i <isopath>     CD/ISO file path
-d               Boot from CD/ISO
-x               Set display to "none"
-z <0|1|2>       Set network configuration (0=default/NAT, 1=gateway, or 2=internal)
-f <port1:port2> Set TCP port forwarding from external/host
                 port <port1> to internal/VM port <port2>.
                 May be used more than once. <port1> must be
                 between 8000 and 8999, inclusive.
                 Cannot be used with -z2

vmexec runs a user-defined virtual machine using qemu/KVM.

• The VM's hard disks are connected via virtio block interface.
• It will have a graphical console.
• The [virtual] CPU will be Intel. There can be a maximum of TWO per VM.
• Disk and ISO paths can be absolute, or relative to the user's subdirectory (e.g. my-vms/disk0.img, ../comp1511/example.img, etc.)

Virtual networking

See also the diagram above.

The VM's are created with one of three possible networking configurations:

0	The default configuration. The VM is standalone (cannot communicate with other VM's belonging to the same user) and connects to the outside world through the physical host's own networking connection using NAT. Port forwarding can be configured (-f) to allow TCP connections from the outside world to the VM.
1	This is the gateway configuration where the VM has two virtual network interfaces. The first, like in the default configuration above, uses NAT to communicate with the outside world. The second network interface connects to a dedicated, private, logical network bridge belonging to the user (created at the same time the VM is created). This second network interface can be used as the gateway for the user's private VM's (see next configuration) to connect to the outside world.
2	This is the internal/private VM configuration where the VM's network interface connects only to the private network created for a gateway VM (see previous).

Examples

These examples are sketches only and have in no way been checked. Modelled on Debian Bookworm installation.

Create your own directory in local storage

$ /usr/local/infrastructure/bin/vmexecmkdir

The above will create a directory for you under /localstorage. E.g.,:

/localstorage/<yourusername>

You will be the owner of the directory and permissions will be set to allow you to create or copy whatever content you wish in it. Note that this directory is NOT BACKED UP.

Download install media (CD/DVD ISO)

The install media must be an ISO file.

For example:

$ curl -o /localstorage/<yourusername>/myinstall.iso http://mirror.aarnet.edu.au/pub/debian-cd/12.4.0/amd64/iso-cd/debian-12.4.0-amd64-netinst.iso

Install initial, standalone virtual machine

$ qemu-img create -f qcow2 /localstorage/<yourusername>/mystandalone.img 20G

$ /usr/local/infrastructure/bin/vmexec -d -i myinstall.iso mystandalone.img

Notes:

1. vmexec automatically assumes (and requires) that disk image files are relatively to, and located in, your own directory in private local storage.
2. The VM will have one Ethernet interface (NAT). Do install, then shutdown the VM.

Create gateway virtiual machine from standalone machine

$ cp -p /localstorage/<yourusername>/mystandalone.img /localstorage/<yourusername>/mygateway.img

$ /usr/local/infrastructure/bin/vmexec -z1 mygateway.img

VM will still have the NAT Ethernet interface, plus the VM will now have a second network interface on the private bridge (see diagram and discussion above). Configure the second network interface as the gateway for the private subnetwork (e.g., 192.168.1.1/24, /etc/network/interfaces), configure IP forwarding (/proc/sys/net/ipv4/ip_forward → /etc/sysctl.conf), masquerading (iptables -t nat -j MASQUERADE …), and create a DHCP server for the private subnetwork.

Leave the gateway VM running for when creating private VM's (next example).

Sample gateway /etc/network/interfaces file for Debian Bookworm

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug ens3
iface ens3 inet dhcp

# The internal subnetwork
allow-hotplug ens4
iface ens4 inet static
    address 192.168.1.1/24;

Enable IP forwarding on gateway for Debian Bookworm

In /etc/sysctl.conf:

net.ipv4.ip_forward=1

Sample gateway iptables configuration for Debian Bookworm

root@debian:/etc# iptables -t nat -L -n -v --line-numbers
Chain PREROUTING (policy ACCEPT 5 packets, 478 bytes)
num   pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 5 packets, 478 bytes)
num   pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 5 packets, 371 bytes)
num   pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 5 packets, 371 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 MASQUERADE  all  --  *      ens3    192.168.1.0/24       0.0.0.0/0           
root@debian:/etc#

Sample gateway DHCP server configuration for Debian Bookworm

Location: /etc/dhcp/dhcpd.conf

option domain-name "mynet.local";
option domain-name-servers 129.94.242.33;

default-lease-time 600;
max-lease-time 7200;

ddns-update-style none;

authoritative;

subnet 192.168.1.0 netmask 255.255.255.0 {
        option routers 192.168.1.1;
        range 192.168.1.10 192.168.1.30;
}

Create private virtual machine from standalone machine

$ cp -p /localstorage/<yourusername>/mystandalone.img /localstorage/<yourusername>/myprivatevm.img

$ /usr/local/infrastructure/bin/vmexec -z2 myprivatevm.img

If the original standalone VM was created as a DHCP client then, providing the gateway's networking and DHCP server have been set up correctly, the private VM should "just work".