Some onway router models provide a hypervisor to run virtual machines on the shared router hardware. The hypervisor builds upon the QEMU open source machine emulator and virtualizer 1. It uses a specific subset of the full QEMU functionality targeting embedded hardware, using a high degree of automation and focusing on reproducibility and availability. Further, virtual machines are deployed, configured and managed through an onway specific toolset.
The emulated hardware to run virtual machines in is restricted to:
Drivers for virtio-net/virtio-block are required in the guest. Linux guests usually have these included, Windows guests require explicit installation of these drivers.
Virtual machines on onway routers support different storage options. These are specifically designed to work in harsh environments and for deployment to large router fleets in a reproducible manner.
Every virtual machine builds upon a base image. Base images are usually deployed by the onway infrastructure in so-called bundles, along with files for other purposes. Bundles are compressed read-only squashfs filesystems that can be effectively synchronized to host storage and updated using binary delta updates.
Due to the properties of bundles (that is, compression, sparse file support, atomic updates etc.), base images do not require a sophisticated image format. Instead, RAW disk images can be efficiently stored in bundles, and bundles get deployed to router host systems.
Base images are read-only, though, and a running virtual machine is unable to write to such images. Instead, different Copy-on-Write overlays can be used that record writes to an upper layer.
By default, virtual machines get an in-memory Copy-on-Write overlay. Any changes to a disk are recorded in host RAM, only. Upon VM restart, any changes are lost, and the VM always starts into the freezed disk state from the RAW base image.
Using such a volatile overlay may be sufficient for many VMs, and provides a convenient way for running reproducible virtual machines across a large fleet of routers. Disk space for writing is limited, though, and a virtual machine must be limited in the amount of data it writes.
In other use-cases, it is required to store changes to the base image in persistent overlays. A persistent overlay is a file that stores the differences to the base image, only. On-disk overlays can be stacked to multiple layers, but a virtual machine always writes changes to the top overlay layer.
Overlay image storage is maintained by the host system, and overlays can be volatile or persistent. Volatile overlay images are similar to in-memory overlays in that the overlay is cleared on VM restart, but it uses on-disk storage and therefore may grow larger than overlays in host RAM.
Persistent overlays are preserved across VM (or host) restarts. VMs using such overlays must use filesystems that can handle power losses of the host gracefully, for example by using a filesystem journal or appropriate filesystem checking.
Persistent overlays build upon base images as well, and a base image may be updated. On base image changes, the persistent overlay is rendered invalid, as it is not compatible to the new base image. To avoid disk corruption, persistent overlays are cleared whenever a base image is changed.
In addition to OS base images with different overlays, a VM can be configured with additional disks. Such disks are created by the host system, but must be partitioned and formatted with appropriate filesystems by the guest VM operating system. There is no automated image deployment for plain disks; they may be deployed manually, but usually it is recommended to use base images in bundles for automated and efficient image updates, instead.
Using plain disks gives a maximum in flexibility; multiple disks may be combined to a single logical volume or using a software RAID setup. The host system does not provide explicit support for that, though, and such advanced storage mechanisms must be implemented completely in the guest. Plain disks may be resized by the host systems; it is up to the guest to adjust partitions and filesystems before or after such a change, though.
A virtual machine may be configured with one or more network adapters. The virtio-net based virtual Ethernet adapter in the guest is connected to a vmtap TAP interface on the host. The host side interface may be included in the network architecture using any mechanism provided by onway routers; they can be bridged to other interfaces, placed into a VRF, etc.
Virtual machines have an emulated ISA serial port. If the guest VM is configured accordingly, the serial port allows diagnosing or logging in to the virtual machine. The serial port is currently accessible from the host system only using a terminal emulator.
A more sophisticated administration interface is provided via VNC logins to the virtual machines. On the host system, VNC listening ports can be configured on different ports and VRFs to allow connecting with a VNC client. All clients connect to the primary VNC display driven by the standard VGA graphics adapter in the guest. For proper keyboard layout mappings, it is highly recommended to use a VNC client supporting the QEMU Extended Key Event Message extension, such as TigerVNC 3.
Running virtual machines on large router fleets brings some challenges compared to running arbitrary virtual machines in a more traditional way. VMs deployed to a large fleet usually implement the same functionality in every vehicle, being different only in the configuration of the virtual machines.
A key requirement when deploying VMs to hundreds of routers is reproducibility; every VM shall run VM images in an expected state. All VMs shall be equal as much as possible across the fleet, only a few site-specific details may be different. OS updates must not be applied to every system individually; instead, the VM image with the latest OS updates shall be applied fleet-wide to avoid incomplete, failing and non-reproducible updates.
onway routers provide the building blocks for a storage strategy to address these challenges:
+-------------------------------------------------------+
| Guest VM |
| +------------------+ +------------+-+-------------+ |
| | Operating System | | Data | | Cache | |
| +------------------+ +------------+ +-------------+ |
| | +----------------------------+ |
| | | (Logical Volume) | |
| | +----------------------------+ |
+---------|---------------------|---------------|-------+
| | |
v v v
+------------------+ +------------+ +------------+
| Overlay | | Plain Disk | | Plain Disk |
+------------------+ +------------+ +------------+
| | | |
v | | |
/ +------------+ | | |
| | Base image | | | |
Updated | +------------+ | | |
by | | | | |
router | v | | |
| +--------+ | | |
| | Bundle | | | |
\ +--------+ | | |
| | | |
v v v v
+-------------------------------------------------------+
| Host SSDs |
+-------------------------------------------------------+The overlay layer may store site-specific state for a VM. That overlay state is not permanent, though. If the base image is updated, the overlay is not valid anymore, as the overlay works on a block device and not on a file level. When detecting base image changes, the overlay is dropped. This implies that an automated approach is required to re-create site-specific changes to the base image whenever:
If a VM requires site-specific modifications, the base image shall contain logic to create that from a plain base image. This can be implemented in scripts, optionally interacting with the router via its REST API, to create such state. Such a mechanism can use proprietary tooling in the VM, or build upon industry standards such as cloud-init 2. When using plain disks, this tooling further requires partitioning, optionally logical volume setup and file system creation. Disks that are newly created by the host read all zeroes.
When running virtual machines on a large fleet, it is usually advised to use a common base image and site-specific overlays. To bootstrap site-specific state, the software in a virtual machine can read per-instance parameters configured from the host system.
The interface for such site-specific parameters is implemented in the form of DMI/SMBIOS variables. In a Linux guest VM, the configured variables can be read from sysfs under /sys/class/dmi/id or using the dmidecode utility. The naming for DMI variables differs between the sysfs file names and the dmidecode --string filtering, but are mapped as follows:
dmidecode --string= variable |
/sys/class/dmi/id filename |
|---|---|
bios-vendor |
bios_vendor |
bios-release-date |
bios_date |
bios-version |
bios_version |
system-manufacturer |
sys_vendor |
system-product-name |
product_name |
system-version |
product_version |
system-serial-number |
product_serial |
system-uuid |
product_uuid |
baseboard-manufacturer |
board_vendor |
baseboard-product-name |
board_name |
baseboard-version |
board_version |
baseboard-serial-number |
board_serial |
baseboard-asset-tag |
board_asset_tag |
chassis-manufacturer |
chassis_vendor |
chassis-version |
chassis_version |
chassis-serial-number |
chassis_serial |
chassis-asset-tag |
chassis_asset_tag |
oem (using --oem-string=) |
unavailable |
On the host system, DMI variables are configured using the variable names from the dmidecode utility. The cloud-init NoCloud datasource can make use of DMI/SMBIOS variables for site-specific VM instance initialization, for example it can seed from HTTP by setting the system-serial-number DMI variable to ds=nocloud-net;s=http://10.0.0.1. Refer to the NoCloud documentation 5 for details.
The hypervisor on onway routers is not intended to create virtual machine images from scratch. Providing installation media, such as ISO images, is not supported. Instead, virtual machine base images shall be created on standard Linux development hosts using the widely available QEMU. All instructions below are run on such a development host.
The RAW disk image is the base of any virtual machine running on routers and shall contain the guest operating system. The size of the RAW disk image is important, because overlay images are restricted to that size. Note that RAW disk images are stored sparse in bundles, that is, unused disk blocks do not take up space.
To create a RAW disk image with a size of 8GB (for Windows 15GB), use:
truncate -s 8G disk.rawTo install a virtual machine to the disk image from ISO installation media, use the following QEMU invocation:
export FW=/usr/share/qemu/OVMF.fd
export DISK=disk.raw
export ISO=fedora.iso
sudo qemu-system-x86_64 \
-machine q35,accel=kvm -nodefaults \
-cpu host -m 4096M -smp cores=4 \
-bios $FW \
-chardev pty,id=ser0 -serial chardev:ser0 \
-device VGA,addr=0b,edid=on,xres=1280,yres=1024 \
-usb -device usb-tablet \
-device virtio-blk-pci,addr=0a,drive=root,disable-legacy=on \
-blockdev driver=raw,node-name=root,file.driver=file,file.filename=$DISK \
-netdev user,id=net1 \
-device virtio-net-pci,addr=01,netdev=net1 \
-cdrom $ISO -boot dWhile other settings are possible, these create emulated hardware very similar to the one used on the router. Once the installation has completed, the last line can be omitted to boot into the installed guest and make further modifications.
Additional NICs can be added by adding additional -netdev/-device options. Use increasing id= names and addr= assignments to ensure identical interface names as on the router. user networking is usually the simplest to start with, but other variants provided by QEMU can be used.
Plain extra disks can be added with the following lines:
export DISK1=extra.img
export ID=1
# ...
-device virtio-blk-pci,addr=1$ID,drive=disk$ID,disable-legacy=on \
-blockdev driver=raw,node-name=disk$ID,file.driver=file,file.filename=$DISK1Use increasing IDs for every added disk.
Windows does not include drivers for VirtIO virtual hardware. For Windows installation, extra drivers are required from an ISO image from the virtio-win github site: 4. In the QEMU commands above, the contents of the ISO can be provided by appending the line below after the -cdrom $ISO -boot d:
-drive file=virtio-win-*.iso,index=3,media=cdromDuring Windows installation, Load Driver when prompted for an installation target disk, Browse to E:\amd64\<windows version>, then select the VirtIO SCSI controller driver for installation.
Once the virtual machine has booted into the Windows installation, open the device manager and locate the Ethernet Controller that misses a driver. Right-click and select Update driver, Browse my computer for drivers, Browse to E:\NetKVM\<windows version> and install the driver.
While changes to the RAW guest base image can be made by booting into the image, for some tasks this may not be necessary. Instead, the RAW disk image can be mounted directly with a Linux development machine.
export MNTPOINT=/mnt
export LOOPDEV=$(losetup -f)
sudo losetup $LOOPDEV $DISK -P
sudo fdisk -l $LOOPDEVIdentify the partition to mount. If the partition contains the filesystem directly, mount it:
export PART=3
sudo mount ${LOOPDEV}p${PART} $MNTPOINTIf it contains an LVM partition, activate the volume and then mount it:
export PART=3
sudo vgscan
export VG=fedora
sudo vgchange -ay $VG
sudo lvs
export LV=root
sudo mount /dev/$VG/$LV $MNTPOINTNow changes can be made to the filesystem in the base image. When done,
sudo umount $MNTPOINTand with LVM volumes, additionally:
sudo vgchange -an $VGNow the loop device can be disabled using:
sudo losetup -d $LOOPDEVMany Linux distributions have no console on the serial port by default, but can be configured to allow login and administration via the serial port. The exact instructions depend on the distribution and the used version. When using grub2, it is usually recommended to do the setup by booting into the QEMU image, as it may involve multiple mounted partitions and tools working reliably only within the real emulated hardware.
Common for all Linux systems is that a kernel parameter is required for the serial console. Multiple consoles can be configured, and the last console is used as the primary Linux console. Using
console=ttyS0,115200 console=tty0keeps the primary console on tty0 for logins via VNC, but enables the console on the emulated ISA serial port as well.
On newer systems from the Red Hat family, the grubby tool can be used to set up the console:
sudo grubby --update-kernel=ALL --args="console=ttyS0,115200 console=tty0"On Ubuntu based systems, kernel command line options can be specified in /etc/default/grub:
# ...
GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0,115200 console=tty0"
# ...The config then must be applied using the update-grub command.
RAW disk images are usually deployed from filesystem bundles. The router looks for virtual machine base images in a filesystem bundle under the path vms/base.
export IMAGE_NAME=vmimg
export BUNDLE_NAME=bundle
mkdir -p squashroot/vms/base
mv $DISK squashroot/vms/base/$IMAGE_NAME
mksquashfs squashroot $BUNDLE_NAME -comp lzo -all-rootThis produces a bundle containing the base image previously created. lzo compression is recommended, as it decompresses faster and provides improved read performance compared to the default.