All onway router models provide a container runtime to execute third party applications in isolated container environments. It works similar to Docker or other container runtimes, but is a custom solution specifically designed for the needs of embedded platforms and large host fleets.
For an embedded container runtime, the requirements for container images are slightly different to traditional (layered) container images used in other container solutions. The layering is less important, as a container host (the router) is not expected to run many containers sharing base layers.
Instead, the container image shall:
To meet these requirements, images based on squashfs are chosen. These read-only images provide compression with a wide range of algorithms, can be mounted on the container host and nonetheless can be relatively good delta-patched using rsync or other algorithms.
squashfs images containing files are called bundles in the onway router ecosystem. A bundle can contain full root filesystems (for containers), but also other files for different purposes (such as media server content). Bundles are synchronized to physical storage on the router, and when mounted provide the foundation for container images.
Alternatively, container images may be provided directly from (removable) storage devices. As there is no mechanism to deploy files to such devices, this is mostly useful for testing and development purposes.
A bundle as discussed above refers to a squashfs image containing arbitrary files. The term is not to confuse with the Open Container Initiative filesystem bundle container format; In fact a squashfs bundle may contain multiple container images in the OCI filesystem bundle format, which each contains a directory tree.
A single squashfs bundle (or any other mounted media) may contain multiple container images (and potentially other files). To do so, in a (bundle) mount, the top-level directory containers contains container images. An image is named according the subdirectory in the containers directory. Under that named subdirectory, an OCI compatible Configuration in a file named config.json must be provided. The configuration is used by the container runtime to maintain the lifecycle of a container.
Only a subset of the OCI Configuration format is supported by the runtime; resource limitations for example are not provided by the container image, but the container runtime configuration. The options supported are:
As the containing config.json and other files associated to a container image are mounted on an dynamic path, specifying the root path property with an absolute path makes no sense. Instead, the path must be relative to the directory containing the config.json file. All other paths specified use absolute paths within the container, which are relative to the root path.
In the specified root path, the directories /sys, /proc and /dev must exist, as the runtime mounts appropriate filesystems at these locations.
Containers that are instantiated use a specific container image. Often a container image is instantiated once, but multiple instances on the same image are possible. An instance provides the runtime information of a container.
Container instances shall run unprivileged with a distinct set of all Linux namespaces, including user namespaces. This brings good isolation from the host system, while still giving the container root permissions in all its namespaces.
To differentiate host and individual container UIDs and GIDs, these are mapped when crossing user namespace boundaries. Each container gets a mapping of an UID/GID range in the container from 0-65535 to a distinct range of the same size, dynamically allocated starting at 65536 on the hosts initial user namespace view.
The container image is shared between multiple instances, and squashfs images are read-only. Therefore a container can not run directly on a squashfs mount. Instead, additional layers are required:
+--------------+ +--------------+ +--------------+
| Container 1 | | Container 2 | | Container 3 |
+--------------+ +--------------+ +--------------+
| | | | |
| | v | v
| | +-----------+ | +-----------+
| | +-----------+ | | +-----------+ |
| | | ID-mapped |-+ | | ID-mapped |-+
| | | share | | | share |
| | +-----------+ | +-----------+
| | | | |
+--------------+ +--------------+ | +--------------+ |
| overlayfs | | overlayfs | | | overlayfs | |
+-----------+--+ +-----------+--+ | +-----------+--+ |
| ext4 | | | ext4 | | | | ext4 | | |
+-----------+ | +-----------+ | | +-----------+ | |
| zram disk | | | zram disk | | | | zram disk | | |
+-----------+ | +-----------+ | | +-----------+ | |
v v | v |
+-----------+ +-----------+ | +-----------+ |
| ID-mapped | | ID-mapped | | | ID-mapped | |
+-----------+ +-----------+ | +-----------+ |
| | | | |
v v | | |
+----------------------------------+ | | |
| squashfs mount | | | |
+----------------------------------+ | | |
| | | |
+----------------------------------+ | | |
| squashfs file (bundle) | v v v
+----------------------------------+----------------------------+
| storage filesystem |
+---------------------------------------------------------------+
| physical disk |
+---------------------------------------------------------------+On top of the squashfs mount (or a container image directly stored on a storage device), an overlayfs is stacked using the lower layer read-only. The overlayfs upper layer is on a traditional ext4 Linux filesystem, dynamically created in a zram device.
Any changes done on the overlay filesystem are written to the zram contained filesystem. zram automatically compresses these changes and keeps them in system memory. It also provides simple filesystem limits based on uncompressed disk size or on compressed memory use.
Between overlayfs and its lower layer, an additional ID-mapped mount is used to reverse UID/GID translation done by the container user namespace UID/GID mapping. This allows the container to see root-owned files in the squashfs container image as root-owned within the container, as the ID-mapped mount translates the UID to the unprivileged user, and the user namespace back to root. This gives the container full access to the container image root filesystem for multiple containers, even if each container has completely distinct UID/GID ranges. Changes to the filesystem, however, are redirected to the per-container private overlayfs upper layer, stored volatile in memory.
For persistent storage, the container may have additional bind-mounts mounted into the overlayfs directory tree at specific mountpoints. Changes to such directories are redirected to physical storage. If multiple containers have the same host-side directory mounted, they additionally may share files over this directory. Shares are mounted ID-mapped, so that multiple containers can use the share with identical UID/GIDs, even if the container users are actually separated.
Given the chosen filesystem layerings and the zram based filesystem overlay, any changes done by the container are lost upon container restarts or host system power losses. This is fully intended for embedded system containers, as:
For some cases, persistent storage may be required by a container. Such storage may be added for specific mountpoints, but containers must ensure to limit disk usage and write cycles to not negatively affect other components or the router hardware.
To build container images, industry standard tools can be used and containers can be based on images from public or private image registries. The following description guides through container creation based on an example.
The container runtime expects a OCI bundle to execute. The bundle contains a config.json and a root filesystem. The root filesystem can be created using arbitrary container build tools such as docker, podman or buildah; important is decent support for building images for the target architecture, usually ARM.
In the below example the buildah tool is used to create a container on Ubuntu 22.04. While buildah can create images as non-root user, these instructions run all commands under root to create the root filesystem with proper UID/GIDs set.
First, buildah is installed along with QEMU for cross-architecture container builds:
apt install buildah qemu-user-staticIn the next steps, a container for the ARMv7 architecture is created based on Alpine Linux, serving a web page over a HTTP server:
buildah from --name hello --platform linux/arm alpine
buildah run hello apk update
buildah run hello apk add apache2
buildah run hello ln -sf /proc/self/fd/1 /var/log/apache2/access.log
buildah run hello ln -sf /proc/self/fd/2 /var/log/apache2/error.log
echo "Hello World!" > hello.txt
buildah copy hello hello.txt /var/www/localhost/htdocs/Note that buildah alternatively supports a build-using-dockerfile command to use Dockerfiles instead of the manual run commands.
The root filesystem is prepared and can be copied to a local directory:
mkdir -p squashroot/containers/hello
cp -a $(buildah mount hello) squashroot/containers/hello/rootfs
buildah umount helloA simple config.json is added configuring the container:
cat > squashroot/containers/hello/config.json <<EOF
{
"ociVersion": "1.0.1",
"root": {
"path": "rootfs"
},
"process": {
"cwd": "/",
"args": [ "/usr/sbin/httpd", "-DFOREGROUND" ]
},
"hooks": {
"prestart": [
{
"args": [ "/sbin/ip", "link", "set", "eth0", "up" ]
},
{
"args": [ "/sbin/ip", "addr", "add",
"10.7.6.5/24", "dev", "eth0" ]
}
]
}
}
EOFWhile the configuration of host side networking is done by the onway router tooling, the in-container configuration of links must be done by the container itself.
To test an image, the onway-container-runtime testing tool can be used. It implements the container runtime in a standalone utility, and requires Ubuntu 20.04 or later to run.
To start the container from the previous section, the following command can be used:
onway-container-runtime -c hello -p squashroot/containers/hello \
-s 32 -e veth0Once it is running, on a different terminal the host-side interface can be configured and the HTTP page fetched:
ip addr add 10.7.6.1/24 dev veth0
wget -q -O - http://10.7.6.5/hello.txtWhile plain OCI images can be directly deployed by placing the containers directory to removable media, usually it is more convenient to deploy SquashFS bundles containing that (and potentially other) folders.
mksquashfs squashroot helloThe resulting bundle can be efficiently synced by the onway infrastructure to a large router fleet and configured to run it accordingly.