onway router container runtime
 

onway router container runtime

All onway router models provide a container runtime to execute third party applications in isolated container environments. It works similar to Docker or other container runtimes, but is a custom solution specifically designed for the needs of embedded platforms and large host fleets.

Container images

For an embedded container runtime, the requirements for container images are slightly different to traditional (layered) container images used in other container solutions. The layering is less important, as a container host (the router) is not expected to run many containers sharing base layers.

Instead, the container image shall:

To meet these requirements, images based on SquashFS are chosen. These read-only images provide compression with a wide range of algorithms, can be mounted on the container host and nonetheless can be relatively good delta-patched using rsync or other algorithms.

SquashFS images containing files are called bundles in the onway router ecosystem. A bundle can contain full root filesystems (for containers), but also other files for different purposes (such as media server content). Bundles are synchronized to physical storage on the router, and when mounted provide the foundation for container images.

Alternatively, container images may be provided directly from (removable) storage devices. As there is no mechanism to deploy files to such devices, this is mostly useful for testing and development purposes.

Container bundles

A bundle as discussed above refers to a SquashFS image containing arbitrary files. The term is not to confuse with the Open Container Initiative filesystem bundle container format; In fact a SquashFS bundle may contain multiple container images in the OCI filesystem bundle format, which each contains a directory tree.

A single SquashFS bundle (or any other mounted media) may contain multiple container images (and potentially other files). To do so, in a (bundle) mount, the top-level directory containers contains container images. An image is named according the subdirectory in the containers directory. Under that named subdirectory, an OCI compatible Configuration in a file named config.json must be provided. The configuration is used by the container runtime to maintain the lifecycle of a container.

Only a subset of the OCI Configuration format is supported by the runtime; resource limitations for example are not provided by the container image, but the container runtime configuration. The options supported are:

As the containing config.json and other files associated to a container image are mounted on an dynamic path, specifying the root path property with an absolute path makes no sense. Instead, the path must be relative to the directory containing the config.json file. All other paths specified use absolute paths within the container, which are relative to the root path.

In the specified root path, the directories /sys, /proc and /dev must exist, as the runtime mounts appropriate filesystems at these locations.

Container instances

Containers that are instantiated use a specific container image. Often a container image is instantiated once, but multiple instances on the same image are possible. An instance provides the runtime information of a container.

User namespaces

Container instances shall run unprivileged with a distinct set of all Linux namespaces, including user namespaces. This brings good isolation from the host system, while still giving the container root permissions in all its namespaces.

To differentiate host and individual container UIDs and GIDs, these are mapped when crossing user namespace boundaries. Each container gets a mapping of an UID/GID range in the container from 0-65535 to a distinct range of the same size, dynamically allocated starting at 65536 on the hosts initial user namespace view.

Filesystem layering

The container image is shared between multiple instances, and SquashFS images are read-only. Therefore a container can not run directly on a SquashFS mount. Instead, additional layers are required:

+--------------+    +--------------+    +--------------+
| Container 1  |    | Container 2  |    | Container 3  |
+--------------+    +--------------+    +--------------+
        |                   |    |              |    |
        |                   |    v              |    v
        |                   |   +-----------+   |   +-----------+
        |                   | +-----------+ |   | +-----------+ |
        |                   | | ID-mapped |-+   | | ID-mapped |-+
        |                   | |   share   |     | |   share   |
        |                   | +-----------+     | +-----------+
        |                   |         |         |          |
+--------------+    +--------------+  |  +--------------+  |
|  overlayfs   |    |  overlayfs   |  |  |  overlayfs   |  |
+-----------+--+    +-----------+--+  |  +-----------+--+  |
|   ext4    | |     |   ext4    | |   |  |   ext4    | |   |
+-----------+ |     +-----------+ |   |  +-----------+ |   |
| zram disk | |     | zram disk | |   |  | zram disk | |   |
+-----------+ |     +-----------+ |   |  +-----------+ |   |
              v                   v   |                v   |
   +-----------+       +-----------+  |     +-----------+  |
   | ID-mapped |       | ID-mapped |  |     | ID-mapped |  |
   +-----------+       +-----------+  |     +-----------+  |
         |                   |        |           |        |
         v                   v        |           |        |
+----------------------------------+  |           |        |
|          SquashFS mount          |  |           |        |
+----------------------------------+  |           |        |
                 |                    |           |        |
+----------------------------------+  |           |        |
|       SquashFS file (bundle)     |  v           v        v
+----------------------------------+----------------------------+
|                  storage filesystem                           |
+---------------------------------------------------------------+
|                     physical disk                             |
+---------------------------------------------------------------+

On top of the SquashFS mount (or a container image directly stored on a storage device), an overlayfs is stacked using the lower layer read-only. The overlayfs upper layer is on a traditional ext4 Linux filesystem, dynamically created in a zram device.

Any changes done on the overlay filesystem are written to the zram contained filesystem. zram automatically compresses these changes and keeps them in system memory. It also provides simple filesystem limits based on uncompressed disk size or on compressed memory use.

Between overlayfs and its lower layer, an additional ID-mapped mount is used to reverse UID/GID translation done by the container user namespace UID/GID mapping. This allows the container to see root-owned files in the SquashFS container image as root-owned within the container, as the ID-mapped mount translates the UID to the unprivileged user, and the user namespace back to root. This gives the container full access to the container image root filesystem for multiple containers, even if each container has completely distinct UID/GID ranges. Changes to the filesystem, however, are redirected to the per-container private overlayfs upper layer, stored volatile in memory.

For persistent storage, the container may have additional bind-mounts mounted into the overlayfs directory tree at specific mountpoints. Changes to such directories are redirected to physical storage. If multiple containers have the same host-side directory mounted, they additionally may share files over this directory. Shares are mounted ID-mapped, so that multiple containers can use the share with identical UID/GIDs, even if the container users are actually separated.

Statelessness

Given the chosen filesystem layerings and the zram based filesystem overlay, any changes done by the container are lost upon container restarts or host system power losses. This is fully intended for embedded system containers, as:

For some cases, persistent storage may be required by a container. Such storage may be added for specific mountpoints, but containers must ensure to limit disk usage and write cycles to not negatively affect other components or the router hardware.

Networking

Docker and other container runtimes support different mechanisms to connect containers to the host, other containers or systems on the hosts physical network. By default, Docker uses bridged interfaces and port forwarding.

Containers running on the onway router container runtime can be integrated into the host network with a similar set of mechanisms. However, the network integration is not done by the container runtime itself. Instead, the existing networking mechanisms of the onway router system are used. The container runtime connects containers to the host system using one or more veth network interface pairs:

System time

As containers share the kernel with the host system, the system time in the container is the same as on the host. On some embedded platforms, the system time may be incorrect after boot. Containers are started nonetheless, so they can perform tasks and provide services even if the host system currently has no online connectivity to synchronize its clock.

If a container requires a correct system time to operate properly, it is recommended to wait for a valid time in the container process or in a prestart hook. Waiting for a date that is at least the date of the container image creation is the recommended approach, for example:

/bin/sh -c "while [ $(date +%Y) -lt 2024 ]; do sleep 1; done"

Alternatively, the container may query the system time from the onway router REST API, which provides a time only once it has been synchronized.

Init systems

Traditional (Docker) container images run a single service in the container through the container main process. This is usually the recommended approach, as it reduces complexity and the footprint of a container.

Nonetheless does the onway container runtime support running images using a full system approach, including an in-container init system. Recent releases of the systemd init system can be spawned as container main process, where it manages services and processes. The runtime mostly provides the interface for init systems as outlined by systemd. There are some limitations for running full OS distributions in containers, though:

Container development

To build container images, industry standard tools can be used and containers can be based on images from public or private image registries. The following description guides through container creation based on an example.

Building images

The container runtime expects a OCI bundle to execute. The bundle contains a config.json and a root filesystem. The root filesystem can be created using arbitrary container build tools such as docker, podman or buildah; important is decent support for building images for the target architecture, usually ARM.

In the below example the buildah tool is used to create a container on Ubuntu 22.04. While buildah can create images as non-root user, these instructions run all commands under root to create the root filesystem with proper UID/GIDs set.

First, buildah is installed along with QEMU for cross-architecture container builds:

apt install buildah qemu-user-static

In the next steps, a container for the ARMv7 architecture is created based on Alpine Linux, serving a web page over a HTTP server:

buildah from --name hello --platform linux/arm alpine
buildah run hello apk update
buildah run hello apk add apache2
buildah run hello ln -sf /proc/self/fd/1 /var/log/apache2/access.log
buildah run hello ln -sf /proc/self/fd/2 /var/log/apache2/error.log
echo "Hello World!" > hello.txt
buildah copy hello hello.txt /var/www/localhost/htdocs/

Note that buildah alternatively supports a build-using-dockerfile command to use Dockerfiles instead of the manual run commands.

The root filesystem is prepared and can be copied to a local directory:

mkdir -p bundle/containers/hello
cp -a $(buildah mount hello) bundle/containers/hello/rootfs
buildah umount hello

A simple config.json is added configuring the container:

cat > bundle/containers/hello/config.json <<EOF
{
  "ociVersion": "1.0.1",
  "root": {
    "path": "rootfs"
  },
  "process": {
    "cwd": "/",
    "args": [ "/usr/sbin/httpd", "-DFOREGROUND" ]
  },
  "hooks": {
    "prestart": [
      {
        "args": [ "/sbin/ip", "link", "set", "eth0", "up" ]
      },
      {
        "args": [ "/sbin/ip", "addr", "add",
                  "10.7.6.5/24", "dev", "eth0" ]
      }
    ]
  }
}
EOF

While the configuration of host side networking is done by the onway router tooling, the in-container configuration of links must be done by the container itself.

Testing images

To test an image, the onway-container-runtime testing tool can be used. It implements the container runtime in a standalone utility, and requires Ubuntu 20.04 or later to run.

To start the container from the previous section, the following command can be used:

onway-container-runtime -c hello -p bundle/containers/hello \
    -s 32 -e veth0

Once it is running, on a different terminal the host-side interface can be configured and the HTTP page fetched:

ip addr add 10.7.6.1/24 dev veth0
wget -q -O - http://10.7.6.5/hello.txt

To inspect and debug a running container, onway-container-runtime allows opening a shell or running a command inside the container. When the runtime and a container is running, from a different terminal, invoke:

onway-container-runtime --login <pid> -- /bin/bash

<pid> specifies the container main process PID, printed by the container runtime during container startup. By default, /bin/sh -l is invoked, but a different shell (bash in the example) or command can be invoked.

Bundling images

While plain OCI images can be directly deployed by placing the containers directory to removable media, usually it is more convenient to deploy SquashFS bundles containing that (and potentially other) folders.

Manual build

The bundle should be created as root, so the file permissions are kept intact.

mksquashfs bundle hello

Via container registry

Instead of creating an OCI runtime bundle and packing it into a SquashFS bundle manually, the onway tooling can automatically fetch container images from OCI distribution compatible registries, such as Docker Hub, GitHub Container Registry or others. The tooling automatically:

The container image configuration must satisfy the following requirements:

OCI runtime configurations support hooks, which can be quite useful to set up the container before spawning the container main process. Unfortunately, there is no official mechanism to define such hooks in the container configuration stored in the registry. Therefore, OCI runtime hooks can be defined via container config labels using a proprietary format. Hooks are defined in labels using a ch.onway.hooks.<type>.<name>.* prefix, where <type> is either prestart or poststart, and <name> is a unique identifier for the hook. The order of hook execution is defined by the natural order of <name>.

For each hook, multiple labels can be defined:

The args option for each hook takes a space separated list of execve() style arguments, where the first item must point to an executable using an absolute path. For additional arguments, single or double quotes can be used to have arguments with spaces. Escaping quotes is currently not supported.

Every <var> under an env label is expected to be a valid environment variable name, and the value is the environment variable value, combined using =. The timeout option is an integer value in seconds before hook execution is aborted with an error.

In a Dockerfile an example of a hook specification with LABELs is:

LABEL ch.onway.hooks.prestart.0.args="/bin/sh -c 'echo foo $FOO.'"
LABEL ch.onway.hooks.prestart.0.env.FOO=bar
LABEL ch.onway.hooks.prestart.0.timeout=1

These labels will be included in the container configuration on the registry, and are converted to OCI runtime hooks in the onway tooling.