The onway mobile router provides a container runtime to execute third party applications in isolated container environments. It works similar to Docker or other container runtimes, but is a custom solution specifically designed for the needs of embedded platforms and large host fleets.
For an embedded container runtime, the requirements for container images are slightly different to traditional (layered) container images used in other container solutions. The layering is less important, as a container host (mobile router) is not expected to run many containers sharing base layers.
Instead, the container image shall:
To meet these requirements, images based on squashfs are chosen. These read-only images provide compression with a wide range of algorithms, can be mounted on the container host and nonetheless can be relatively good delta-patched using rsync or other algorithms.
squashfs images containing files are called bundles in the onway router ecosystem. A bundle can contain full root filesystems (for containers), but also other files for different purposes (such as media server content). Bundles are synchronized to physical storage on the router, and when mounted provide the foundation for container images.
A bundle as discussed above refers to a squashfs image containing arbitrary files. The term is not to confuse with the Open Container Initiative filesystem bundle container format; In fact a squashfs bundle may contain multiple container images in the OCI filesystem bundle format, which each contains a directory tree.
A single squashfs bundle (or any other mounted media) may contain multiple container images (and potentially other files). To do so, in a (bundle) mount, the top-level directory containers contains container images. An image is named according the subdirectory in the containers directory. Under that named subdirectory, an OCI compatible Configuration in a file named config.json must be provided. The configuration is used by the container runtime to maintain the lifecycle of a container.
Only a subset of the OCI Configuration format is supported by the runtime; resource limitations for example are not provided by the container image, but the container runtime configuration. The options supported are:
As the containing config.json and other files associated to a container image are mounted on an dynamic path, specifying the root path property with an absolute path makes no sense. Instead, the path must be relative to the directory containing the config.json file. All other paths specified use absolute paths within the container, which are relative to the root path.
In the specified root path, the directories /sys, /proc and /dev must exist, as the runtime mounts appropriate filesystems at these locations.
Containers that are instantiated use a specific container image. Often a container image is instantiated once, but multiple instances on the same image are possible. An instance provides the runtime information of a container.
Container instances shall run unprivileged with a distinct set of all Linux namespaces, including user namespaces. This brings good isolation from the host system, while still giving the container root permissions in all its namespaces.
To differentiate host and individual container UIDs and GIDs, these are mapped when crossing user namespace boundaries. Each container gets a mapping of an UID/GID range in the container from 0-65535 to a distinct range of the same size, dynamically allocated starting at 65536 on the hosts initial user namespace view.
The container image is shared between multiple instances, and squashfs images are read-only. Therefore a container can not run directly on a squashfs mount. Instead, additional layers are required:
+--------------+ +--------------+ +--------------+ \
| Container 1 | | Container 2 | | Container 3 | | container
+--------------+ +--------------+ +--------------+ | mount
| shiftfs view | | shiftfs view | | shiftfs view | | namespaces
+--------------+ +--------------+ +--------------+ /
| | |
+--------------+ +--------------+ +--------------+ \
| shiftfs mark | | shiftfs mark | | shiftfs mark | |
+--------------+ +--------------+ +--------------+ |
| | | |
+--------------+ +--------------+ +--------------+ |
| unionfs | | unionfs | | unionfs | |
+-----------+--+ +-----------+--+ +-----------+--+ |
| ext ovl | | | ext ovl | | | ext ovl | | |
+-----------+ | +-----------+ | +-----------+ | |
| zram disk | | | zram disk | | | zram disk | | | host mount
+-----------+ | +-----------+ | +-----------+ | | namespace
v v v |
+----------------------------------+ +--------------+ |
| squashfs mount 1 | | sqfs mnt 2 | |
+----------------------------------+ +--------------+ |
| | |
+----------------------------------+ +--------------+ |
| squashfs file (bundle) | | sqfs file | |
+----------------------------------+----+--------------+ |
| storage filesystem | |
+------------------------------------------------------+ /
| physical disk |
+------------------------------------------------------+On top of the squashfs mount, a unionfs is added using the squashfs as read-only lower layer. The upper layer is on a traditional Linux filesystem such as Ext4 created in-memory on a zram device.
Any changes done on the unified filesystem are written to the zram contained filesystem. zram automatically compresses these changes and keeps them in system memory. It also provides simple filesystem limits based on uncompressed disk size but also on compressed memory use.
On top of the unionfs, an additional filesystem layer using shiftfs is added. Because the files in the squashfs mount (and the union overlay) are owned by root and other system users, containers using user namespaces are unable to access these files with sufficient permissions. Marking the unionfs mount with shiftfs on the host allows the container to mount a shiftfs view with its user namespace UID/GID mapping reversed. So the container, while unprivileged outside of its namespaces, has full permissions on the unionfs mount and the zram based memory overlay.
The following snippets demonstrate the use of the filesystem layering:
# per image
sudo mkdir squashdir
sudo mount -t squashfs image.squashfs squashdir
# per guest: zram
sudo modprobe zram
sudo sh -c "echo 32M > /sys/block/zram0/disksize"
sudo mkfs.ext2 /dev/zram0
sudo mkdir zramdir
sudo mount /dev/zram0 zramdir
# per guest: union
sudo mkdir uniondir
sudo mkdir zramdir/work
sudo mkdir zramdir/upper
sudo mount -t overlay \
-o lowerdir=squashdir,upperdir=zramdir/upper,workdir=zramdir/work \
overlay uniondir
# per guest: shiftfs and chroot
sudo mkdir shiftmark
sudo mount -t shiftfs -o mark uniondir shiftmark
sudo mkdir shiftview
unshare -muinpUrf sh -c \
"mount -t shiftfs shiftmark shiftview && chroot shiftview"Given the chosen filesystem layerings and the zram based filesystem overlay, any changes done by the container are lost upon container restarts or host system power losses. This is fully intended for embedded system containers, as:
Container setups must consider this fact. While there are ways to provide additional persistent storage to containers through bind mounts, currently their use is not intended.
To build container images, industry standard tools can be used and containers can be based on images from public or private image registries. The following description guides through container creation based on an example.
The container runtime expects a OCI bundle to execute. The bundle contains a config.json and a root filesystem. The root filesystem can be created using arbitrary container build tools such as docker, podman or buildah; important is decent support for building images for the target architecture, usually ARM.
In the below example the buildah tool is used to create a container on Ubuntu 22.04. While buildah can create images as non-root user, these instructions run all commands under root to create the root filesystem with proper UID/GIDs set.
First, buildah is installed along with QEMU for cross-architecture container builds:
apt install buildah qemu-user-staticIn the next steps, a container for the ARMv7 architecture is created based on Alpine Linux, serving a web page over a HTTP server:
buildah from --name hello --platform linux/arm alpine
buildah run hello apk update
buildah run hello apk add apache2
buildah run hello ln -sf /proc/self/fd/1 /var/log/apache2/access.log
buildah run hello ln -sf /proc/self/fd/2 /var/log/apache2/error.log
echo "Hello World!" > hello.txt
buildah copy hello hello.txt /var/www/localhost/htdocs/Note that buildah alternatively supports a build-using-dockerfile command to use Dockerfiles instead of the manual run commands.
The root filesystem is prepared and can be copied to a local directory:
mkdir -p squashroot/containers/hello
cp -a $(buildah mount hello) squashroot/containers/hello/rootfs
buildah umount helloA simple config.json is added configuring the container:
cat > squashroot/containers/hello/config.json <<EOF
{
"ociVersion": "1.0.1",
"root": {
"path": "rootfs"
},
"process": {
"cwd": "/",
"args": [ "/usr/sbin/httpd", "-DFOREGROUND" ]
},
"hooks": {
"prestart": [
{
"args": [ "/sbin/ip", "link", "set", "eth0", "up" ]
},
{
"args": [ "/sbin/ip", "addr", "add",
"10.7.6.5/24", "dev", "eth0" ]
}
]
}
}
EOFWhile the configuration of host side networking is done by the onway mobile router tooling, the in-container configuration of links must be done by the container itself.
To test an image, the onway-container-runtime testing tool can be used. It implements the container runtime in a standalone utility, and requires Ubuntu 20.04 or later to run.
To start the container from the previous section, the following command can be used:
onway-container-runtime -c hello -p squashroot/containers/hello \
-s 32 -e veth0Once it is running, on a different terminal the host-side interface can be configured and the HTTP page fetched:
ip addr add 10.7.6.1/24 dev veth0
wget -q -O - http://10.7.6.5/hello.txtWhile plain OCI images can be directly deployed by placing the containers directory to removable media, usually it is more convenient to deploy SquashFS bundles containing that (and potentially other) folders.
mksquashfs squashroot helloThe resulting bundle can be efficiently synced by the onway infrastructure to a large router fleet and configured to run it accordingly.