onway uplink management
 

onway uplink management

Introduction

For its product portfolio of onway routers, onway ag has developed an advanced uplink management mechanism over the last years. The solution uses some novel ideas for dynamic uplink management in an industry-leading and field-proven implementation.

Intelligent uplink management is a key component of a mobile router system, as these devices require a reliable uplink over a wide variety of transport networks to fulfill their service. The onway router hardware can be equipped with multiple mobile broadband modems using LTE or UMTS/GSM connectivity. In addition, uplinks can be realized using WiFi networks where available, or using fixed wire networks such as Ethernet.

The availability and quality of such uplinks is affected by many factors, and mobile operator network coverage is one of them. Using multiple competing mobile operators can not only improve availability, uplink quality and capacity, but also opens new opportunities in cost optimizations.

Terminology

Requirements

Uplink availability and quality can change within seconds or below, especially when using mobile routers on vehicles moving at fast speed. So a competitive uplink management solution must be capable of migrating traffic flows between uplinks, in a way that it is transparent to the application.

If multiple uplinks are available, one obviously wants to bundle the capacity of the available uplinks. An intelligent load-balancing mechanism shall distribute the load as uplinks permit, combining individual uplinks to a virtual uplink with the summed up capacity.

Also, an uplink management solution shall allow combining the uplinks of multiple routers to a virtual uplink. This provides redundancy at router level, but also can scale bundling to dozens of potentially physically distributed uplinks.

Not all uplinks are trustworthy, and providing integrity and privacy services to applications at some layer is a key requirement for any today’s mobile router solution. Also, securely separating traffic classes, both locally on the mobile router, but also in a targeted data center network is a desirable property well known from enterprise networking.

Last but not least, providing Quality of Service mechanisms has become a major requirement, as uplinks are often shared between multiple critical and non-critical applications. Also, uplink networks involving radio transmissions often suffer from highly over-dimensioned on-path buffers, yielding to buffer bloat that makes interactive applications on highly loaded uplinks unusable. A QoS solution requires mitigation techniques for these buffer-bloat issues.

In an ideal solution, multiple uplinks are seamlessly combined to a virtual uplink for capacity and availability reasons, both on a per-router level, but also distributed over multiple routers. On top of that, the combined uplink shall allow network virtualization by separating individual applications in a way that they can not negatively affect each other.

System description

This chapters will discuss the design and rationale of the onway uplink management solution. It shall give a broad overview about the properties and capabilities of the system, and the technologies it builds upon.

Architecture

Providing data confidentiality and integrity for applications is an important component when connecting mobile routers over untrusted networks. Often a VPN solution is used to secure traffic. Bundling the capacity of uplinks makes VPN connectivity difficult, as the VPN tunneling must either work above or below the link bundling technology. Both approaches have severe and complex implications in practice, though.

The onway uplink management solution takes a different approach; it uses VPN tunneling as a core component to bundle uplinks in a transparent way. The use of VPN tunnels can provide seamless connectivity across uplinks, as it hides transport network complexity with different uplink layer 3 addressing, different address families (IPv4 and IPv6) and network address translation along the uplink path.

The core concept is to establish redundant and identical VPN tunnels over each uplink to the same terminating VPN gateway system: Any packet can be exchanged over any of the available tunnel, as the VPN layer abstracts away the details of the underlying network.

Redundant tunnels over multiple uplinks

The main benefit of such a concept is that applications are completely unaware of the (potentially multiple) uplinks involved. Connections and flows can be transparently migrated to a different uplink and VPN tunnel, and applications on either end see no change in addresses or ports. As application sessions are not affected by uplink migrations, flows can be migrated freely for the purpose of load balancing or to recover from failing links.

For VPN tunneling, onway builds upon standard IPsec with the modern IKEv2 key exchange. Usually all traffic is UDP encapsulated for reliable NAT traversal and firewall penetration. Router and gateway authentication is realized using X.509 certificate authentication, modern encryption transforms and hardware acceleration provide the required throughput to saturate bundled uplinks.

One severe implication of the VPN based approach is that all traffic sent over the uplink management system is concentrated to one (or more) VPN gateways, and not directly from the uplink network to the target system. However, in practice, neither traffic costs nor the added latencies actually matter, because they only add a fraction to the cost/latency of the uplinks used in mobile routers. Also, gateway systems can be placed near the uplink network exit or the target application, further reducing any noticeable effect.

If multiple uplinks are available and redundant VPN tunnels have been installed over each, a mobile router has to decide over which tunnel (and uplink) it sends traffic. To balance load, the mobile router has to estimate the quality and/or capacity of each uplink, so it can use appropriate weighting of tunnels.

The most important criteria of an uplink is the general usability. Radio uplink quality may change very rapidly, and sending traffic to a black hole can impact the application behavior severely. onway monitors link state at several layers: radio modem control plane notifications are often not fast enough to provide instant feedback about network connectivity. Directly monitoring the data plane can give link availability feedback within below a second. Such link failures can be compensated by migrating flows to other uplinks within a second, so applications often do not notice any network coverage issues for single mobile network providers.

To further quantify link quality, layer 1 information is included in calculating link weight. This can work fine for radio links providing signal quality indicators. For wired links there is not much feedback, though, as link degradation may often happen in the intermediate network.

To detect any issues in the full uplink path, the onway uplink management solution uses active path probing. Using an adaptive algorithm, latency probes are exchanged between a mobile router and the VPN gateways it is connected to. The measured link latency can give valuable feedback about link quality and saturation, which will be discussed later for realizing QoS mechanisms.

Load balancing and priority groups

Having estimated the availability and the quality of an uplink allows the calculation of an uplink weight. The properties that influence the uplink weight include:

Another property associated to an uplink is its priority group. Uplinks sharing the same priority group balance load according to the weight calculated from the criteria above. For different priority groups, only uplinks for the highest priority get load assigned, unless no link in the highest priority group is considered working. The combination of these mechanism allows flexible configuration for (weighted) load balancing, but also for hot-standby uplink configurations.

Having such fine-grained control about traffic distribution over uplinks can enable very effective cost control mechanisms. If a mobile network operator with good coverage is unwilling to provide fair pricing per volume, one may dynamically reduce the administrative weight to reduce the volume over that operator, without sacrificing service availability at all. These can be good arguments when negotiating pricing with an operator.

Flow distribution

When having assigned weights for the individual uplinks, the traffic forwarded by a mobile router must be assigned accordingly to the associated VPN tunnels.

Assigning individual packets using a weighted round-robin mechanism is not a feasible approach, as this leads to packet reordering. While TCP can handle some level of packet reordering, other protocols are much more sensitive when receiving out-of-order packets. Instead, onway uses per-flow assignments when sending packets from the mobile router. A flow is defined by the packet 5-tuple for source/destination address, layer 4 protocol and layer 4 source/destination ports (if available).

Flow distribution

Using different uplinks for the individual flows of an client or even an application is usually not a problem, as on the VPN layer all flows share the source and destination addresses. The latencies may be slightly different for the individual flows, but this is usually no problem for applications. Due to full address transparency, flows can be reassigned within a second or below to a different uplink to balance load dynamically.

Reverse flow distribution

For traffic sent in the reverse direction from the VPN gateway to the mobile router, the gateway can indirectly send traffic to a specific mobile router uplink by routing it over the corresponding VPN Tunnel. The VPN gateway has much less information to make an educated routing decision, though. onway therefore uses symmetric routing for individual flows, so that the reverse flow of the same (TCP) connection is sent to the same uplink as it has been received from. This allows the mobile router to control the reverse path flow distribution and effectively balance load.

There is a caveat, though, in that a VPN gateway may not notice a failing uplink in useful time (within a second). Therefore, it uses the symmetric flow routing only for connections it recently (within the last second) has received traffic on. If no traffic has been received, it routes the reverse flow over the mobile router uplink it most recently has seen packets for any incoming flow: This ensures that retransmissions are sent over a working path; the mobile router may force the flow to a different uplink again by sending traffic.

Clustering

The idea and the concept of bundling uplinks on a single router using VPN tunnels can be extended to multiple routers. By forming a cluster, multiple router nodes can contribute their uplink capacity to increase the overall bandwidth available, but also provide higher availability by allowing individual nodes to fail.

The onway cluster solution uses a custom synchronization protocol to negotiate uplink state and quality across a group of nodes. Each node learns the available uplink state for all other nodes in the cluster, and a distributed algorithm calculates responsibility for clients and their connections, so only one node actually forwards the client traffic. On the reverse path, the previously discussed mechanisms for flow distribution can be reused to work with tunnels distributed over multiple nodes.

On the mobile router side, the cluster distributes traffic to its nodes by hooking the ARP resolution done for the default gateway. Clients resolving MAC addresses for the shared router IP yield different results for different clients. The addresses of a pool of these handed out virtual MAC addresses can be migrated between cluster nodes depending on the available overall uplink capacity of each node. The node responsible for that MAC address then distributes the traffic to its uplinks. Nodes detecting an uplink failure or failing completely give away their share of the MAC pool to other nodes.

The mechanism with virtual MAC addresses and ARP mangling is a well known clustering strategy in the industry, but has its limitations. All traffic from a single client is tied to a virtual MAC address, and ultimately to a single node. The responsible node can change, but only one node can be responsible. This may be insufficient for certain scenarios, because:

To overcome this limitation, the onway cluster solution adds an additional load balancing layer, called rebalancing. A node receiving traffic can partially redirect it to a different node by reflecting it with changed MAC addresses. The shared algorithm can use information from layer 3/4 to rebalance traffic, so multiple flows from a single client can be distributed over all nodes in the cluster.

Clustering works as an additional layer in front of the individual nodes uplink management, and so can logically extend the uplink bundling across redundant hardware. Priority groups, cost control, traffic separation and even QoS can still be enforced even in cluster mode due to the custom cluster synchronization protocol.

Classless QoS improvements

Bundling uplinks can be beneficial for both capacity and availability. However, it also implies (or allows) that multiple applications share the same uplink infrastructure and fixed costs. While it is possible to use separate uplinks for critical and non-critical applications, this often makes it impossible to make the most efficient use of the available uplinks.

Radio uplinks, for example over LTE networks, in practice often suffer from significant buffer bloat along the full network path. Some buffering is required to communicate over a shared medium, as send opportunities and airtime are available only occasionally. However, loaded links may add a buffering latency of several seconds in both directions. This makes any interactive application almost unusable. For an uplink shared by multiple applications, a non-critical application may increase uplink latency to several seconds by just saturating the link with a few TCP flows. This is unacceptable for critical interactive applications that need the uplink latency to be within a certain range.

As the mobile network, including the radio link, are usually not under control of the mobile router system, deploying QoS mechanisms within the uplink network is usually infeasible. Mobile network operators are often not cooperative in implementing QoS mechanisms as required.

To approach the problem, the onway’s classless QoS solution builds upon the fact that in most networks, the link latency is directly related to the link load. Once the carried traffic reaches the maximum link capacity, the packet buffers in the network start to fill up and increase the time a packet is buffered before forwarding. At a certain point, the additional buffering can not increase the link capacity anymore. Instead, the added latency can even negatively affect the usable throughput (goodput) due to unneeded retransmissions.

Uplink Goodput vs. Latency

By explicitly reducing the send rate over an uplink, the system can indirectly control the buffering induced latencies seen on such an uplink. An intelligent send rate regulation can dynamically adapt the send rate on both ends according to the currently measured latencies.

For the system to work effectively, the latency probes sent on all VPN links must not only measure round trip delays, but also the individual One Way Delays for an uplink. The send rate per tunnel is then regulated per direction, so that the latencies get close to a configurable target latency. Calculating One Way Delays can be difficult, as network based time synchronization is affected by the very same asynchronous link delay. onway therefore uses a proprietary secure network time synchronization protocol specifically designed for accurate One Way Delay estimations.

The target latency defines the sweet spot to regulate to, and usually depends on the uplink medium and the requirements from the application. A low target latency may negatively affect throughput, but depending on the application this may be preferable.

In the implementation, the whole latency measurement and send rate calculation is implemented on the mobile router. However, in downstream direction, the VPN gateway must take care of effectively reducing the send rate to a specific mobile router uplink. Therefore, the mobile router includes instructions for the VPN gateway in the periodically sent latency probes. The VPN gateway uses this information to limit send rate for that mobile router uplink accordingly.

Adaptive queue rate management

As a pleasant side effect, the rate adoption algorithm gives an estimation about the current link capacity (which can change radically within seconds or below). The estimated link capacity is used as additional and very valuable input for the uplink load balancing decision done by the mobile router.

Classful QoS mechanisms

For certain scenarios, it is desirable to handle different classes of traffic differently. Prioritizing certain traffic classes in a queue has an effect only on the queues in the path that are before a bottleneck in that path. Other queues forwarding traffic into a non-bottleneck drain empty anyway. An empty queue by definition can’t do any useful queuing, hence any intelligent queuing does not help. Usually the bottleneck is on the transport network, and therefore not under the control of the mobile router system. So any priority queue on the mobile router (or VPN gateway) system has no effect.

A well known mechanism to implement priority queuing before sending traffic into a transport network is to artificially create the bottleneck on the last system under control (mobile router or VPN gateway). So the traffic sent must be limited to somewhat below the link capacity. Such a queue will start to fill; priority queuing can take place, and some traffic classes can be handled differently than others. With wireless radio uplinks, though, the link capacity is not known and also changes rapidly. This is especially true for mobile routers moving fast.

With onway’s classless QoS mechanism discussed previously, the system has, however, estimated the link capacity. This link capacity can be used to configure a priority/shaping queue for individual classes, and therefore implement classful QoS queuing. This will artificially create and maintain a bottleneck queue under control of the uplink management system.

Having this foundation in place, arbitrary queuing can be implemented rather easily. onway’s solution uses arbitrary layer 3 and layer 4 selectors to tag packets into QoS classes. Classification in the dataplane send routine is done on plaintext packets before IPsec encapsulation to flexibly select application packets. Queuing and shaping, on the other hand, happens after encapsulation in common for a single uplink based on its estimated capacity.

Classful adaptive queuing

VRF based traffic separation

Priorization and fairness is one aspect when sharing bundled uplinks between multiple applications. Another, even more important aspect is security and isolation. Due to the common use of hardware by applications in different security domains, some mechanism is required to prevent crossing these domain boundaries.

Virtual Routing and Forwarding (VRF) is a common technique from Enterprise networking to solve this problem. In onway’s solution, VRFs are again deeply integrated in the whole concept. While the uplink management creates redundant tunnels over multiple uplinks for a single application, multiple applications can share those bundled uplinks using per application domain VPN tunnels. The system is very flexible in which tunnels can use which uplinks.

Having created those multiple application domain tunnels, those (per uplink redundant) tunnels terminate into an application specific VRF domain. Tunnels are strictly bound to that VRF, so traffic can enter into a VPN tunnel from the associated VRF domain, only, and traffic received over a tunnel is strictly forced into the target VRF domain.

VRF isolation is supported both on the mobile router, but also in the backend VPN gateway system. Also, application domain specific tunnels can be terminated to different VPN gateways for physical isolation.

The unmatched flexibility of creating many application domains and dynamically share uplinks for all these applications brings some challenges, especially in scaling the system. onway has invested significantly in scaling and reliability of the overall system. Vertical scaling allows serving thousands of mobile routers, multiplied by horizontal scaling options available.

System showcase

The following chapter, will take a closer look at the behavior of some properties in the onway uplink management solution. It shall demonstrate how the system performs in production, and what the expected benefits of this solution are.

The first measurement shows the complete additive bundling of multiple uplinks with different layer 1 technologies when using multiple flows. The following graphs shows three test runs using iperf with ten concurrent connections in upstream direction. The first run is configured to use the LTE radio uplink only, the seconds uses just the DSL uplink to send data. In the third run the LTE and DSL uplinks are bundled to a virtual uplink.

Bundling LTE and DSL uplinks (stacked)

As expected, the bundling with multiple flows happens completely additive, and the individual rates of LTE (11Mbit/s) and DSL (27MBit/s) are bundled to a total of 38Mbit/s (chart bandwidth is stacked).

Noteworthy is that the application is completely unaware of the bundling on either end; all iperf flows are distributed automatically and use the same endpoint addresses.

The next measurement shows the link failover capabilities of the system. The same two LTE and DSL uplinks are used. This time, the LTE uplink is used as hot-standby fallback uplink only, and shall take over if the DSL uplink fails.

In this test, a single iperf connection sends traffic.

DSL failover with LTE Hot-Standby (stacked)

Somewhere after second 8, the DSL uplink is unplugged and gets non-functional. In less than two seconds, the outage has been detected and the fallback link has been activated. While the the iperf stream has stalled for about a second, the flow immediately recovers; the application did not see any disconnect, as the same TCP connection has transparently been rerouted over the LTE uplink.

Around second 20, the DSL link has been recovered. The system detects the situation and re-establishes the redundant VPN tunnel over DSL. Once running, the LTE link is now degraded to its hot-standby role and the single iperf stream has been re-routed over the (more capable) DSL uplink.

Link failover can work in a active/passive setup, but due to the advanced priority management it is also possible to run links in active/active configuration. Here the link bundling provides the summed up bandwidth of the individual link, and a failing link just degrades the uplink capacity for the physical capacity the uplink contributed.

In the below measurement, iperf receives traffic for twelve constant rate 1Mbit/ s connections. The DSL uplink is bundled active/active with two separate LTE uplinks, forming a virtual uplink bundle.

After startup, both LTE uplinks get two iperf flows assigned, carrying 2Mbit/s traffic each. After some time, connectivity issues are simulated by removing radio antennas. Around second 25, the first LTE uplink gets unusable and is unable to forward traffic. At second 35, the second LTE uplink also fails (bandwidths are cumulatively stacked).

DSL/LTE bundle failover (stacked)

During the bad connectivity, more load gets assigned to the more reliable DSL uplink. Around second 16, the DSL uplink takes over the full load of the LTE uplinks transparently. Around second 26, some TCP flows somewhat stall due to the completely failing LTE uplink. Once the situation has been detected in under two seconds, the system regulates more traffic over the DSL uplink. The TCP flows for a short time compensate for the bandwidth reduction, temporarily sending more traffic from the accumulated TCP buffers. In second 36, the DSL uplink carries all traffic. Short after, the LTE uplinks recover, and contribute again to the uplink bundle.

While individual flows can be degraded due to link outages, traffic is affected only for about 1-2 seconds. Connections stay active, get migrated to alternative links very quickly, and applications can transparently use the bandwidth available over any physical uplink.

Cluster operation

The combination of uplinks can be extended to more than one physical router by creating a cluster. In the setup below, two routers are connected to a layer 2 switch and form a cluster. On the same switch, an external WiFi access point is connected, providing a WiFi network to clients.

Cluster setup

The first router R1 provides an LTE uplink, whereas the second router R2 has both a wireless LTE uplink and a fixed line DSL uplink over a DSL modem. This example shows the combination of cost control with priority groups and link failover, all distributed over a cluster. The DSL uplink is configured with a higher priority, and the two metered LTE uplinks provided by R1 and R2 are used as a fallback only, concurrently taking over if the DSL uplink is unavailable.

For the test, an iperf downstream session is created with 10 connections, initiated by the client to a server in the Internet. In the measurement below, in the beginning all links are operational, and the DSL uplink of R2 is preferred for cost reduction. It provides a downstream bandwidth of about 75 Mbit/s.

Cluster migration (stacked)

Around second eight, the DSL uplink fails. The cluster synchronizes this fact, and enables the lower priority uplink via the LTE modems, combining the LTE capacity of both routers with an active/active configuration. Detecting the failing DSL uplink and re-routing the TCP streams to the LTE uplinks takes about two seconds.

At second 27, the R1 with the faster LTE uplink completely fails, and only the LTE uplink over R2 using a different operator is left. All TCP connections get migrated to R2. The uplink has limited bandwidth, but can still provide basic service. At second 44, the R1 recovers, and R1 rejoins the cluster with its LTE uplink, taking over some TCP sessions to increase the total available bandwidth to the combining individual LTE uplink rates.

At second 57, the DSL uplink recovers. The cluster restores its active/passive mode and re-routes all traffic over the unmetered DSL uplink.

Important to note is that all the iperf TCP sessions survived all link failovers. They have traffic stalled for at most 2 seconds during these major link or hardware failures, but all TCP connections have perfectly survived. The client did not notice any disconnects, but its connections have transparently been migrated back and forth between uplinks and physical routers.

Classless QoS

The onway classless QoS mechanism is about uplink usage fairness and latency reduction on uplinks carrying high load. For the latter, a target latency is defined that the system tries to maintain by reducing send rate accordingly.

The first example shows a DSL uplink saturated at the link capacity of roughly 30Mbit/s. The first 30 second iperf run using ten TCP connections fills the send buffer in the DSL modem, adding about 200ms latency. While this may be acceptable for some applications, the latency can be much higher when using other network equipment or layer 1 technologies.

DSL upstream latency reduction

In the second 30 second iperf run, the system is configured for a 100ms target latency. The mobile router reduces its send rate dynamically to keep the latency within limits, and most of the time under 100ms. This gives more responsiveness for interactive applications, without sacrificing throughput noticeably.

In an alternative test setup, the latency reduction system is applied to an LTE uplink in downstream direction. The mobile router measures latencies, and instructs the VPN gateway to reduce the send rate to not exceed the target latency.

The first 30 second iperf run has the algorithm disabled, and RTT latencies go up to over a second. As loading a web page usually involves many dozen round-trips to even show any content, the user experience is unacceptable.

The second 30 second iperf run uses a target latency of 500ms, and adaptively reduces downstream send rate once the latency target is exceeded. Throughput is minimally reduced compared to the first run; however, trading some raw throughput is favorable given the latency reduction achieved, especially for interactive applications.

The third iperf run has a target latency of 300ms configured. This implies a more aggressive rate reduction with slightly less overall throughput. The latencies get into acceptable ranges, though, and the user experience for loading a web page improves dramatically.

LTE downstream latency reduction

As shown by these results, reducing latency on different uplink media by using dynamic send rate adaption does work in practice. While the latency reduction alone can be a huge improvement for the user experience, there is more: The system can estimate the link bandwidth available in real-time. This allows much more qualified load balancing decisions, but also is the foundation for introducing classful QoS mechanisms over third party transport networks.

Classful QoS

By configuring traffic classes, based on layer 3 and layer 4 selectors, certain traffic can be priorized. Non-critical traffic can be shaped relative to the available bandwidth.

In the measurement below, a non-critical application competes with a critical application; here both are TCP based iperf streams. The non-critical iperf session uses 20 TCP connections, while the critical application uses only a single TCP connection.

Without classful priorization, each TCP connection competes equally with all others; the critical application theoretically only gets 1/21 of the available bandwidth. In the shown test runs, the non-critical application sends traffic for 30 seconds. Ten seconds after the non-critical application starts, the critical application competes for bandwidth for ten seconds. As TCP usually favors existing flows over new ones, the critical application usually would have severe difficulties in establishing flows with acceptable rates.

LTE classful priorization (stacked)

The first measurement shows the non-critical application above the critical application without any explicit priorization (rates stacked in the graph). As the critical application starts later and has significantly less flows, it is unable to compete for bandwidth. It does not get above a rate of roughly half a Mbit/s.

In the second measurement, classful priorization is used, where the single critical flow gets priorized over the 20 non-critical flows. As can be seen, the single flow can immediately push back the non-critical application and gain adequate throughput of several Mbit/s. The non-critical application is throttled, and only can recover to the original rate after the critical application does not send traffic after another 10 seconds.

QoS-enabled test drive

The previously showcased data has been collected in stationary measurements. However, much more challenging are scenarios with varying LTE coverage that mobile routers must handle. Available throughput can change fast when on the move, requiring the QoS solution to prefer critical traffic.

To measure the performance of the overall system in moving conditions, a test drive of roughly ten minutes and a distance of 5km has been examined. A mobile router with two LTE modems using two different operators provided the measured uplink. The scenario simulates and measures traffic using four QoS classes:

The first three traffic classes are considered critical and shall be favoured over the passenger WiFi in all cases. WiFi traffic is best effort and shall degrade under bad signal conditions. The expectation is that the uplink management system balances traffic over the modems dynamically, that latencies of the uplinks are reduced to an acceptable limit, and that non-critical traffic is limited so that the critical applications experience low loss, low latencies and a guaranteed bandwidth.

In the following diagrams, the left part of the chart shows the measurement of the the test drive without any QoS configuration. All traffic competes equally. The right part of the chart shows the very same test drive, but this time with QoS configured according to the traffic classes outlined above. The uplink target latency is configured to 300ms. Both uplinks carry traffic from all classes equally.

The first chart summarizes the total throughput measured over the total of available LTE uplinks. At the first kilometre, the signal quality has been rather good with combined peak rates of 100 Mbit/s. The rate then drops before recovering in the last third of the test course.

System total throughput

The total throughput is higher with QoS disabled in this test. This is expected, as Guest WiFi traffic is throttled to ensure low latencies and guaranteed bandwidth for the critical applications. When looking at the WiFi traffic class consisting of HTTP bursts and file downloads, it can been seen that mostly the high volume file downloads are affected:

Throttling WiFi traffic

The raw file downloads indicated by the DL rate curve, shown as average for each individual user, is the main driver of total throughput: the curves closely correlate. If QoS steering is in effect, these downloads are actively limited. The average round trip time is significantly lower with QoS enabled, as file download data is prevented from bloating up on-path buffers. Interactive applications, but even web surfing, can benefit from the lower latency. For guest WiFi users, the experience for surfing the web is positively affected by the QoS latency reduction, even if plain file downloads are somewhat throttled.

When looking at critical application traffic classes, we see a lot of improvements if a QoS configuration is applied. For the vehicle management application, average round trip times are roughly halved:

Management traffic

The simulated application sends a burst of eight request/response pairs, repeating these bursts every five seconds. A burst is considered successful if it can complete within the time window of these five seconds. The Success rate shows, in %, the average number of bursts that could complete within the time limit. Due to shorter round trip times, the success rate can be improved significantly for management traffic if QoS steering is applied.

The third class of traffic to look at is video streaming. The main criteria for smooth video playback is the playback buffer state: The buffer is consumed by video playback, and filled constantly using a TCP stream. A small four second playback buffer is used in this example. Without QoS applied, the playback buffer runs dry very often and video playback would stop several times. Using QoS bandwidth reservation, the playback buffer can always keep above around 25% fill state for continuous video playback.

Video buffering

The last and most sensitive application is realtime audio, which requires low latencies and minimal jitter:

Audio latency

The median of RTT measurements shows that the use of QoS can significantly push down latency and provide a much more constant audio experience. The worst 20% of realtime audio packets without QoS experience a round trip time of half a second and more. Using QoS bandwidth reservation, the latency can be reduced to about 150ms even for the worst 20% of the packets. This may or may not be enough depending on the application; Use cases that require an even lower latency can adjust the target round trip time even further using a simple parameter, but at the cost of more aggressive bandwidth limitation for WiFi guests.

Conclusion

Building a state-of-the-art uplink management system is a challenging task, as the requirements range from security, fail-safety, capacity bundling, Quality of Service, fair application multiplexing to cost optimization and more. Delivering a solution that can address all these issues is what a modern mobile router solution requires.

By building upon proven VPN technologies, onway has managed to build a system that excels in all areas. A distributed system takes care of getting the most out of the available uplink hardware and networks, while allowing to serve multiple applications over the same infrastructure.

Even simple use cases can greatly benefit from the fail-safety, load balancing and zero-configuration QoS capabilities. Where more complex scenarios previously required dedicated hardware for individual applications, onway’s uplink management solution can provide significant cost benefits by re-using the same networking infrastructure with advanced network virtualization.