Multipath[TCP] support for ConnMan

March 22, 2016

We’ve been working on adding multipath routing to ConnMan. It’s supported both at ConnMan’s service and session level. On top of that we also added Flow Selectors for identifying sessions based on traffic type. The basic idea was to make Multipath TCP work with ConnMan nicely. This work is beyond basic multipath routing support. We are in the process of upstreaming the work to ConnMan.

In this post we’ll present ConnMan and Multipath TCP. Then we will show in detail what we implemented and how it can be used.


Introduction to ConnMan and Multipath TCP

ConnMan is a lightweight network manager for Linux. Think NetworkManager for embedded devices. It’s modular through it’s plugin architecture. It also has native DHCP and NTP support. It can deal with a plethora of connection technologies: Ethernet, WiFi, Cellular (via oFono), USB Gadget, Bluetooth. A networking interface is called a Service in ConnMan terminology. The concept of a Session is also relevant for us: it’s a way to control connectivity for applications with different requirements.


MultipathTCP is a TCP extension that allows using multiple paths to maximize resource usage and increase redundancy. Normally TCP uses only one stream per socket. MultipathTCP will try to open sub-streams on every available interface for one socket and juggle the data across the sub-streams based on TCP congestion information. This is an oversimplified explanation of course. There’s a MultipathTCP IETF working group if you want to know all the gory details. The biggest advantage of MPTCP is that it’s perfectly compatible with existing applications. All the magic happens underneath.

From now on, when we talk about MPTCP we refer to the Linux implementation. Although this implementation has been deployed in the wild, it’s not present in the upstream kernel yet.


Basic Multipath support

The recommended way to use MPTCP in Linux is to configure source routing for every network interface. MPTCP binds sub-streams to different source IPs and the routing configuration is necessary to be able to use multiple interface. Automatic configuration is supported only via some setup scripts for Debian-based and Gentoo-based systems.

Now it’s possible to do it via ConnMan. We added a per-Service option called MultipathRouting. It’s off by default but it can be switched on via the D-Bus API. There’s also the –mpathrouting option which turns it on by default for all services. So, let’s have a look how this works.

Let’s start with a bunch of interfaces that have MultipathRouting off:

$ connmanctl services
*A R Wired ethernet_36e0bdee6014_cable
*A R Wired ethernet_265676f2a6d2_cable
*A R Wired ethernet_fe96f100e49b_cable
$ ip rule
 0: from all lookup local
 32766: from all lookup main
 32767: from all lookup default

Now let’s turn on MultipathRouting for them:

$ connmanctl config ethernet_36e0bdee6014_cable --mpath_routing yes
$ connmanctl config ethernet_265676f2a6d2_cable --mpath_routing yes
$ connmanctl config ethernet_fe96f100e49b_cable --mpath_routing yes
$ connmanctl services
 *AMR Wired ethernet_36e0bdee6014_cable
 *AMR Wired ethernet_265676f2a6d2_cable
 *AMR Wired ethernet_fe96f100e49b_cable

Okay, the services have MultipathRouting property enabled. Let’s see the routing rules:

$ ip rule
 0: from all lookup local
 32763: from lookup 283
 32764: from lookup 281
 32765: from lookup 279
 32766: from all lookup main
 32767: from all lookup default

$ sudo ip route show table 283
 default via dev vethc3 dev vethc3 scope link

Neat. So we have 3 interfaces, all 3 with multipath routes created.

You don’t need a MultipathTCP kernel to be able to use this stuff. It’s a pretty cool standalone feature. The only MPTCP dependent feature we added was the per interface MPTCP on/off/backup state flag. The property name is MultipathTcpLink. It’s on by default. It can also be set via connmanctl:

$ connmanctl config  --mpath_tcp_link <on|backup|off>


Fancier stuff – Sessions with multipath support

Now sessions come into play. At a first glance it’s not really clear how sessions are useful. But once you realize that different applications have different connectivity needs, the role of sessions becomes clear. Examples of various requirements from applications:

  • Different applications want to use different technologies.
  • … or have different preferences for technologies.
  • Some applications may need policy routing.
  • Applications that have the ability to piggyback on other application’s session being online.

The effect of a session on the system is that routing and iptables rules are added to obtain a “policy routing” table. Here’s how it looks like:

$ ip rule
0: from all lookup local
32762: from all fwmark 0x10000 lookup 65536
32763: from lookup 265
32764: from lookup 263
32765: from lookup 261
32766: from all lookup main
32767: from all lookup default

$ ip route show table 65536
default via dev vethc2

Notice the fwmark 0x10000 part? Each _session_ has it’s traffic tagged with a fwmark  so that the routing rules to take effect. But where to route the traffic? ConnMan chooses one service for each session. The criteria for choosing the service are based on the AllowedBearers list and the ConnectionType. You can find more details in the session overview document. Well talk a bit more about how the criteria for configuring the session fwmark in the next section.

Now let’s talk about our addition: session based multipath support. What does that mean? Basically you can have traffic for one session going over multiple services. Each session contains a MultipathRoutedServices list. This list is populated automatically with services that

  • are in the AllowedBearers list.
  • have MultipathRouting enabled.
  • are connected in state. Based on the ConnectionType setting.

…or manually via the D-BUS API.

Ok, so let’s say we created a session. By default the ethernet services will attach to it. If you monitor the session you would get the following output:

Update called at /foo/bar
 AllowedBearers = [ ethernet ]
 Bearer = ethernet
 ConnectionType = any
 State = connected
 FlowSelectors = [ ]
 MultipathRoutedServices = [ ]
 IPv4 = { Netmask= Method=manual Address= }
 IPv6 = { }
 Interface = vethc2
 Name = Wired
 Update called at /foo/bar
 MultipathRoutedServices = [ ethernet_36e0bdee6014_cable ethernet_fe96f100e49b_cable ]

Notice how the session uses vethc2 as the default interface. You could also see that in the routing tables above. What’s new is the 2 other services present in the MultipathRoutedServices list. Let’s check out the routing table:

$ ip rule
0: from all lookup local
32760: from fwmark 0x10000 lookup 261
32761: from fwmark 0x10000 lookup 265
32762: from all fwmark 0x10000 lookup 65536
32763: from lookup 265
32764: from lookup 263
32765: from lookup 261
32766: from all lookup main
32767: from all lookup default

Notice how rules 32760 and 32761 point to the service routing tables. The default session didn’t change.

Of course, when one of these services gets disconnected, the rules will disappear as well.


Flow selectors

Currently ConnMan supports only user id and group id as session policy selectors. That’s quite limited. It would be nice to be able to bind certain kinds of traffic to a session. Enter Flow Selectors: a flow is defined by a source/destination ip, protocol and source/destination port. Every flow will get it’s iptables rule with the session’s fwmark. However, you can also omit some of the fields. So, it’s just a fancy name SDN-ish name for a 5-tuple iptable rule.

The user can set the FlowSelectors list for a certain session. Let’s look at an example. We add a couple of rules by using the test-session script provided by connman:

$ ./connman/test/test-session change /foo bar FlowSelectors \ \*-*-* \

Normally if you use this from your application, you should get the following output

Update session settings
Update called at /foo/bar
 FlowSelectors = [*-*-* *-*-udp-7777-8888]

And the iptables rules look like this:

$ iptables -t mangle -L
Chain connman-OUTPUT (1 references)
target prot opt source destination
MARK tcp -- tcp spt:1111 dpt:2222 MARK set 0x10000
MARK all -- MARK set 0x10000
MARK udp -- anywhere anywhere udp spt:7777 dpt:8888 MARK set 0x10000

And there you are.

The nice thing about the flow selectors is that you don’t need a policy file to configure Flow Selectors. They work in without or in conjunction with the user/group id associations.



So, that’s it. Now we are working on cleaning up the patches and sending them upstream. Let’s see where we go from there. Although this work was meant for Multipath TCP, it turns out that is usable on any Linux kernel.