Question: Multipath: Enabling on box with multiple WAN interfaces

[jdrews]

I am trying out multipath on 1.4.0. Based on the manual it says:

will automatically detect available physical interfaces and begin allocating across all paths.

I'm a little stuck on how to get ZeroTier to recognize the multiple WANs available to it on one linux machine. Any pointers?

How does ZeroTier determine that a physical interface is "available" for use in multipath?

Use Case

Site A 3 WAN connections, all of which are plugged into a Debian 10 box via separate NICs. The Debian box has ZT 1.4.0 with local.conf "multipathMode": 2.

Site B 1 WAN connection and a Debian 10 box. The Debian box has ZT 1.4.0 with local.conf "multipathMode": 2.

Details

1. Site A and Site B are in the same ZeroTier network and can ping each other on ZT network IPs.
2. Also enabled FRRouting at both sites and running OSPF network across them.

Goal Able to load balance traffic across all 3 WANs at Site A to aggregate and improve bandwith at Site A. Site B acts as the 0.0.0.0/0 default router for all traffic from Site A.

[jdrews]

Playing around with this some more. I thought maybe ZeroTier needed to see multiple routes in ip route to a peer, each going out a different WAN interface. So I constructed multiple routes via my 3 WAN interfaces, all with different metrics.

That didn't seem to work as I still see "multipath": null, and "multipathMode": 2, in zerotier-cli -j status on both sides.

[jdrews]

I drew it out as pictures are worth more than words.

In this case Site A and Site B are Debian 10 boxes. Site A has multiple WAN connections, albeit through routers that hand out private subnet IPs. Site B has a public IP directly attached to it.

[jdrews]

I also confirmed via zerotier-cli -j peers that it's only seeing one path on either side.

[jdrews]

A bit of feedback here: It's really hard to get multipath working correctly over multiple unique WAN connections. I've got it to detect two of the WAN connections, but the last one is very elusive. I've had to fudge with the routes a ton.

It would be useful if we could tell ZeroTier what interfaces+gateways it should use for the paths.

[2600box]

I'm in the same boat. Can you let me know what you did with your routes to get the two recognised?

[jdrews]

In my case, I configured Site B to have 3 WAN IPs. I then configured Site A's routing table to push traffic through one of the three WANs for one of the three Site B IPs.

[dhbaird]

This affects me too. I have devices with only 1 internet connection, that I would like to connect to both of 2 connections available at another site, and haven't figured out how to get this case working. (I did try @jdrews solution mentioned in the above comment, and it indeed works for me with 2+2 connections and I manually configure the routes on each side; But what about a 1+2 configuration and without manual route configuration?)

[justinh-rahb]

Any solid instructions for this? Plenty of mention of this awesome new feature but absolutely nothing on how to actually make it work...

[jbfuzier]

Same here :(

[jocelynke]

Any news ?

[o-s-e]

Hi, is there any update? I tried to enable it, but it doesn't work: zerotier-cli -j status { "address": "d6214ff7fc", "clock": 1602235526986, "config": { "physical": null, "settings": { "allowTcpFallbackRelay": false, "bind": null, "controllerDbPath": null, "multipathMode": 2, "portMappingEnabled": true, "primaryPort": 9993, "rabbitmq": null, "softwareUpdate": "disable", "softwareUpdateChannel": "release" } }, "multipath": null,

[laduke]

New release coming shortly. https://github.com/zerotier/ZeroTierOne/commit/fe1ce885a04ae68894b8a8fc1892db212addbf6e

[joseph-henry]

Hello everyone,

There's been a beta release of our new multipath functionality (tagged as 1.5.0), in addition to @laduke's comment above you can find the documentation here: ~~https://zerotier.atlassian.net/wiki/spaces/SD/pages/568459265/Multipath~~ https://docs.zerotier.com/zerotier/multipath

Let us know if you have any questions, feature requests or discover any bugs. Thanks.

[wyattanderson]

I have a functioning DIRECT connection between two ZeroTier peers, each running v1.5.0. Is there anything more I need to do to enable multipath beyond the following local.conf?

{
    "settings": {
        "allowTcpFallbackRelay": false,
        "portMappingEnabled": false,
        "defaultBondingPolicy": "balance-aware"
    }
}

This is the configuration on both peers. Peer A has three interfaces available, each on separate subnets. Peer B has only one interface. This the output from zerotier-cli listbonds on either peer:

    <peer>                        <bondtype>    <status>    <links>
      NONE                              NONE        NONE       NONE

[Vicidel]

Hello, I know the issue is old, but I have a similar issue on 1.6.4

I have two devices connected with two links: one LAN and one WAN. The intended behavior is to send traffic to both links, and when the LAN link disconnects (out of range is expected) the bond switches over to the WAN link. It is the case, but only after a timeout of 250 seconds, which is way too long for my application Both devices have "broadcast" multipath configuration, but if the LAN disconnects it never switches over to the WAN

Below the output of zerotier-cli info -j, which reflects my configuration file /var/lib/zerotier-one/local.conf

{
 "address": "xxxx",
 "clock": 1617788744639,
 "config": {
  "physical": null,
  "settings": {
   "allowTcpFallbackRelay": false,
   "bind": null,
   "controllerDbPath": null,
   "defaultBondingPolicy": "broadcast",
   "portMappingEnabled": true,
   "primaryPort": 9993,
   "softwareUpdate": "disable",
   "softwareUpdateChannel": "release"
  }
 },
 "online": true,
 "planetWorldId": xxxx,
 "planetWorldTimestamp": 1591823606334,
 "publicIdentity": "xxxx:xxxx,
 "tcpFallbackActive": false,
 "version": "1.6.4",
 "versionBuild": 0,
 "versionMajor": 1,
 "versionMinor": 6,
 "versionRev": 4
}

Here is the link information obtained with zerotier-cli bond list -j, I have weird values for bondingPolicy, numAliveLinks/numTotalLinks which I cannot explain. The IP 192.168.168.194 is the one with the LAN link

{
  "address": "xxxx",
  "bondingPolicy": 150988360,
  "isBonded": false,
  "isHealthy": true,
  "latency": 25,
  "numAliveLinks": 17,
  "numTotalLinks": 1,
  "paths": [
   {
    "active": true,
    "address": "192.168.168.194/9993",
    "expired": false,
    "lastReceive": 1617788944795,
    "lastSend": 1617788944766,
    "preferred": true,
    "trustedPathId": 0
   }
  ],
  "role": "LEAF",
  "version": "1.6.4",
  "versionMajor": 1,
  "versionMinor": 6,
  "versionRev": 4
 }

[Intensity]

@joseph-henry / @laduke : I'm wondering if multipath is considered feature complete at this stage. Also are there forthcoming enhancements to this in 2.0?

[Sergey-Afonin]

@Intensity I can't make it working as well. I tried I lot of different configurations. I have one node with good connection to internet and one node with two weak interfaces. My dream is to run broadcast there. But I can't make working any multipath configuration.

[scimitar4444]

@Sergey-Afonin Unfortunately, I didn't make it in any constellation either. Probably it does not work with the current version. I hope that something new comes soon. I've been waiting for 3 years ;-)

[scmtnboy]

Each WAN interface needs to have it's own routing table with default route, then you need to configure the appropriate policies for each routing table. I've been able to get 3x WAN links up in a balance-aware configuration. I can post my /etc/network/interfaces if you have questions.

[Mohamed-GN]

scmtnboy can you post your /etc/network/interfaces file because I am having problems with running two WANs with zerotier