Segregate routable and non-routable networks

[Stebalien]

Currently, we announce all IP addresses we're listening, even private ones, and will dial any address we find for a peer, even private ones. We do this intentionally so we can connect to nodes on our local network. Unfortunately, this causes issues like https://github.com/ipfs/go-ipfs/issues/5511 and really pisses off users (for obvious reasons).

One reasonable solution (IMO) is to segregate routable and non-routable networks. That is:

1. If we learn about a non-routable IP address from a routable network, discard it.
2. If we know a non-routable IP for a peer, only distribute it to peers with non-routable addresses.

We'll still be able to discover and dial nodes on non-routable networks through local discovery (mDNS). Additionally, if we're running IPFS on a VPN and all peers have non-routable addresses, everything should "just work".

This will mostly require changes to the DHT and the Identify protocol.

Future work:

• https://github.com/libp2p/notes/issues/3
• Separate DHTs? We could have a private one and a public one.

[rob-deutsch]

Potentially a silly question, but what's the definition of "routable network" and "non-routable network" in this proposal?

[vyzo]

The usual interpretation is publicly routable address range -- check wikipedia for a list of private networks.

[raulk]

I've been thinking about this as well in the context of https://github.com/libp2p/libp2p/issues/47.

One possible design is to add an API in go-multiaddr-net that allows to query Routability() of multiaddrs, returning an "enum" with the following values:

• PUBLIC.
• PRIVATE => as per Private Network.
• PROXIMITY => future: Bluetooth or proximity-based technologies.
• LOOPBACK.
• NA => not applicable.

[raulk]

^ Identify could use this API to filter addresses. And the peerstore could use this to prune existing entries.

[upperwal]

Maybe after segregation we can try connecting top to down like if WAN network is same reject it and move to LAN network if that is same dial using loopback on different port.

---

Peer A: WAN (1.2.3.4/24) -> LAN (192.168.1.100) -> LOOPBACK (127.0.0.1:3000)

Peer B: WAN (1.2.3.4/24) -> LAN (192.168.1.100) -> LOOPBACK (127.0.0.1:4000)

If A wants to connect with B: Because they are on the same LAN and WAN network they can connect using loopback:port

---

Peer A: WAN (1.2.3.4/24) -> LAN (192.168.1.100) -> LOOPBACK (127.0.0.1:3000)

Peer B: WAN (1.2.3.4/24) -> LAN (192.168.1.200) -> LOOPBACK (127.0.0.1:4000)

WAN is same, connect on LAN.

---

Peer A: WAN (1.2.3.4/24) -> LAN (192.168.1.100) -> LOOPBACK (127.0.0.1:3000)

Peer B: WAN (2.2.3.4/24) -> LAN (192.168.1.100) -> LOOPBACK (127.0.0.1:4000)

Totally different network. Dial on WAN.

Can this work?

[Stebalien]

@upperwal I'm having trouble parsing your first sentence.

[upperwal]

@Stebalien So instead of trying all the observed addresses we can segregate the addresses into different categories like PRIVATE, PUBLIC or LOOPBACK (as @raulk suggested) and then we can try connecting in the following order:

1. PUBLIC: Connect using PUBLIC endpoint only if peer A and B are on different PUBLIC IP's. If they are on the same PUBLIC IP's a shorter path exist. Case 3 above shows this.
2. PRIVATE: If they are on the same PUBLIC IP check if they are on the same PRIVATE IP. If they are not, connect using PRIVATE endpoint. Case 2.
3. LOOPBACK: If their PUBLIC and PRIVATE IP's or endpoints are same they are on the same machine. Use loopback:port to dial. Case 1.

[Stebalien]

~I'm not sure that'll solve the issue. Many nodes simply aren't reachable so we'll end up spamming a bunch of non-routable addresses anyways. Additionally, IMO, we generally want to take the opposite approach when dialing. We should prefer private/loopback addresses as those addresses should fail faster (and the resulting connections may often be faster). However, we don't want to do this unless we have a reason to believe that these addresses will actually work.~

edit: I need to read entire comments before responding.

[rob-deutsch]

The usual interpretation is publicly routable address range -- check wikipedia for a list of private networks.

Okay. Maybe it wasn't such a silly question, because I assumed a different definition.

I think we should limit the distribution of non-routable addresses even more. I think that non-routable IPs should only be distributed to peers with non-routable addresses in the same subnet as the non-routable IP we want to distribute. I.e. the line in the OP could should be changed is:

2. If we know a non-routable IP for a peer, only distribute it to peers with non-routable addresses in same same subnet as the non-routable IP.

Also, could the proposal be amended to say that we send only our non-routable IP to non-routable nodes in the same subnet. I.e. we don't also send the routable IP. This will avoid NAT and IP-level routing problems, and I would definitely prefer it. However, I don't know if there are edge cases, or advanced network setups, that this might be non-ideal for. I'm especially unsure about IPv6.

In general, how do other protocols/software handle this problem of public/private addresses? Is there some prior art we can refer to.

[Stebalien]

I think that non-routable IPs should only be distributed to peers with non-routable addresses in the same subnet as the non-routable IP we want to distribute. I.e. the line in the OP could should be changed is:

(that was actually my first thought as well)

Unfortunately, that won't work on complex networks: it's entirely possible to have multiple private subnets joined into a single network. As far as I know, this is actually an extremely common setup on corporate networks. Different buildings/floors will be on different subnets and the subnets will be connected by, e.g., a VPN.

In general, how do other protocols/software handle this problem of public/private addresses? Is there some prior art we can refer to.

Manual configuration and service discovery through mDNS (local multicast). We have the second but the first probably isn't going to work well for us (although we could provide some kind of configuration language...).

---

However, I'd would like to amend my original proposal:

1. Localhost addresses should only be advertised to other localhost nodes.
2. IPv6 link-local addresses (when we eventually add support for them) should only be advertised to nodes on the same link. This isn't an issue currently (we don't support these) but we should keep it in mind.

[upperwal]

We should prefer private/loopback addresses as those addresses should fail faster (and the resulting connections may often be faster). However, we don't want to do this unless we have a reason to believe that these addresses will actually work.

@Stebalien The given approach does prioritise loopback/private addresses and it also gives you a way to check if these addresses work.

Let me explain it again.

Solution to this problem is, to somehow identify the location (network) of peer A and B.

if samePhysicalHost(A, B):
  dial(loopback)
else if sameLocalNetwork(A, B):
  dial(private_ip)
else
  dial(public_ip)

1. Now we start by comparing the public_ip (comparing is simple string matching which can happen on the node) of the destination node with the source node, If they match, we can safely assume they are on the same network hence destination should not be dialed using public_ip (we skip dial using this address so no overhead). If they don't match, they are on different network, safe to dial using this address and stop dialing using other addresses

2. We do the same with private _ip but only if public_ip matches because now the best candidate is private_ip. If private_ip for both matches we again skip using it (because they are on the same private network, loopback is the best option here). If they do not match we assume the private network for peer A and B are different hence safe to dail using private_ip. Do not dial with any other address.

3. If private_ip and public_ip is same go with the loopback because both the peers are on the same machine.

This way we are checking for the subnets in which peer A and B lies and dialing accordingly.

This can also work for complex networks like you described above given we can query the subnet router for it's interface address (and subnets will have different IP signatures so distinguishable). If not, a fallback strategy can be used where:

If peer A and B are on different subnet within an organisation connected to a VPN with observable addresses as follows:

Peer A: 1.2.3.4/24:1234 -> [some subnet X which isn't visible] -> 192.168.1.2 -> 127.0.0.1:3000 Peer B: 1.2.3.4/24:6789 -> [some subnet Y which isn't visible] -> 192.168.1.2 -> 127.0.0.1:4000

Now we will assume that they are on the same host but they are not. We can try connecting using loopback but if it fails we try private IP if that fails we goto the public one. This case is applicable only when we could not find the subnet IP.

Man, I need a diagram. 😛 I hope I am much more clear this time.

[rob-deutsch]

Unfortunately, that won't work on complex networks: it's entirely possible to have multiple private subnets joined into a single network. As far as I know, this is actually an extremely common setup on corporate networks. Different buildings/floors will be on different subnets and the subnets will be connected by, e.g., a VPN.

That's a very good point, but there's a few other sides to the coin:

1. Information leakage - It's 'bad form' to pass information about one subnet to another subnet. Multi-subnet corporate networks are common, but so are PCs that connect to a local network and a privacy-conscious VPN. Or home network and corporate network.

2. Existent but unrouteable private addresses - Another rare case, but I might have a node that's connected to two independant private networks that can't actually talk to each other (e.g. VPN). In a way, this is an extension of the problem we're trying to solve here i.e. not publishing private IPs to public networks.

(To be clear, I am not married to any particular proposal solution here. I just think its an interesting topic, and I hope I'm contributing to the conversation.)

For what its worth, I think that your proposal is a strict improvement on the current setup, and I would be all-for it being implemented, but I'm not sure I'm convinced that its the end of the road in terms of solutions.

[Stebalien]

@rob-deutsch

Information leakage - It's 'bad form' to pass information about one subnet to another subnet. Multi-subnet corporate networks are common, but so are PCs that connect to a local network and a privacy-conscious VPN. Or home network and corporate network.

Yeah, this bugs me too (related https://github.com/ipfs/go-ipfs/issues/1771). And you're right, that's bad form. Really, we should just make this configurable (to some extent). Personally, I'd start with my proposal (as you said, it's a strict improvement but is less restrictive) and we can then add configuration options for more restrictive setups.

Existent but unrouteable private addresses. Another rare case, but I might have a node that's connected to two independant private networks that can't actually talk to each other (e.g. VPN)

This I'm less worried about. Yes, it can happen but oh well, a few dials fail in rare cases. IMO, the bigger issue is dialing private IP addresses on every dial.

---

@upperwal

Ah. I saw the in-order part, read the headers, and completely misread your proposal. That's actually a really cool solution.

There are a few drawbacks:

• We could both have different public IP addresses but be on the same VPN.
• One or both peers may not know their public IP addresses. They may even have multiple public IP addresses on some of the nastier NATs. In this case, we'd have to just dial every address.
• It still leaks private information (@rob-deutsch's issue).

However, it's a really cool heuristic and we can probably just plug it into the dialer with little work.

[Stebalien]

@upperwal want to try plugging this logic into go-libp2p-swarm/swarm_dial.go (if you have time)?

[upperwal]

@Stebalien Yeah, its a simple solution with little refactor. Sure, let me give it a try.

[upperwal]

@Stebalien

I am able to order the ip list as []public, []private, []loopback by implementing Routability()

unordered
[/ip4/192.168.2.229/tcp/28447 /ip4/14.114.26.227/tcp/12540 /ip4/14.114.26.227/tcp/10024 /ip4/127.0.0.1/tcp/4001 /ip4/192.168.203.113/tcp/4001 /ip4/172.17.0.1/tcp/4001 /ip6/::1/tcp/4001 /ip4/14.114.26.227/tcp/10782]

ordered
[/ip4/14.114.26.227/tcp/12540 /ip4/14.114.26.227/tcp/10024 /ip4/14.114.26.227/tcp/10782 /ip4/192.168.2.229/tcp/28447 /ip4/192.168.203.113/tcp/4001 /ip4/172.17.0.1/tcp/4001 /ip4/127.0.0.1/tcp/4001 /ip6/::1/tcp/4001]

Next step would be to find the correct subnet. The only problem is that the peer dialing does not know it's observed address. InterfaceListenAddresses() can only give [/ip4/127.0.0.1/tcp/52575 /ip4/192.168.1.101/tcp/52575 /ip6/::1/tcp/52576]. Hence segregating subnet would be difficult. Any suggestion?

[cannium]

A quick and (somehow) dirty solution would be to pass *BasicHost when creating swarm instance(https://github.com/libp2p/go-libp2p/blob/master/config/config.go#L94), and get observed addresses by *BasicHost.IDService().OwnObservedAddrs() A more graceful solution might need some refactoring, store observed addresses together with peerstore. Just my personal opinion.

[vyzo]

Unfortunately that can't work -- it would create a circular dependency.

[cannium]

So some refactoring is unavoidable...

[upperwal]

I think identity protocol can help (with little refactoring) where destination peer can send us back our observed address during protocol negotiation. Observed address can be stored and made available by some API.

Edit: Keeping our observed address can help us take intelligent and optimised decisions when it comes to routing and connectivity.

[Stebalien]

Damn. That's annoying. I don't know of a simple fix.

Next step would be to find the correct subnet.

We don't really need to know anything about subnets to get this to work, we just need our observed external address.

[upperwal]

Damn. That's annoying.

I know.

By "Next step would be to find the correct subnet.", I meant to find the shortest path to dial on and yes that would only require our observed address.

@Stebalien Is #427 related to the other peer's observed addresses (the one I would be connected to)?

[Stebalien]

Yes but that won't help. The issue here is that the swarm doesn't have access to information like this.

One solution is to tell the peerstore about observed addresses. That's likely the best solution. Thoughts @raulk/@bigs?

[raulk]

@Stebalien I feel we need a component in the system that serves the role of a "topology manager", so that it can take decisions based on a number of factors like:

1. our observed address(es),
2. the node's primary role (e.g. specialised types: relay, DHT, rendezvous, etc. or just a general node),
3. our target network (e.g. public, private, etc.)

Personally I'd like to keep the peerstore on the thin side, ideally circumscribed to storing data about peers without taking decisions.

In the context of this issue, the topology manager would segment peers based on their routability, and it would decide which maddrs to keep for each peer, based on the host's interest and role.

[Stebalien]

Yeah, you're probably right. Basically, we'd have:

• Peerstore: Storage for peer addresses.
• Peer routing: Source for new peer addresses.
• Dialing decision manger? This is the special sauce that needs to know which addresses should and shouldn't be dialed (pulling them from the peer routing and peerstore subsystems).

(at least that's what I'd do)

It would also be nice to push access to the router down into the swarm while we're at it.

[h1z1]

What happens in the case of say client isolated networks? That is networks where clients cannot connect directly to each other but technically have the same networks. Very common example is a /24 broken up into /28 or /29's and most home networks being 192.168.0.0/24. Rather then trying to decern or evern assume local routing policies, can IPFS be told to listen when told a host is unreachable? Right now there's nothing to stop a host from advertising networks it's neither part of nor has any relation to it.

[rob-deutsch]

Next step would be to find the correct subnet. The only problem is that the peer dialing does not know it's observed address. InterfaceListenAddresses() can only give [/ip4/127.0.0.1/tcp/52575 /ip4/192.168.1.101/tcp/52575 /ip6/::1/tcp/52576]. Hence segregating subnet would be difficult. Any suggestion?

Does the dialling peer need to know its observed address? I think that it doesn't.

Maybe the dialling peer can just look at net.Interfaces()?

This address-choice-mechanism was predicated on "after segregation". After the segregation it will be reasonable to assume that if a remote IP is in the peerstore then our node must be listening on an local IP that we can use to reach the remote IP.

[upperwal]

Different colour arrows show shortest path to connect.

Does the dialling peer need to know its observed address? I think that it doesn't.

Yes, it is needed to identify if peer A and E are on the same network or not. Look at the diagram above. You cannot distinguish between peer A and E because they have the same private (10.X.X.X) and local (192.168.X.X) addresses but still they are on entirely different networks.

net.Interfaces() can only give you local interfaces and not the one on the public side. That can only be obtained from a router gateway which has a public IP.

After the segregation it will be reasonable to assume that if a remote IP is in the peerstore then our node must be listening on an local IP that we can use to reach the remote IP.

Can you please elaborate on this.

[upperwal]

@raulk @Stebalien totally agree. Can we use identity protocol to get our observed addresses from other peers?

So whenever peers connect and agree on protocols they exchange routable addresses of the other peer. This way we get all observed addresses which can be used later.

[rob-deutsch]

Can you please elaborate on this.

In your first post in this topic you said "maybe after segregation we can try connecting top to down like if WAN network is same" so I was commenting as the Routability() change will be made AFTER the segregation of announcements.

I now realise that the intention is to integrate this Routability() logic before the segregation of announcements.

Open question: Would it be worthwhile doing the segregation of announcements prior to adding this Routability() logic? Especially since the next steps of the Routability() integration require non-trivial development effort.