NAT traversal

[iduartgomez]

This is a general tracking issue to monitor the status of the different techniques that will help with clients behind NAT when it comes to establishing direct connection between peers.

Prelude

One of the problems of NAT traversal is the variance in the network characteristics and configurations when trying to establish a connection between two peers, at every layer of the network and from the pov of both hw (router configuration, LAN, ISP, etc.) and software (for example, is a browser handling the connection for you or do you have direct access to the interface?). A good overview and taxonomy of the different scenarios can be read here.

Some important/interesting resources re. NAT traversal status within libp2p:

• High level overview: https://docs.libp2p.io/concepts/nat/
• Holistic specification for different hole punching techniques: https://github.com/libp2p/specs/blob/master/connections/hole-punching.md
• Overall tracking issue for NAT traversal amongst the different libp2p implementations: https://github.com/libp2p/specs/issues/312
• Project flare (decentralized hole punching): https://github.com/protocol/web3-dev-team/pull/21
• Very good practical and theoretical read on the problems posed by NAT: https://tailscale.com/blog/how-nat-traversal-works/

rust-libp2p status

Overall tracking issue in their repo: https://github.com/libp2p/rust-libp2p/issues/2052

• [ ] Automatic router configuration (e.g. UPnP). Not implemented, only an issue: https://github.com/libp2p/rust-libp2p/issues/558
• [x] STUN-like methods (hole-punching)
  • [x] Port reuse in TCP. Implemented. We should enable this and should help.
  • [x] In theory the Identify protocol (which we will be using) helps with this since it provides the observed addresses to anyone querying, so they can communicate through those addresses back (this is handled automatically by libp2p AFAIK).
  • [x] AutoNAT. Lets peers request dial-backs. (https://github.com/libp2p/rust-libp2p/pull/2262)
• [x] TURN-like methods
  • https://docs.libp2p.io/concepts/circuit-relay/
  • [x] Circuit Relay v1 (spec). Completed and we could use it in theory. The problem with TURN like strategies is one of scalability and cost though. Not a reliable long term solution probably.
  • [x] Circuit Relay v2 (spec). Under development: https://github.com/libp2p/rust-libp2p/pull/2059
  • [x] https://github.com/libp2p/rust-libp2p/pull/2438

For more detail check their tracking issue.

Notes

• Also worth mentioning is QUIC support, which we could use instead of TCP, QUIC builds on top of UDP and will make NAT traversal issues easier too. On top I believe is an straight upgrade in our case since our use case will trend towards a high number of connections but relatively short-lived and small in size on average, and here QUIC will be a good improvement over TCP, and I don't see any downsides to using it over TCP really. QUIC support within rust-libp2p seems to be finally near completion: https://github.com/libp2p/rust-libp2p/pull/2159
  • Hole punching success rate in QUIC is higher that over TCP and equivalent to that of UDP, without having to deal with all the quirks of UDP cause those are handled by QUIC (and is encrypted by default too). 80% according to (MIT paper on Hole Punching, 2006).
• Lack of concurrent attempts is a hindrance towards successful hole punching and there is an outstanding issue to address this: https://github.com/libp2p/rust-libp2p/issues/1896#issuecomment-885894496
  • Solved in https://github.com/libp2p/rust-libp2p/pull/2248
• Multistream select is important for hole punching, since both peers are attempting to initiate the connection and this is at odds to how is currently implemented (there must be one initiator). Ongoing effort to extend the protocol to allow for multiple initiators:
  • [ ] https://github.com/libp2p/rust-libp2p/pull/2066

Solutions

Working on top of UDP to add our own solutions disqualifies a big chunk of libp2p, as is not possible to use UDP directly as transport protocol in libp2p (support is under development but only in the Go impl, and is just a prototype). There may be an option, in theory, to build up our own protocol and plugin it in the libp2p (is all based around traits/interfaces so I believe is doable, but we would have to look up) and to use other components that we find useful (e.g. all the identity protocol).

However this would mean building up a significant part of the plumbing and packet handling libp2p does for you and would be a large effort so I would say this should be a last option (unless we decide we don't want to use libp2p at all). Then there is the obvious caveat: If a higher number of developers with more resources have taken a long time to tackle this and haven't succeeded (yet) why would we be faster moving? This would be a major distraction from building our core logic, so if we have to at least would be nice to have the support of other developers. Then, I would favor an other approach, that is:

• Identify what additional functionalities/methods are beneficial regarding the efforts to add NAT traversal capabilities.
• Monitor their implementation, and if necessary contribute upstream and help the libp2p folks to push them through the finish line (this would have the added benefit of gaining familiarity with libp2p internals). A bunch of the work to be done has not been done at all, but there is a lot of work which is near completion so I expect within reasonable time there will be more improvements. I believe their idea is to extend https://github.com/protocol/web3-dev-team/pull/21 (e.g. Go impl: https://github.com/libp2p/go-libp2p/issues/1039) to all implementations (including Rust) so that is what we can use as guideline and to be the end status when it comes to NAT traversal support in libp2p.
  • If necessary fork and replace/modify whatever parts we deem necessary, however that would be a last option too, ideally we want to to contribute any work upstream and have synergies with the libp2p team and benefit from their expertise.

So we can continue with the current efforts and focus on other areas which are core to our project and make a reasonable assumption that, within reasonable time this will be improved, and if necessary help upstream to improve the situation re. NAT when we run into practical problems as we test.

[iduartgomez]

For the record I am, right now at least, behind a normal (full-cone) NAT, which is the most usual case for home routers and the reason hole-punching is relatively successful apparently.

https://dh2i.com/kbs/kbs-2961448-understanding-different-nat-types-and-hole-punching/

[iduartgomez]

Other options we have been pondering is working directly with a QUIC implementation, this way we could add NAT traversal functionality ourselves and they have encryption built-in:

• the cons is that we would lose some of the libp2p building blocks and it's modular architecture; we would have to include a layer for reliably identifying and connecting peers.
• the pros is that we would have more control from the interface up while leveraging all the goods that the implementations give us (like built-in encryption). That means we could even potentially add UPnP along hole punching. We would be using one of this two crates to build on top:
  • https://github.com/quinn-rs/quinn
  • https://github.com/cloudflare/quiche

[mxinden]

:wave: Max here, maintainer of (rust-) libp2p. It is great to see interest in libp2p and great to see the many tracking issues and specifications being useful. Feel free to reach out in case you have any questions.

In theory the Identify protocol (which we will be using) helps with this since it provides the observed addresses to anyone querying, so they can communicate through those addresses back (this is handled automatically by libp2p AFAIK).

:+1: though the identify is not sufficient in all cases, as e.g. NAT mappings need to be kept alive. Long term we need to support AutoNAT in rust-libp2p.

Lack of concurrent attempts is a hindrance towards successful hole punching and there is an outstanding issue to address this: core/: Concurrent connection attempts - aka. happy eyeball libp2p/rust-libp2p#1896 (comment)

My current work item ;)

[iduartgomez]

Hi @mxinden, thanks for answering! Do you have a rough estimate on when we could see AutoNAT land? And QUIC? (Correct me if wrong but when QUIC is added it will indirectly unblock some of the NAT traversal stuff.)

Likewise, there is anything we can do to help things move re. NAT traversal (helping implementing any of the issues, even if is something easy to get started).

[mxinden]

Do you have a rough estimate on when we could see AutoNAT land?

It is hard to estimate when AutoNAT will land. Though note that AutoNAT is only a small piece of the puzzle and that basic hole punching does not depend on it. I hope for basic hole punching (https://github.com/libp2p/rust-libp2p/pull/2059, https://github.com/libp2p/rust-libp2p/pull/2076, https://github.com/libp2p/rust-libp2p/issues/1896) to be merged and released by end of this year.

And QUIC? (Correct me if wrong but when QUIC is added it will indirectly unblock some of the NAT traversal stuff.)

While not as successful as QUIC, NAT hole punching via TCP is already a great win. In other words, NAT hole punching is not blocked on QUIC support in rust-libp2p. That said, we definitely want to support QUIC, among other things for its increased NAT hole punching success rate.

Likewise, there is anything we can do to help things move re. NAT traversal (helping implementing any of the issues, even if is something easy to get started).

Help is always very much appreciated.

• At first it would be great if you could report any issues you face getting started with rust-libp2p. It's always useful to gain insights from newcomers.

• Test https://github.com/libp2p/rust-libp2p/pull/2059.

• In my eyes, implementing the AutoNAT protocol in rust-libp2p would be a good follow-up task. It is relatively self-contained. A first attempt can be found on https://github.com/libp2p/rust-libp2p/pull/1672. A draft of the protocol specification is on https://github.com/libp2p/specs/pull/362. I find https://datatracker.ietf.org/doc/html/rfc8445#section-5 to be a great read. I hope our implementation can adopt many of its concepts.

[iduartgomez]

Does not the relay protocol depend on having one server which is not behind NAT listening? In our use case this is not a reliable strategy (although could be fit somehow to cover the extra peers which are behind symmetric NAT maybe).

Help is always very much appreciated. ...

Awesome, I will report any issues I have (none so far, except for this current issue, and I like the modular nature quite a bit) and will try to help out contributing.

[mxinden]

Does not the relay protocol depend on having one server which is not behind NAT listening? In our use case this is not a reliable strategy (although could be fit somehow to cover the extra peers which are behind symmetric NAT maybe).

Yes, libp2p-circuit-relay requires the relay server to be publicly routable. Note however that there is no need to restrict a network to a single central relay server. Instead all public nodes of a network can act as such a server. With libp2p-circuit-relay-v2 the overhead for a server is small. Non public nodes can discover relay servers via e.g. the Kademlia DHT, e.g. looking for the 20 closest public relay servers to their own peer id. This allows us to do hole punching in a decentralized fashion.

I am not aware of any hole punching mechanism that does not require some TURN-like mechanism via some public nodes. E.g. WebRTC requires some mechanism to exchange SDP packets.

[github-actions[bot]]

Pivotal Tracker story: https://www.pivotaltracker.com/story/show/184207067

[PrParadoxy]

This is a great project but I am afraid the bottom to top approach for solving NAT issues wouldn't make it very fruitful in some countries in a long run.

TL; DR: I had more technical insights to add, but this post has already become too long. Nevertheless, what I want to say is that the real underlying issue of NAT is actually the naked exposure of relaying/signal/etc server to consumers, not the underlying protocol of doing so.

In a nutshell we have 2 scenarios,

1. Either both peers or at least one of them have accessible public IP address, in this case p2p is easy.
2. Both of peers are behind NAT, in which case we need some form of hole punching, relaying server, signal server, etc to make communication between peers possible. This is hard, and the very reason as to why many p2p projects has failed so far.

From here onward I will talk about second scenario. To solve this issue, the bottom to top approach (which I see here) is to study different protocol, network configuration, etc to find some ways to pass through firewall and NAT. This is great and all, but only for less restrictive countries, where we have no whitelist approach for censorship. In these countries however, the very need of p2p is not usually felt by people. From the perspective of developers, developing p2p services is not commercially (usually) justified and why should one learn new frameworks in the first place?, and from the view point of consumers, whatever works, works, why seek alternatives? If they want privacy, well of course they can use any form of hidden services, which is not censored (Tor, for example), so it is a bit hard to convince people to use p2p in the first place, and let's be honest, p2p works the best when there are a lot of people and developers (or better said, content) in it.

At the moment p2p is needed the most in middle east (Iran, to be more specific), China, and other similar restrictive countries. I am sure any "working" p2p project would immediately attract generate a lot of attention, because it solves an underlying issue that has been going on for more than a decade or two (please just look at projects like Xray-core, V2fly, etc to see this need of censor free internet, in these countries).

Let me become more specific, in a country like Iran, the censorship itself is based on protocol. UDP is completely blocked (with a few exception) so let's forget about QUIC and anything similar to it. As for TCP, only TCP connection that is encrypted by SSL (with some specific fingerprints, at that!) can pass through firewall. Literally nothing else, not even obfuscated TCP goes unnoticed. Blocking SSL would come at a huge cost for government, so it is not going to happen. In a bottom to top approach that is followed here (and similar projects), one would develop N different ways to pass through a non-restrictive firewall, and the project would then become mature, so mature that developing a new layer on top of these ways to mimic SSL behaviour in p2p becomes way too hard (I tried to make old freenet to make use of a TLS proxy, with no success). And even somehow one manages to do that, it is only matter of time until government blocks every known relay/signal/stun/etc server. So this project would end up working only on some less restrictive countries, just like old Freenet (hyphanet), whereas it could possibly gain much more love that it actually deserves. to where it actually belongs.

In a top to bottom approach, one would ask themselves:

1. Who is going to be our consumers? Why would they need this product?
2. Can we make some features commercialized and make a revenue while attracting new developers for faster development?
3. What is the underlying issue? Can we maintain the product, when there is a larger adversary (e.g. government) at work and being open source at the same time?
4. ...

From what I saw in "How we will decentralize the Internet with Freenet" talk, I got the feeling that perhaps I could give a different perspective in some possible, interesting uses of this project, but I don't know if anyone can share my perspective, so I stop here.

[alexlyee]

@PrParadoxy Your comment was very thoughtful and well said. I agree that it is an extremely important thing to try and get right to make p2p available for those without the privilege of a non-NAT'ed connection to the internet

[iduartgomez]

I am closing the issue and will follow up on pivotal tracker: ##5207398.The scope of this task was/is rather limited to implementing NAT traversal over our freenet protocol. Since we have our own DHT implementation, connection protocol etc. seems the more sensible approach for us will be creating our own NAT hole punching procedure integrated with our connection protocol, we will leverage libp2p as much as possible for

Re. all things p2p communication, bypassing goverment controls, etc. feel free to open new discussions to discuss them, seems like valuable input. For now we need to start somewhere and get the network running so will focus on making connectivity possible working around NAT issues as much as possible, in the future we can improve all things when it comes to establishing connectivity between peers with whatever approaches are available (this may also mean more departures with libp2p so in that regard is good we start decoupling both things as much as possible).