Networking via UDP Hole Punching

[katywings]

First and foremost thank you for this awesome open source project, its really nice to see other people also trying to break out from the bigtech services!

In the README you mention the requirement for DNS and SSL. Do you know about UDP Hole Punching and have you considered, adding it as an alternative node2node communication method? E.g. via udp hole punching a databag node could be run on a normal consumer pc, without a static port :).

[balzack]

Firstly, thank you for the compliment! Also, UDP Hole Punching is new to me. I looked at the link & wiki, but I struggle to get my head around it; would it apply to the communication from a mobile phone to the server node? Avoiding a static port would be a huge improvement.

[katywings]

would it apply to the communication from a mobile phone to the server node?

Hole punching is just a way to establish direct udp communication between two devices, usually for voip, realtime chats and stuff like that. It could be used for communication between databag nodes or it could also be used between the mobile phone app and node.

Essentially it works like this:

1. Normally a device A behind a router cannot receive data from another device B outside of the router network
2. But if the device A first contacts a known device C outside of the router network, e.g. over the internet, and if device C answers successfully, the router will assign a dynamic port mapping to device A, which stays alive as long as communication is happening on that port
3. Now device C can tell devices A and B their respective public ips and dynamic ports
4. From that point forward, device A and B can talk directly over the dynamically assigned ports. They have to keep contacting each other so that the dynamic port stays open.
5. Important is that device C is only needed to punch the hole on the router. After the communication between A and B is established, device C is not involved any longer. Device C is not a relay!

In this example device C acts as a "STUN"-server.

Architecture:

• Each databag node can theoretically take on the role as a STUN server (and there can be multiple STUN servers)
• A STUN server has the job to tell other databag nodes their own public ip and dynamic ports
• Usually such STUN servers have a static ip which is known to other databag nodes
• If a databag node wants to figure out its ip and dynamic port, it just has to contact another known node acting as STUN.

Cons:

• Not all routers / network types are supported. Some routers just don't keep the dynamic ports open. And some network admins disable the NAT functionality because they don't like holes in their infrastructure :D.
• Firewalls might block the UDP communication (especially in corporate networks)
• UDP is very rudimentary and communication is not magically encrypted, you have to create an own little SSL layer over UDP.
• Totalitarian regimes like China don't like, if their citizens can talk privately and have implemented nation-wide firewalls trying to block STUN. Companies like Apple therefore also might not like it, if your Mobile App would use STUN 😅.

More about STUN Example nodejs implementation: https://npm.io/package/stun

[balzack]

Thank you for the thorough explanation! I am starting to plan the audio/video calling feature, and was looking at STUN/TURN servers for that purpose and maybe included with webrtc (?). In this implementation however the databag node would act as the STUN server for two client apps. If I understand correctly, with the architecture you suggest, I would need to host a general STUN server and act as a registry for the nodes. A service being built with this would have significant benefits, allowing people to buy a device, and have it plug in an work without configuring ports and certs, but also have the disadvantage of centralizing some of the communication. While the effort to build a custom reliable and secure communication mechanism over UDP is a bit out of my league, I can see that being a good option for a consumer product. Based on your cons list, there would need to be fallback mechanisms for communication. Perhaps I misunderstood key aspects, but from what I understand it seems really interesting and I will keep it in mind as this project matures.

[katywings]

If I understand correctly, with the architecture you suggest, I would need to host a general STUN server and act as a registry for the nodes.

Yes and no: I think that every databag node should have the feature-set to act as a STUN server and registry. Initially most nodes will use your "official" STUN nodes, but as the community size increases and there are many STUN servers from the community, a node can use any of the available STUN servers / registries. You also could make it, so that on first databag boot, a random STUN server from a publicly rotating list is used.

Based on your cons list, there would need to be fallback mechanisms for communication.

Exactly :+1: . Communication over UDP is not a full replacement, but an addition/alternative to a static ip/port setup.

Perhaps I misunderstood key aspects, but from what I understand it seems really interesting and I will keep it in mind as this project matures.

Nice! I am looking forward how your project matures and I hope you can grow a dedicated community from it!