IPv8 v3.0, IPv6 support and network interface refactoring

[qstokkink]

2021 update: This issue is almost resolved. The base Community class can now handle IPv6. The TunnelCommunity is the only component left in IPv8 that is required to be ported.

---

Old O.P.

Title is slightly suggestive: only one change is needed for all three

Right now, Peer objects have a most up to date address (.address). In the future we will want to support multiple network interfaces (requiring multiple sockets, which probably need to be separately punctured).

The biggest change to our current codebase will be the refactoring of Peer objects to support a set/list of interface specifications. The Peer object will have to be in charge of implementing a send() and puncture() method (opening at least one communication channel), this is currently handled by the Community class.

Pros:

• Overlay programmers have an opaque identity messaging layer, while still having control over the byte format of each message. No more worrying about addresses.
• It just works, all of the network magic is in the IPv8 core.
• More efficient: right now we just register any IPv4 we find as walkable and hope there is some user on the other side. If we bundle the possible IPv4 addresses under one key, one address working excludes the need of checking the other addresses.
• We can store historical addresses per peer and attempt to reconnect to the last known interface.

Cons:

• Need to open a socket for each interface (IPv4, IPv6, NFC?, ..)
• ~~Breaking change to all existing code.~~ EDIT: No breaking change was required
• Even though you really shouldn't need it in the first place, IPv4 addresses can no longer be seen as semi-stable identifiers.
• A lot of work.

[qstokkink]

I think the road map should be:

1. Refactor existing code to use Peer.send (and internally update the endpoint accordingly).
2. Create a new version of Community logic, that allows for multiple interfaces. This second step is a hard wire-protocol break for any overlays that use it - as the IntroductionRequest and IntroductionResponse are tied to IPv4. Some Communities will inevitably be left behind, like the DiscoveryCommunity that we have kept going for so long.

[qstokkink]

After first attempt today, priority is as follows:

• [ ] Don't use the IPv4 address in the DHT community.

[qstokkink]

Once we have this abstraction layer, it would probably make sense to use Scapy.

It supports many of the interfaces we would like to use, for example:

• IPv6
• DHCP
• Bluetooth
• TFTP
• USB
• ..and much more.

This should also solve our complex NAT local service discovery (#703).

[qstokkink]

I have changed my mind. After failing to implement this (starting over 3 times) and talking with @egbertbouman a litle while ago, I believe I have a more natural fit for IPv8 that doesn't break everything.

In short:

• Each Overlay maintains a single network interface "endpoint" as before, which can send and receive information as before. No changes here.
• This endpoint has multiple implementations (IPv4, IPv6, Bluetooth, etc.) which it opaquely sends information over. This is already partially the way it works with the UDPEndpoint class.

Nitty gritty details:

• The send() method of Endpoint get a interface=InterfaceEnum.interface option, for the case of a particular interface being desired. For example: endpoint.send(some_address, some_data, interface=InterfaceEnum.IPv4). In case it's not given, the interface is interpreted from the address.
• A Peer gets a dictionary of addresses instead of just one address. For backwards compatibility we can have the .address be a property that points to Peer.addresses[InterfaceEnum.IPv4]. This would be the most impactful change for overlay designers, as the IPv4 address may not be available in the future.
• The Community will have to puncture each endpoint implementation individually. We already do it like this.

[qstokkink]

Here is the class diagram (using n instead of *) of the current state of affairs and the newly proposed design (see my previous post).

Current

Proposed

[qstokkink]

With #830, the first (easy) step is complete: we can now send and receive IPv6 interface packets.

Remaining:

• Refactor the Peer.address so a Peer can have one address per interface (also includes limited changes to Network).
• Refactor existing communities so they can handle IPv6 addresses. This includes both business and puncturing logic.
• Add and support a configuration directive for interface loading.

[qstokkink]

I believe we can maintain full backwards compatibilty by making sure all introduction requests and responses using IPv4 still use the old messages. For all other interfaces we will define a new introduction request and response message that allow for generic (str, int) tuples as addresses, reserving message ids 233 and 234. Hereby, old versions should never receive new-style introduction requests and responses.

We can have both types (new and old) of introduction requests and response messages call their respective introduction_request_callback or introduction_response_callback callbacks, leading to an opaque design for overlay programmers.

Edit: pucturing will also have to change.

[qstokkink]

Naively implementing these new messages turns the community.py code into a complete mess. I'll take some time to (hopefully) figure out a good way to combine the old and the new logic.

Some positive news is that, due to the Payload abstraction, most of the logic can stay intact. By adding a new serializer format we can have drop-in support for the new payload types ((varlenH, H) instead of (4s, H)).

[qstokkink]

Update on the current best plan to make sure the new-style introduction messages don't spaghettify the community logic:

Both the IntroductionRequest and IntroductionResponse have unused bits (in the advice and connection) for future use. For both messages I will claim two of these bits:

• Bit 1: "I also support new-style introduction logic"
• Bit 2 (IntroductionResponse): "The introduced peer supports new-style logic"

Naturally, receiving any new-style introduction logic from a peer automatically flags it as supporting new style logic.

This turns most of the connection management that would've had to reside in the community.py into a single flag on Peer objects: new_style_intro.

[qstokkink]

With #913 merged we now move into the final phase: porting the DHTCommunity and the TunnelCommunity.

With that, after an entire year of development, my involvement with this issue is also over.

[qstokkink]

Once the last component has been ported, the configuration should also support specifying the address and port of the IPv6 interface.

https://github.com/Tribler/py-ipv8/blob/76da077fef59225cfd12937880ff8f7a85e60b6a/ipv8_service.py#L79-L80

In turn, the documentation should also be updated:

https://github.com/Tribler/py-ipv8/blob/master/doc/reference/configuration.rst

[Solomon1732]

What is preventing from doing the switch? Asking because if what is left isn't too big of a task then I might be interested in tackling it.

[qstokkink]

@Solomon1732 The only Community left to port is the anonymization Community (the TunnelCommunity). We have assigned this to its original author @egbertbouman as it is quite some complex logic/cryptography that is deeply rooted in IPv4.

If you don't need the anonymization, you can disable it and locally revert this commit and see IPv8 automatically hand off to IPv6.

[Destroyinator69420]

Not to forget more closely implementing the V3 hidden services protocol.

[qstokkink]

[Memo] The test suite can be run using IPv6 fake addresses by changing the following code from "UDPv4Address" to "UDPv6Address" .

https://github.com/Tribler/py-ipv8/blob/fcf2c7971a9b3d91c4dfb159f6e17f68c081f1d0/ipv8/test/mocking/endpoint.py#L74-L76