lighthouse
lighthouse copied to clipboard
sigp
IPv6 IPv4 dual compatibility
Description
Care has not been taken to handle computers with IPv4 and IPv6 dual stacks. Preferences have been made throughout Lighthouse and the underlying network protocols to favour IPv4. Examples include CLI args setting up the HTTP API, ENR socket addresses, Libp2p listening address (only supports one address and IPv6 "::" currently doesn't function).
We should update Lighthouse and sub-protocols to preference IPv6 if supported and fallback to IPv4 if not.
prysm is behaving like that, it makes it very hard to find the culprit why port forwardings are not working. only a netstat give the user a hint to disable IPv6 via GRUB to get it working... you really want to make those ipv6 hell at the lighthouse? Which benefit?
It should be parameterized if implemented...
Firstly, love the name and github profile pic!
I guess preferencing IPv6 by default isn't ideal as you've pointed out. I think having the support and checking all protocols have the correct support will be useful and then preferencing via a CLI param, such as --ipv6.
Automatic preference of IPv6 may not have been ideal 15+ years ago, but today is not 15+ years ago. :wink: IPv6 is (quite fortunately) preferred on most systems by default.
(IMHO, instead of an --ipv6 flag, it might be better to require extra flags for sticking with IPv4, e.g. --my-network-is-misconfigured.)
Side-note: “Wildcard” sockets (bound to ::) work perfectly fine with all sorts of IPv4 port forwarding (DNAT, SNAT, etc.). They accept both IPv6 and IPv4 connections. (More details are in man 7 ipv6, for example.)
That said, APIs around getaddrinfo() and BSD sockets (intentionally) make dual-stack support a simple and effortless default. By contrast, exclusion of IPv6 requires extra effort, in addition to being unkind to legacy-free, IPv6-only networks.
For the record, there is a very similar problem in openethereum: One can set IPv6 addresses for certain services (while others stubbornly reject them), but even with IPv6 addresses set there seem to be no IPv6 connections.
Sadly, such^^^ bugs are quite hard to fix, because they are often embedded deep in the underlying libraries and not directly in lighthouse or openethereum. Which I discovered the hard way when attempting to run eth1 and eth2 on different machines on my IPv6-only network.
These are (some of) the code locations in lighthouse that explicitly require IPv4. I’m struggling to understand why: Is anything IPv4-related hardcoded into the Ethereum protocol(s), making IPv4 addresses a necessity? Many (most?) IPv6 address parsers can parse IPv4 addresses in a backward-compatible way while also supporting native IPv6.
The main issue is discv5 and libp2p.
Last i checked libp2p had full support but there was an issue with binding to ::.
As for discovery, we have things called ENR records, which store key value of ip4/ip6 ports on which to connect to. It needs to be updated to handle the cases where nodes are on different ip versions and make decisions about how we are going to connect.
Doing this update is on our to-do list.
I'd like to bump this issue, ipv6 is not the future any more, it's the present. Not having dual stack support excludes a large user base (those behind carrier grade NAT) from inbound connections, and thereby makes the peering process unnecessarily inefficient.
Are there upstream tickets for the mentioned issues in libp2p? Can you maybe link them to improve awareness?
@torfbolt I fully agree with you. This feature is work in progress, you can see the progress there:
https://github.com/sigp/lighthouse/issues/2023 https://github.com/sigp/discv5/pull/107
I have done a lot of testing with different Ethereum clients, the IPv6 support is pretty bad. No client seems to be capable to run within IPv6-only networks by default. I'm currently doing Gitcoin bounties and hope to find developers who are willing to change this. (https://gitcoin.co/issue/sigp/lighthouse/2023/100028713)
I also found a developer who wants to add full IPv6 support to Nimbus, we are currently waiting for a response from the Nimbus team. I think getting IPv6 support within the whole (infrastructure) ecosystem will take multiple years.
Yes. @divagant-martian is currently working on this. The discovery side (arguably) the hardest part is mostly complete and under-review.
The support is non-trivial also in p2p networks. If the majority of nodes use ipv4 (currently the case) and can only contact ipv4 nodes, ipv6-only peers could get segregated and fall victim to various attacks such as eclipses. I guess in a p2p setting, its not as simple as a client/server model where you can just enable ipv6 on the server and all the clients. But we are getting there and will need to do a bit of testing.
For those interested in this feature and those watching #2023
Update: @divagant-martian has completed most of the work required to support this. There are still a few smaller things we need to cover, but for the reasons I'll list below, we do not plan to support this feature until after the merge (the upcoming hard-fork merging the current Ethereum proof of work chain with the beacon chain).
There are caveats with IPv6 and dual-stack support:
- We will support IPv6 sockets with mapped addresses - This allows for IPv6-only nodes (to be discussed below) and dual-stack nodes by supporting both IPv4 and IPv6 packets through a single socket (which most modern OS should support).
- All consensus clients use a discovery mechanism called Discv5. Identities on this network are encapsulated as ENRs (Ethereum Node Records) which have the ability to store ipv4 and ipv6 sockets. This protocol has a built-in mechanism that allows us to update our local ENR with our external contactable socket based on what other nodes on the network see our source packets as. So by default, we do not advertise an IPv4 OR an IPv6 address and only after other nodes tell us what they see our external address is, do we update our local ENR. This means, if the current network is dominated by ipv4 nodes, a dual stack node will communicate by ipv4 and subsequently we will only advertise ipv4 in our ENR and ipv6 will never be used. This will only change when more and more users start advertising ipv6, which we may need to bootstrap manually, by manually setting ipv6 in our ENRs. We can aid with this by adjusting the ethereum consensus bootnodes to advertise and support ipv6 addresses.
- IPv6-only nodes - These technically can be supported. But as they cannot connect to ipv4 nodes (which is predominantly the current network) we may see partitions in the network between IPv4 and IPv6-only networks.
Because the merge is our current priority and extensive tests have already been done ahead of the mainnet hard fork, we don't want to add any significant changes to the network and possible partitioning before this critical junction. The changes to bootnodes and new code that remains to tested at scale will therefore be released after a successful merge and healthy network.
Really looking forward to this, and sad I won't be able to test it out before the merge. My nodes will certainly be advertising ipv6.
Because the merge is our current priority and extensive tests have already been done ahead of the mainnet hard fork, we don't want to add any significant changes to the network and possible partitioning before this critical junction. The changes to bootnodes and new code that remains to tested at scale will therefore be released after a successful merge and healthy network.
A short update about the current status would be great. 🙂
Hey. Yep. So pretty much all of our code is ready. I'll update Lighthouse's bootnodes and have a chat with other client teams to see what level of support they have. Once our bootnodes are updated and (hopefully a few other clients) we can start onboarding a bunch of ipv6/ipv4 nodes onto the network. We'll start with the testnets before moving to mainnet. We'll try to include as many clients as possible, as this is a network change, and should be driven by a few clients in unison.
This has been officially completed with discv5, lighthouse and our mainnet bootnodes updated. Ipv6 should be supported on the network now.