valkey
valkey copied to clipboard
valkey-io
[NEW] Dual stack (IPv4 & IPv6)
The problem/use-case that the feature addresses
Simultaneously support clients using IPv4 and clients using IPv6.
Description of the feature
Cluster redirects such as MOVED slot ip:port currently return the destination node's TLS port or non-TLS port based on whether the client is using TLS or not. Similarily, it would need to return the IPv4 or IPv6 address depending on whether the client is connected over IPv4 or IPv6.
For this, nodes would need to exchange both address families over the cluster bus, which in turn would need a cluster bus extension.
Configs cluster-announce-ipv6 and cluster-announce-ipv4?
CLUSTER SLOTS/SHARDS/NODES would return IPv4 and IPv6 addresses of all nodes.
Alternatives you've considered
It's possible to avoid using IPs, by using hostnames instead. I'm not sure if there are any drawbacks. (Maybe some clients don't support hostnames.)
Additional information
(nope)
It's possible to avoid using IPs, by using hostnames instead. I'm not sure if there are any drawbacks. (Maybe some clients don't support hostnames.)
Traditionally, redis has refused to support hostnames in any configuration settings because of how everything is single-threaded so the gethostbyname() call blocks the entire server (and nobody really looked into more proper async dns calls).
It's not the best design, but it's what just survived over the years.
@mattsta Valkey supports hostnames. You're not up to date. :)
Valkey doesn't do dns lookups but can present hostnames to clients by configuring cluster-announce-hostname and other configs. Cluster-preferrend-endpoint-type iirc.
Oh, I see what you mean. I was thinking internal node-to-node announcements and not external client updates.
It is always a bit of a problem for a system to know how clients can connect back to it without being explicitly configured for each situation (e.g. servers have only a private IP but they are behind a router with a public address, or a server has multiple IPs so which to advertise automatically, etc).
Firstly, I believe this is meaningful. There is a scenario we once encountered: how to gradually phase out IPv4only machines and ultimately support IPv6only machines. At that time, we replaced all the business with a dual stack machine, and then gradually launched an ipv6 only machine. Then there was a malfunction in the middle due to unclean replacement.
Secondly, for common operation and maintenance platforms, node management is an important part, and we should not expect Redis clusters to solve all problems. Most companies have their own deployment management services to allocate and manage IPs. For example, using Redis Operator with k8s Regarding the selection of multiple IPs and how external customers can request access to the cluster. Perhaps it's not something we need to solve?
Good point that cluster nodes themselves may need to support multiple IPs in the cluster bus for an upgrade scenario from IPv4 to IPv6. That yet another feature.
We implemente a “announce” module in our private version to support NAT or dual stack scenario.
@kamyuentse Interesting! Do you want to describe how it works?
There is no magic. Just maintain the address mappings policy then replace the response of "cluster nodes", "moved", etc. I think the point here is: in our implementation, the mappings configuration not exchange via "clusterbus", the configuration "announce-mappings" contains all nodes mappings info.
Ah, ok. I believe the official implementation should use the cluster bus for this. We already have some extensions for cluster-announce-hostname, etc.
@zuiderkwast Maybe we can extend the Module API to enable more flexible announce policy. For example, ValkeyModule_GetAnnounce(ctx, connInfo).
The announce IPs do not need to be circulated over the cluster bus if you know them a priori and they are also static. I think in the case of ipv4 to ipv6 migrations though, we will not have this property, which leads to this question of how to distribute the new ip address dynamically throughout the cluster. The cluster bus extensions do give us the ability to circulate dynamic states like this. However, zooming out a bit, this is fundamentally the same problem raised in #58 (though with this being concerned with metadata and #58 being concerned with the user data). So there is actually a class of problems. This is why I think it makes sense to explore a more generic solution as I have a concern about the sustainability of the cluster extension approach. At some point we will hit a ceiling. Outside of the more obvious option of cluster v2, I wonder if we could create a generic module that owns up all "global" configs/states to improve the status quo for cluster v1 holistically.
The announce IPs do not need to be circulated over the cluster bus if you know them a priori and they are also static.
How would we know them a priori to be able to use the right address in MOVED redirects depending on the current client's address family?
We do distribute 'cluster-announce-ip' in the cluster bus already today, but it's just one IP of unspecified version.
I think another cluster bus extension for this would be the most logical and simple thing to do in this case (since it's a very small piece of information and similar to other things we distribute in this way).
Idea: We can allow the cluster-announce-ip config to be a space-separated list of IP addresses. ~~This is combined with a capability bit in the cluster bus MEET/PING/PONG.~~ There's already a capability bit for ping extensions. Unknown extensions are ignored (a warning is logged).
Another idea: cluster-announce-alt-ip, alternative IP address.
The first announce IP is sent in clusterMsg. All the others are sent in a ping extension only if the peer supports it. This is backward compatible.
Discussed in core team meeting today. Considering also the possibility of cluster nodes and clients seeing different IP addresses to other nodes due to NAT, we decided that we can add announce IP addresses specific for clients to use. We added "option 4" and decided on this. cluster-announce-client-ipv4 and cluster-announce-client-ipv6 are sent only to clients. They're not to be used for cluster nodes when they connect to each other.