Is it possible to follow cluster mode redirects with rueidis?

[Ghilteras]

Hey guys I have an Elasticache instance in cluster mode and we are using rueidis to connect via HAProxy (because the client is in another VPC) and we get redirects to a shard from the configuration endpoint so the connection goes back to the client and then the client tries to connect directly to the shard, but it fails because HAProxy only forwards to the configuration endpoint.

Example with redis-cli:

redis-cli -h elasticache-cluster.us-east-2.aws.k8s.sendgrid.net --tls -p 6379 -a 'password' -c

> get "some key"

-> Redirected to slot [15500] located at elasticache-cluster-0001-001.6wgagi.use2.cache.amazonaws.com:6379
Could not connect to Redis at elasticahe-cluster-0001-001.6wgagi.use2.cache.amazonaws.com:6379: SSL_connect failed: Connection reset by peer

Is there a way to tell rueidis to follow the redirect? Since this is Redis TCP protocol I don't think we can instrument HAProxy to do so because it only handles raw TCP forwards. Would it make sense to use a Redis proxy instead of HAProxy here?

[rueian]

Hi @Ghilteras, rueidis does follow the redirect. The error you saw is the proof. And you are right. You can't proxy an Elasticache Cluster with just one HAProxy in total. You need one HAProxy for each Elasticache node and a private DNS resolver that solves Elasticache addresses to corresponding HAProxies or you may consider VPC peering so that you will not need to setup HAProxies.

I don't recommend Redis proxy since it is not actively maintained.

[Ghilteras]

Thanks for the answer. How about Envoy Redis plugin? https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/other_protocols/redis#arch-overview-redis

[Ghilteras]

Specifically the enable redirect option https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/redis_proxy/v3/redis_proxy.proto#envoy-v3-api-field-extensions-filters-network-redis-proxy-v3-redisproxy-connpoolsettings-enable-redirection

[rueian]

Worth trying. Remember to set ForceSingleClient to true on rueidis. https://github.com/redis/rueidis/blob/1ae39a75e5e7501abd5bd0dd419dd4cc8a3ed7f8/rueidis.go#L167

[Ghilteras]

@rueian may I ask how would that flag be useful with Envoy?

[rueian]

Hi @Ghilteras,

rueidis sends CLUSTER SLOTS to redis in order to determine which client implementation (cluster or single) to use. In your case, you can't use the cluster client implementation because you use Envoy to hide cluster details.

Setting the ForceSingleClient to true allows rueidis to skip the CLUSTER SLOTS process and just use the single client implementation to connect to your Envoy.

[rueian]

Hi @Ghilteras, does envoy work for you?

[Ghilteras]

yes it seems it works. We just need to wrap Envoy into an ingress controller (we're currently using nginx which does not have envoy, so we have envoy as sidecar of nginx which is not ideal) to expose elasticache through the ingress (we can't use the Gateway API due to our k8s version being 1.21)

[rueian]

Glad to know it works. However, I am not sure if it is possible to wrap a Redis proxy into a k8s ingress. I thought it could only expose HTTP. Exposing your Envoy with a k8s service of type=Loadbalancer may be much easier.

[Ghilteras]

ingresses normally expose L7 yes, but if you create a LB you can instruct it to passthrough any traffic (like a NLB would do with L4 traffic) which would work for TCP as well. You need a controller to secure the traffic though (in our case we don't have it) if the LB is public facing