Elasticache redis in cluster mode Could not execute Redis pipeline: MOVED

[huzlak]

Gloo Edge Version

1.11.x (latest stable)

Kubernetes Version

1.22.x

Describe the bug

When using elasticache redis in cluster mode as ratelimit backend, the ratelimiting does not work and errors are logged.

Steps to reproduce the bug

1. deploy EKS cluster
2. create elasticache redis in cluster mode without encryption and authentication
3. Deploy gloo and point it to the previously created redis

redis:
  disabled: true
  service:
    name: test-cluster-mode.ap9uwo.clustercfg.euc1.cache.amazonaws.com
global:
    glooRedis:
      enableAcl: false

4. deploy a service and apply a ratelimit configuration on it
5. query virtualservice more times than ratelimit specified and you still get 200 and error message are logged

Expected Behavior

I expect to get 429 after reaching the limit and no error message logged.

Additional Context

This happened after customer's upgrade from 1.10.0 to 1.11.20. In 1.10.0 similar error messages are logged, but the ratelimit is applied properly. Attaching logs from both tested versions for comparison

1-10-cluster-mode-ratelimit.log

1-11-20-cluster-mode-ratelimit.log

[elcasteel]

Seems very similar to this: https://github.com/solo-io/gloo/issues/6436

[ianmacclancy]

Attaching "how-to" docs for any future issues to reference - https://www.solo.io/blog/from-zero-to-gloo-edge-in-15-minutes-eks-edition/

[ianmacclancy]

This is an issue with the way redis handles clusters and the redis client connects to the clusters - we need to catch the moved error and re-attempt the query to the new location in the error

https://redis.io/docs/reference/cluster-spec/#moved-redirection

[ianmacclancy]

In order to access an ElastiCache Redis cluster from outside of a VPC you will need to set up a VPN with a VPN client endpoint through the VPC that the cluster is in as well as security groups that allow this connection

Documents here - https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/accessing-elasticache.html#authorize-access-vpc

Gotcha that I encountered - you will have to chmod the downloaded ovpn client config file to properly edit it. It is also easier to add the key and certificate directly to the ovpn file rather than rely on path. Docs for this here - https://www.brainfart.sg/index.php/2012/05/embedding-certificate-into-openvpn-config/