charts icon indicating copy to clipboard operation
charts copied to clipboard

[bitnami/redis] Redis Sentinel 8 - External DNS - External Access - AWS EKS Setup

Open samuelexferri opened this issue 7 months ago • 3 comments

Name and Version

bitnami/redis:8.0.2

What architecture are you using?

amd64

What steps will reproduce the bug?

Hi, I'm facing an issue with setting up Redis 8 Sentinel on Kubernetes (AWS EKS). I want to expose Redis Sentinel externally using ExternalDNS, which is already configured and working for other services in the cluster.

I have this config:

architecture: replication

global:
  security:
    allowInsecureImages: false

image:
  registry: docker.io
  repository: bitnami/redis
  tag: 8.0.2 # !: Version

auth:
  enabled: true
  sentinel: true
  password: '123456'
  acl:
    enabled: false
    users: []

replica:
  replicaCount: 3
  extraFlags:
    - '--protected-mode no'
  resources:
    requests:
      cpu: '250m'
      memory: '512Mi'
    limits:
      cpu: '1'
      memory: '2Gi'
  podAffinityPreset: ''
  podAntiAffinityPreset: 'hard' # !: Set to hard or soft
  persistence:
    enabled: true
    size: 25Gi
    storageClass: auto-ebs-sc
  automountServiceAccountToken: true

sentinel:
  enabled: true
  quorum: 2
  persistence:
    enabled: true
    size: 25Gi
    storageClass: auto-ebs-sc
  masterSet: 'mymaster'
  masterService:
    enabled: true
    type: ClusterIP
    ports:
      redis: 6379
  externalAccess:
    enabled: false # TODO: ENABLING THIS WILL BREAK
    service:
      # disableLoadBalancerIP: true # Exist in code not in docs
      type: LoadBalancer
      redisPort: 6379
      sentinelPort: 26379
      annotations:
        external-dns.alpha.kubernetes.io/hostname: redis-bitnami-test.redis8.MYSITE.COM
        service.beta.kubernetes.io/aws-load-balancer-type: 'external'
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: 'ip'
        service.beta.kubernetes.io/aws-load-balancer-scheme: 'internet-facing'

rbac:
  create: true

serviceAccount:
  create: true
  name: 'redis-service-account'
  automountServiceAccountToken: true

metrics:
  enabled: false

useExternalDNS:
  enabled: true
  suffix: redis8.MYSITE.COM
  additionalAnnotations:
    ttl: 10

When I enable sentinel.externalAccess.enabled: true, the first node fails to deploy. Here's the relevant log output:

 13:32:42.85 INFO  ==> about to run the command: REDISCLI_AUTH=$REDIS_PASSWORD  timeout 90 redis-cli -h redis-bitnami.MYNAMESPACEsvc.cluster.local -p 26379 sentinel get-master-addr-by-name mymaster
Could not connect to Redis at redis-bitnami.MYNAMESPACEsvc.cluster.local:26379: Connection refused
Could not connect to Redis at redis-bitnami.MYNAMESPACEsvc.cluster.local:26379: Connection refused
 13:32:52.88 INFO  ==> Configuring the node as master
1:C 28 May 2025 13:32:52.901 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 28 May 2025 13:32:52.901 * Redis version=8.0.2, bits=64, commit=00000000, modified=1, pid=1, just started
1:C 28 May 2025 13:32:52.901 * Configuration loaded
1:M 28 May 2025 13:32:52.901 * monotonic clock: POSIX clock_gettime
1:M 28 May 2025 13:32:52.902 * Running mode=standalone, port=6379.
1:M 28 May 2025 13:32:52.903 * Server initialized
1:M 28 May 2025 13:32:52.906 * Creating AOF base file appendonly.aof.1.base.rdb on server start
1:M 28 May 2025 13:32:52.909 * Creating AOF incr file appendonly.aof.1.incr.aof on server start
1:M 28 May 2025 13:32:52.909 * Ready to accept connections tcp
new master elected, updating label(s)...
pod/redis-bitnami-node-0 labeled
 13:32:54.06 INFO  ==> Found previous master  in /opt/bitnami/redis-sentinel/etc/sentinel.conf
 13:32:54.07 INFO  ==> about to run the command: REDISCLI_AUTH=$REDIS_PASSWORD timeout 90 redis-cli -h redis-bitnami.MYNAMESPACEsvc.cluster.local -p 26379 sentinel get-master-addr-by-name mymaster
Could not connect to Redis at redis-bitnami.MYNAMESPACEsvc.cluster.local:26379: Connection refused
pod/redis-bitnami-node-0 unlabeled
rm: cannot remove '/etc/shared/previous': No such file or directory
Could not connect to Redis at redis-bitnami.MYNAMESPACEsvc.cluster.local:26379: Connection refused
 13:33:04.10 ERROR ==> Redis master host is configured incorrectly (host: , port: 6379)

It looks like Redis starts in standalone mode (instead sentinel), but Sentinel is unable to elect or discover a master. Eventually, it fails with a configuration error related to an undefined master.

This issue seems to be related to the following open issues on the Bitnami GitHub repository:

  • https://github.com/bitnami/charts/issues/32479
  • https://github.com/bitnami/charts/issues/33270
  • https://github.com/bitnami/charts/issues/33828

Could someone confirm if there's a working configuration that allows Redis Sentinel with external access using ExternalDNS on AWS? If I’m missing something or misconfigured a value, I’d appreciate any help.

Thanks in advance!

samuelexferri avatar May 28 '25 13:05 samuelexferri

Hi!

In the case of external access, it will create a service per pod. Would this work correctly with external DNS? Note that all the created service would have the same annotations

javsalgar avatar May 29 '25 06:05 javsalgar

Hi, thank you for your response.

With my previous configuration, enabling externalAccess.enabled: true caused the installation to fail (see the error on the first node during setup).

However, I managed to expose only the master service node without using externalAccess, by configuring it like this:

architecture: replication

global:
  security:
    allowInsecureImages: false

image:
  registry: docker.io
  repository: bitnami/redis
  tag: 8.0.2 # !: Version (Update also Bitnami in GitHub Actions)

auth:
  enabled: true
  sentinel: true
  password: '123456' # TODO: K8S Secret

replica:
  replicaCount: 3
  extraFlags:
    - '--protected-mode no' # TODO: Remove?
  resources:
    requests:
      cpu: '250m'
      memory: '512Mi'
    limits:
      cpu: '1'
      memory: '2Gi'
  podAffinityPreset: ''
  podAntiAffinityPreset: 'hard' # !: Set to hard or soft
  persistence:
    enabled: true
    size: 30Gi
    storageClass: auto-ebs-sc
  automountServiceAccountToken: true

sentinel:
  enabled: true
  replicaCount: 3
  extraFlags:
    - '--protected-mode no' # TODO: Remove?
  quorum: 2
  persistence:
    enabled: true
    size: 25Gi
    storageClass: auto-ebs-sc
  masterSet: 'mymaster'
  masterService:
    enabled: true
    type: LoadBalancer
    annotations:
      external-dns.alpha.kubernetes.io/hostname: redis-bitnami-master.redis8.MYSITE.com # TODO: redis-bitnami-master.<BASE-DOMAIN>
      service.beta.kubernetes.io/aws-load-balancer-type: 'external'
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: 'ip'
      service.beta.kubernetes.io/aws-load-balancer-scheme: 'internet-facing'

rbac:
  create: true

serviceAccount:
  create: true
  name: 'redis-service-account'
  automountServiceAccountToken: true

metrics:
  enabled: false

useExternalDNS:
  enabled: true
  suffix: redis-bitnami.MYSITE.com # TODO: redis-bitnami.<BASE-DOMAIN>
  additionalAnnotations:
    ttl: 10

Now, I’d like to achieve two things:

Expose all Sentinel nodes externally I want to access Redis using Sentinel mode (e.g., in Redis Insight from my local machine), not just through standalone access to the master node. How can I properly expose all Sentinel instances?

Redis 8.0.2 and JSON support I'm using the latest Redis version (8.0.2), but JSON support doesn’t seem to work in Redis Insight or other clients I tested. Why is the ReJSON module not enabled by default? How can I enable Redis JSON support in the Bitnami Helm chart?

Thanks in advance for your detailed help!

samuelexferri avatar May 29 '25 09:05 samuelexferri

Any updates on this? I see many open issues regarding Redis

samuelexferri avatar Jun 16 '25 08:06 samuelexferri

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] avatar Jul 02 '25 01:07 github-actions[bot]

Hi @samuelexferri,

Sorry for the delay here. I'll review the information and will update this ticket as soon as I have more information.

jotamartos avatar Jul 03 '25 07:07 jotamartos

Hi @samuelexferri,

I just reproduced the error without using AWS so this is not related to the cluster. Will continue reviewing it

Could not connect to Redis at redis-bitnami.MYNAMESPACEsvc.cluster.local:26379: Connection refused
 13:33:04.10 ERROR ==> Redis master host is configured incorrectly (host: , port: 6379)

If I disable externalAccess, the deployment works as expected

➜  redis git:(main) k get pods 
NAME                            READY   STATUS    RESTARTS   AGE
external-dns-58ccd89d68-4wz6m   1/1     Running   0          35m
redis-node-0                    2/2     Running   0          3m21s
redis-node-1                    2/2     Running   0          2m45s
redis-node-2                    2/2     Running   0          2m1s
➜  redis git:(main) k logs -f redis-node-0 -c sentinel
...
1:X 03 Jul 2025 09:41:10.098 * Sentinel ID is 2a09ba7abbb41ee71e79087310d75f9809c3c815
1:X 03 Jul 2025 09:41:10.098 # +monitor master mymaster redis-node-0.redis.jota.com 6379 quorum 2
1:X 03 Jul 2025 09:42:10.143 # +sdown master mymaster redis-node-0.redis.jota.com 6379

jotamartos avatar Jul 03 '25 09:07 jotamartos

Hi @jotamartos, thanks a lot for reproducing the issue!

I believe the root of the problem lies in the sentinel.externalAccess configuration. Even if you're able to work around the startup failure, could you please also verify whether ExternalDNS correctly picks up and applies the annotations for the Sentinel services?

Thanks again for your support!

samuelexferri avatar Jul 03 '25 09:07 samuelexferri

I'm reviewing this new PR and linked it with this issue. Can you take a look and confirm it works for you too?

https://github.com/bitnami/charts/pull/34806

jotamartos avatar Jul 16 '25 10:07 jotamartos

Hi, I tried using Redis version 8.0.3 with the Bitnami Helm chart version 21.2.12, and I was able to expose only the master service via ExternalDNS in AWS EKS, simply by using annotations. However, I haven’t been able to find a way to expose the entire Sentinel service.

helm upgrade --install redis-bitnami redis --atomic --repo https://charts.bitnami.com/bitnami --version 21.2.12 --namespace redis-8 --values ./k8s/redis-bitnami.yaml --create-namespace

My current values.yaml:

architecture: replication

global:
  security:
    allowInsecureImages: false

image:
  registry: docker.io
  repository: bitnami/redis
  tag: 8.0.3 # !: Version (Update also Bitnami in GitHub Actions)

commonConfiguration: |
  # loadmodule /opt/bitnami/redis/lib/redis/modules/redisbloom.so
  loadmodule /opt/bitnami/redis/lib/redis/modules/redisearch.so
  loadmodule /opt/bitnami/redis/lib/redis/modules/rejson.so
  # loadmodule /opt/bitnami/redis/lib/redis/modules/redistimeseries.so

auth:
  enabled: true
  sentinel: true
  password: '123456'

# See redis-bitnami-crt in Secrets to copy CA and TLS
tls:
  enabled: true
  autoGenerated: true
  authClients: false # Avoid TLS Client Authentication (Two Way TLS)

replica:
  replicaCount: 3
  resources:
    requests:
      cpu: '250m'
      memory: '512Mi'
    limits:
      cpu: '1'
      memory: '2Gi'
  podAffinityPreset: ''
  podAntiAffinityPreset: 'hard' # !: Set to hard or soft
  persistence:
    enabled: true
    size: 30Gi
    storageClass: auto-ebs-sc
  automountServiceAccountToken: true

sentinel:
  enabled: true
  replicaCount: 3
  quorum: 2
  persistence:
    enabled: true
    size: 25Gi
    storageClass: auto-ebs-sc
  masterSet: 'mymaster'
  masterService:
    enabled: true
    type: LoadBalancer
    annotations:
      external-dns.alpha.kubernetes.io/hostname: redis-bitnami-master.MYSITE.com
      service.beta.kubernetes.io/aws-load-balancer-type: 'external'
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: 'ip'
      service.beta.kubernetes.io/aws-load-balancer-scheme: 'internet-facing'
  externalAccess:
    enabled: true
    service:
      type: LoadBalancer
      disableLoadBalancerIP: false # TODO
      annotations:
        external-dns.alpha.kubernetes.io/hostname: redis-bitnami-sentinel.MYSITE.com
        service.beta.kubernetes.io/aws-load-balancer-type: 'external'
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: 'ip'
        service.beta.kubernetes.io/aws-load-balancer-scheme: 'internet-facing'

rbac:
  create: true

serviceAccount:
  create: true
  name: 'redis-service-account'
  automountServiceAccountToken: true

metrics:
  enabled: false

useExternalDNS:
  enabled: true
  suffix: redis-bitnami.MYSITE.com
  additionalAnnotations:
    ttl: 10

Even when I tried exposing Sentinel without using externalAccess, it didn’t work as expected.

Also i noticed that in my Route53 records — the master record points to the LoadBalancer, but the headless service (redis-bitnami.redis-bitnami.MYSITE.com) and the nodes (redis-bitnami-node-0.redis-bitnami.redis-bitnami.MYSITE.com) record points to the ClusterIP.

Regarding the PR, I ran in the same issue as you. I deployed it locally with the unkls's modifications, but I encountered the following problem:

1:S 17 Jul 2025 09:07:52.573 * <search> Disabled workers threadpool of size 4
1:S 17 Jul 2025 09:07:52.573 * <search> Loading event ends
1:S 17 Jul 2025 09:07:52.573 * DB loaded from disk: 0.001 seconds
1:S 17 Jul 2025 09:07:52.573 * Ready to accept connections tls
1:S 17 Jul 2025 09:07:52.573 * Connecting to MASTER :0
1:S 17 Jul 2025 09:07:52.573 # Unable to connect to MASTER: (null)
1:S 17 Jul 2025 09:07:53.580 * Connecting to MASTER :0
1:S 17 Jul 2025 09:07:53.580 # Unable to connect to MASTER: (null)
 09:07:54.23 INFO  ==> Found previous master  in /opt/bitnami/redis-sentinel/etc/sentinel.conf
 09:07:54.32 INFO  ==> about to run the command: REDISCLI_AUTH=$REDIS_PASSWORD timeout 90 redis-cli -h redis-bitnami.MYNAMESPACE-8.svc.cluster.local -p 26379 --tls --cert /opt/bitnami/redis/certs/tls.crt --key /opt/bitnami/redis/certs/tls.key --cacert /opt/bitnami/redis/certs/ca.crt sentinel get-master-addr-by-name mymaster
Could not connect to Redis at redis-bitnami.MYNAMESPACE-8.svc.cluster.local:26379: Connection refused
1:S 17 Jul 2025 09:07:54.584 * Connecting to MASTER :0
1:S 17 Jul 2025 09:07:54.584 # Unable to connect to MASTER: (null)

It looks like Sentinel is trying to connect to a master that isn't properly configured or is unavailable. The :0 suggests that the master's address cannot be resolved, or it's incorrectly set (appearing as null) in the configuration map.

Additional Note: I also discovered how to correctly load the RedisJSON, RediSearch, RedisBloom, and RedisTimeSeries modules in Redis 8. I’m considering opening a PR to improve the Bitnami documentation so others can benefit from this.

samuelexferri avatar Jul 16 '25 14:07 samuelexferri

PR for updating the README.md regarding the loading of Redis Open Source 8+ modules: https://github.com/bitnami/charts/pull/35171

samuelexferri avatar Jul 17 '25 09:07 samuelexferri

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] avatar Aug 02 '25 01:08 github-actions[bot]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

github-actions[bot] avatar Aug 07 '25 03:08 github-actions[bot]