charts icon indicating copy to clipboard operation
charts copied to clipboard

[bitname/redis] k8s redis Failed to resolve hostname

Open pilchita opened this issue 6 months ago • 4 comments

Name and Version

bitname/redis

What architecture are you using?

None

What steps will reproduce the bug?

Hello. I have deployed Redis via Helm on Kubernetes, and I see that the redis-node pod is restarting because it fails the sentinel check. In the logs, I only see this.

1:X 09 Jun 2025 16:22:05.606 # +tilt #tilt mode entered
1:X 09 Jun 2025 16:22:34.388 # +tilt #tilt mode entered
1:X 09 Jun 2025 16:22:55.134 # Failed to resolve hostname 'redis-node-2.redis-headless.redis.svc.cluster.local'
1:X 09 Jun 2025 16:22:55.134 # +tilt #tilt mode entered
1:X 09 Jun 2025 16:23:01.761 # +tilt #tilt mode entered
1:X 09 Jun 2025 16:23:01.761 # waitpid() returned a pid (2014) we can't find in our scripts execution queue!
1:X 09 Jun 2025 16:23:31.794 # -tilt #tilt mode exited
1:X 09 Jun 2025 16:23:31.794 # -sdown sentinel 33535e4e17bf8f9f9ff9ce8f9ddf609e558ff4f2 redis-node-1.redis-headless.redis.svc.cluster.local 26379 @ mymaster redis-node-2.redis-headless.redis.svc.cluster.local 6379
1:X 09 Jun 2025 16:23:32.818 # +sdown sentinel 33535e4e17bf8f9f9ff9ce8f9ddf609e558ff4f2 redis-node-1.redis-headless.redis.svc.cluster.local 26379 @ mymaster redis-node-2.redis-headless.redis.svc.cluster.local 6379
1:X 09 Jun 2025 16:24:21.244 # -sdown sentinel 33535e4e17bf8f9f9ff9ce8f9ddf609e558ff4f2 redis-node-1.redis-headless.redis.svc.cluster.local 26379 @ mymaster redis-node-2.redis-headless.redis.svc.cluster.local 6379

I use the param: useHostnames: true

Repo: https://github.com/bitnami/charts/tree/main/bitnami/redis Version: 2.28

Svc config:

$ kubectl describe svc redis -n redis
Name:                     redis
Namespace:                redis
Labels:                   app.kubernetes.io/component=node
                          app.kubernetes.io/instance=redis
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=redis
                          app.kubernetes.io/version=7.4.1
                          argocd.argoproj.io/instance=redis
                          helm.sh/chart=redis-20.6.0
Annotations:              <none>
Selector:                 app.kubernetes.io/component=node,app.kubernetes.io/instance=redis,app.kubernetes.io/name=redis
Type:                     ClusterIP
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.43.254.34
IPs:                      10.43.254.34
Port:                     tcp-redis  6379/TCP
TargetPort:               6379/TCP
Endpoints:                10.42.12.72:6379,10.42.11.248:6379,10.42.13.60:6379
Port:                     tcp-sentinel  26379/TCP
TargetPort:               26379/TCP
Endpoints:                10.42.12.72:26379,10.42.11.248:26379,10.42.13.60:26379
Session Affinity:         None
Internal Traffic Policy:  Cluster
Events:                   <none>

StateFulSet Config:

$ kubectl describe statefulset redis -n redis
Name:               redis-node
Namespace:          redis
CreationTimestamp:  Thu, 13 Mar 2025 11:53:38 -0300
Selector:           app.kubernetes.io/component=node,app.kubernetes.io/instance=redis,app.kubernetes.io/name=redis
Labels:             app.kubernetes.io/component=node
                    app.kubernetes.io/instance=redis
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=redis
                    app.kubernetes.io/version=7.4.1
                    argocd.argoproj.io/instance=redis
                    helm.sh/chart=redis-20.6.0
Annotations:        <none>
Replicas:           3 desired | 3 total
Update Strategy:    RollingUpdate
Pods Status:        3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app.kubernetes.io/component=node
                    app.kubernetes.io/instance=redis
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=redis
                    app.kubernetes.io/version=7.4.1
                    helm.sh/chart=redis-20.6.0
  Annotations:      checksum/configmap: f15e3630a7dbe4a9a6ae60e5c819cf30054c84d679cbcacb6aa5708b71631e57
                    checksum/health: 1eace71641fb5bd753e90012419e82c10b1180d694f91d83743b727a5aacca73
                    checksum/scripts: ecbf832037572ccc6eb37cf99a8f339513d8e5de2aacb834b0895e8f3f7e4ac1
                    checksum/secret: 44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
                    prometheus.io/port: 9121
                    prometheus.io/scrape: true
  Service Account:  redis
  Containers:
   redis:
    Image:           docker.io/bitnami/redis:7.4.1-debian-12-r2
    Port:            6379/TCP
    Host Port:       0/TCP
    SeccompProfile:  RuntimeDefault
    Command:
      /bin/bash
    Args:
      -c
      /opt/bitnami/scripts/start-scripts/start-node.sh
    Limits:
      cpu:     2
      memory:  4Gi
    Requests:
      cpu:      1
      memory:   1Gi
    Liveness:   exec [sh -c /health/ping_liveness_local.sh 15] delay=120s timeout=15s period=30s #success=1 #failure=15
    Readiness:  exec [sh -c /health/ping_readiness_local.sh 1] delay=20s timeout=1s period=5s #success=1 #failure=5
    Startup:    exec [sh -c /health/ping_liveness_local.sh 15] delay=10s timeout=5s period=10s #success=1 #failure=22
    Environment:
      BITNAMI_DEBUG:               false
      REDIS_MASTER_PORT_NUMBER:    6379
      ALLOW_EMPTY_PASSWORD:        no
      REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'redis-secret'>  Optional: false
      REDIS_MASTER_PASSWORD:       <set to the key 'redis-password' in secret 'redis-secret'>  Optional: false
      REDIS_TLS_ENABLED:           no
      REDIS_PORT:                  6379
      REDIS_SENTINEL_TLS_ENABLED:  no
      REDIS_SENTINEL_PORT:         26379
      REDIS_DATA_DIR:              /data
    Mounts:
      /data from redis-data (rw)
      /health from health (rw)
      /opt/bitnami/redis-sentinel/etc from sentinel-data (rw)
      /opt/bitnami/redis/etc from empty-dir (rw,path="app-conf-dir")
      /opt/bitnami/redis/mounted-etc from config (rw)
      /opt/bitnami/scripts/start-scripts from start-scripts (rw)
      /tmp from empty-dir (rw,path="tmp-dir")
   sentinel:
    Image:           docker.io/bitnami/redis-sentinel:7.4.1-debian-12-r2
    Port:            26379/TCP
    Host Port:       0/TCP
    SeccompProfile:  RuntimeDefault
    Command:
      /bin/bash
    Args:
      -c
      /opt/bitnami/scripts/start-scripts/start-sentinel.sh
    Limits:
      cpu:                150m
      ephemeral-storage:  2Gi
      memory:             192Mi
    Requests:
      cpu:                100m
      ephemeral-storage:  50Mi
      memory:             128Mi
    Liveness:             exec [sh -c /health/ping_sentinel.sh 15] delay=120s timeout=15s period=30s #success=1 #failure=15
    Readiness:            exec [sh -c /health/ping_sentinel.sh 10] delay=90s timeout=10s period=15s #success=1 #failure=15
    Startup:              exec [sh -c /health/ping_sentinel.sh 15] delay=30s timeout=10s period=15s #success=1 #failure=30
    Environment:
      BITNAMI_DEBUG:               false
      REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'redis-secret'>  Optional: false
      REDIS_SENTINEL_TLS_ENABLED:  no
      REDIS_SENTINEL_PORT:         26379
    Mounts:
      /data from redis-data (rw)
      /health from health (rw)
      /opt/bitnami/redis-sentinel/etc from sentinel-data (rw)
      /opt/bitnami/redis-sentinel/mounted-etc from config (rw)
      /opt/bitnami/scripts/start-scripts from start-scripts (rw)
      /tmp from empty-dir (rw,path="tmp-dir")
   metrics:
    Image:           docker.io/bitnami/redis-exporter:1.66.0-debian-12-r2
    Port:            9121/TCP
    Host Port:       0/TCP
    SeccompProfile:  RuntimeDefault
    Command:
      /bin/bash
      -c
      if [[ -f '/secrets/redis-password' ]]; then
          export REDIS_PASSWORD=$(cat /secrets/redis-password)
      fi
      redis_exporter

    Limits:
      cpu:     2
      memory:  4Gi
    Requests:
      cpu:      1
      memory:   1Gi
    Liveness:   tcp-socket :metrics delay=10s timeout=5s period=10s #success=1 #failure=5
    Readiness:  http-get http://:metrics/ delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      REDIS_ALIAS:                        redis
      REDIS_EXPORTER_WEB_LISTEN_ADDRESS:  :9121
      REDIS_USER:                         default
      REDIS_PASSWORD:                     <set to the key 'redis-password' in secret 'redis-secret'>  Optional: false
    Mounts:
      /tmp from empty-dir (rw,path="tmp-dir")
  Volumes:
   start-scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      redis-scripts
    Optional:  false
   health:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      redis-health
    Optional:  false
   config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      redis-configuration
    Optional:  false
   empty-dir:
    Type:          EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:     <unset>
  Node-Selectors:  <none>
  Tolerations:     <none>
Volume Claims:
  Name:          redis-data
  StorageClass:  nfs-infra
  Labels:        app.kubernetes.io/component=node
                 app.kubernetes.io/instance=redis
                 app.kubernetes.io/name=redis
  Annotations:   <none>
  Capacity:      5Gi
  Access Modes:  [ReadWriteOnce]
  Name:          sentinel-data
  StorageClass:  nfs-infra
  Labels:        app.kubernetes.io/component=node
                 app.kubernetes.io/instance=redis
                 app.kubernetes.io/name=redis
  Annotations:   <none>
  Capacity:      5Gi
  Access Modes:  [ReadWriteOnce]
Events:          <none>

What do you see instead?

1:X 09 Jun 2025 16:22:55.134 # Failed to resolve hostname 'redis-node-2.redis-headless.redis.svc.cluster.local'

pilchita avatar Jun 10 '25 15:06 pilchita

I have this same on chart "21.2.2", with set:

architecture: replication # Options: standalone or replication
  auth:
    password: "ddd"
  master:
    persistence:
      enable: false
    resources:
      limits:
        cpu: 200m
        memory: 1024Mi
      requests:
        cpu: 200m
        memory: 1024Mi

  replica:
    persistence:
      enable: false
    resources:
      limits:
        cpu: 200m
        memory: 1024Mi
      requests:
        cpu: 200m
        memory: 1024Mi

  sentinel:
    enabled: true
    resources:
      limits:
        cpu: 100m
        memory: 200Mi
      requests:
        cpu: 100m
        memory: 200Mi

WMP avatar Jun 10 '25 19:06 WMP

With useHostnames: false works good.

WMP avatar Jun 10 '25 22:06 WMP

Hi, the issue may not be directly related to the Bitnami container image/Helm chart, but rather to how the application is being utilized, configured in your specific environment, or tied to a particular scenario that is not easy to reproduce on our side.

If you think that's not the case and would like to contribute a solution, we'd like to invite you to create a pull request. The Bitnami team is excited to review your submission and offer feedback. You can find the contributing guidelines here.

Your contribution will greatly benefit the community. Please don't hesitate to contact us if you have any questions or need help.

Suppose you have questions about the application, customizing its content, or using technology and infrastructure. In that case, we strongly recommend that you consult the forums and user guides provided by the project responsible for the application or technology.

With that said, we'll keep this ticket open until the stale bot automatically closes it, in case someone from the community contributes valuable insights.

carrodher avatar Jun 12 '25 11:06 carrodher

I think that problem can be in redis binary. If pull request will change only useHostnames to default false, it will be accepted?

WMP avatar Jun 12 '25 12:06 WMP

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] avatar Jun 28 '25 03:06 github-actions[bot]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

github-actions[bot] avatar Jul 04 '25 01:07 github-actions[bot]