Sentinel not picking up new master IP address after container is redeployed

[mark4409]

Name and Version

bitnami/redis-sentinel:7.0.4

What steps will reproduce the bug?

In this environment:

version: '3.9'

networks:
  redis-nw:
    external: true

services:
  redis-master:
    image: 'bitnami/redis:latest'
    hostname: redis-master
    ports:
      - '6379:6379'
    networks:
      redis-nw:
        aliases:
          - redis-master
    command: /opt/bitnami/scripts/redis/run.sh --maxmemory 4gb --min-replicas-to-write 3
    environment:
      - REDIS_REPLICATION_MODE=master
      - REDIS_PASSWORD=xxxxxxxxxx
    volumes:
      - '/datadrive/redis:/bitnami'
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - "node.hostname==vmt-swarm1-mgr-tst-uks-01"

  redis-slave1:
    image: 'bitnami/redis:latest'
    hostname: redis-slave1
    networks:
      redis-nw:
        aliases:
          - redis-slave1
    ports:
      - '6379'
    depends_on:
      - redis-master
    volumes:
      - '/datadrive/redis:/bitnami'
    command: /opt/bitnami/scripts/redis/run.sh --maxmemory 4gb --min-replicas-to-write 3
    environment:
      - REDIS_REPLICATION_MODE=replica
      - REDIS_MASTER_HOST=redis-master
      - REDIS_MASTER_PORT_NUMBER=6379
      - REDIS_MASTER_PASSWORD=xxxxxxxxxxxxxxxx
      - REDIS_PASSWORD=xxxxxxxxxxxxxxx
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - "node.hostname==vmt-swarm1-mgr-tst-uks-02"

  redis-slave2:
    image: 'bitnami/redis:latest'
    hostname: redis-slave2
    networks:
      redis-nw:
        aliases:
          - redis-slave2
    ports:
      - '6379'
    depends_on:
      - redis-master
    volumes:
      - '/datadrive/redis:/bitnami'
    command: /opt/bitnami/scripts/redis/run.sh --maxmemory 4gb --min-replicas-to-write 3
    environment:
      - REDIS_REPLICATION_MODE=replica
      - REDIS_MASTER_HOST=redis-master
      - REDIS_MASTER_PORT_NUMBER=6379
      - REDIS_MASTER_PASSWORD=xxxxxxxxxxxxxxxx
      - REDIS_PASSWORD=xxxxxxxxxxxxxxx
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - "node.hostname==vmt-swarm1-mgr-tst-uks-03"

  redis-sentinel:
    image: 'bitnami/redis-sentinel:latest'
    networks:
      - redis-nw
    ports:
      - '26379:26379'
    depends_on:
      - redis-master
    volumes:
      - 'redis-sentinel-volume:/bitnami'
    environment:
      - REDIS_SENTINEL_PORT_NUMBER=26379
      - REDIS_SENTINEL_RESOLVE_HOSTNAMES=yes
      - REDIS_MASTER_HOST=redis-master
      - REDIS_MASTER_PORT_NUMBER=6379
      - REDIS_MASTER_PASSWORD=xxxxxxxxxxxxxx
      - REDIS_SENTINEL_DOWN_AFTER_MILLISECONDS=5000
      - REDIS_SENTINEL_FAILOVER_TIMEOUT=5000
    deploy:
      mode: global
      placement:
        constraints: [node.role == manager]

volumes:
  redis-sentinel-volume:
    driver: local

when deployed on an overlay network with docker swarm across 3 nodes, the service initially works - however when the redis services are redeployed and pick up new IP addresses, Sentinel doesn't update its master redis IP address and is still trying to connect to the old IP.

What is the expected behavior?

I would expect Sentinel to update its IP address for redis-master from DNS

What do you see instead?

Sentinel container log files are still showing connection attempts to the old IP address.

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[aoterolorenzo]

Hey @mark4409,

Using hostname (introduced in redis's Add hostname support in Sentinel may could solve the issue? It solves a pretty similar issue Sentinel: resolve master_host to IP address.

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.