bitnami/redis: sentinel cannot startup if enable TLS

[latituder]

Name and Version

bitnami/redis 17.0.7

What steps will reproduce the bug?

1. in K8S 1.21
2. with this HELM value configuration:

architecture: replication
auth:
  enabled: true
  password: ""
  sentinel: true
  usePasswordFiles: true
master:
  persistence:
    size: 8Gi
metrics:
  enabled: true
  serviceMonitor:
    enabled: true
networkPolicy:
  enabled: true
replica:
  persistence:
    size: 8Gi
sentinel:
  automateClusterRecovery: true
  downAfterMilliseconds: 2000
  enabled: true
  masterSet: redis-inst1-sentinel-master
  persistence:
    enabled: true
    size: 100Mi
tls:
  authClients: false
  autoGenerated: true
  enabled: true

getting error:

$ kc logs redis-inst1-node-0 -c sentinel
 10:08:31.41 INFO  ==> about to run the command: REDISCLI_AUTH=$REDIS_PASSWORD timeout 220 redis-cli -h redis-inst1.shared-redis.svc.cluster.local -p 26379 --tls --cert /opt/bitnami/redis/certs/tls.crt --key /opt/bitnami/redis/certs/tls.key --cacert /opt/bitnami/redis/certs/ca.crt sentinel get-master-addr-by-name redis-inst1-sentinel-master
Could not connect to Redis at redis-inst1.shared-redis.svc.cluster.local:26379: Temporary failure in name resolution

*** FATAL CONFIG FILE ERROR (Redis 7.0.4) ***
Reading the configuration file, at line 18
>>> 'sentinel "--port" "0"'
Unrecognized sentinel configuration statement.

Checking the cm of sentinel.conf, it doesn't have like 18

sentinel.conf:
----
dir "/tmp"
port 26379
sentinel monitor redis-inst1-sentinel-master redis-inst1-node-0.redis-inst1-headless.shared-redis.svc.cluster.local 6379 2
sentinel down-after-milliseconds redis-inst1-sentinel-master 2000
sentinel failover-timeout redis-inst1-sentinel-master 18000
sentinel parallel-syncs redis-inst1-sentinel-master 1
# User-supplied sentinel configuration:
# End of sentinel configuration
master.conf:
----

[carrodher]

Can you try adding a new parameter to increase the verbosity of the logs?

image:
  debug: true

Apart from that, are you able to access the container and check the real config file in the container file system instead of the CM?

[latituder]

yes, exec in the container, and I can see the start-sentinel.sh looks like this:

kc exec -it redis-inst1-node-0 -c sentinel -- sh
$ cd /opt/bitnami/scripts/start-scripts
$ cat start-sentinel.sh
#!/bin/bash

. /opt/bitnami/scripts/libos.sh
. /opt/bitnami/scripts/libvalidations.sh
. /opt/bitnami/scripts/libfile.sh

HEADLESS_SERVICE="redis-inst1-headless.shared-redis.svc.cluster.local"
REDIS_SERVICE="redis-inst1.shared-redis.svc.cluster.local"

get_port() {
    hostname="$1"
    type="$2"

    port_var=$(echo "${hostname^^}_SERVICE_PORT_$type" | sed "s/-/_/g")
    port=${!port_var}

    if [ -z "$port" ]; then
        case $type in
            "SENTINEL")
                echo 26379
                ;;
            "REDIS")
                echo 6379
                ;;
        esac
    else
        echo $port
    fi
}

get_full_hostname() {
    hostname="$1"
    echo "${hostname}.${HEADLESS_SERVICE}"
}

SERVPORT=$(get_port "$HOSTNAME" "SENTINEL")
REDISPORT=$(get_port "$HOSTNAME" "REDIS")
SENTINEL_SERVICE_PORT=$(get_port "redis-inst1" "TCP_SENTINEL")

sentinel_conf_set() {
    local -r key="${1:?missing key}"
    local value="${2:-}"

    # Sanitize inputs
    value="${value//\\/\\\\}"
    value="${value//&/\\&}"
    value="${value//\?/\\?}"
    [[ "$value" = "" ]] && value="\"$value\""

    replace_in_file "/opt/bitnami/redis-sentinel/etc/sentinel.conf" "^#*\s*${key} .*" "${key} ${value}" false
}
sentinel_conf_add() {
    echo $'\n'"$@" >> "/opt/bitnami/redis-sentinel/etc/sentinel.conf"
}
host_id() {
    echo "$1" | openssl sha1 | awk '{print $2}'
}
get_sentinel_master_info() {
    if is_boolean_yes "$REDIS_SENTINEL_TLS_ENABLED"; then
        sentinel_info_command="REDISCLI_AUTH="\$REDIS_PASSWORD" timeout 220 redis-cli -h $REDIS_SERVICE -p $SENTINEL_SERVICE_PORT --tls --cert ${REDIS_SENTINEL_TLS_CERT_FILE} --key ${REDIS_SENTINEL_TLS_KEY_FILE} --cacert ${REDIS_SENTINEL_TLS_CA_FILE} sentinel get-master-addr-by-name redis-inst1-sentinel-master"
    else
        sentinel_info_command="REDISCLI_AUTH="\$REDIS_PASSWORD" timeout 220 redis-cli -h $REDIS_SERVICE -p $SENTINEL_SERVICE_PORT sentinel get-master-addr-by-name redis-inst1-sentinel-master"
    fi
    info "about to run the command: $sentinel_info_command"
    eval $sentinel_info_command
}

[[ -f $REDIS_PASSWORD_FILE ]] && export REDIS_PASSWORD="$(< "${REDIS_PASSWORD_FILE}")"

master_in_persisted_conf="$(get_full_hostname "$HOSTNAME")"
if ! get_sentinel_master_info && [[ "$master_in_persisted_conf" == "$(get_full_hostname "$HOSTNAME")" ]]; then
    # No master found, lets create a master node
    export REDIS_REPLICATION_MODE="master"

    REDIS_MASTER_HOST=$(get_full_hostname "$HOSTNAME")
    REDIS_MASTER_PORT_NUMBER="$REDISPORT"
else
    export REDIS_REPLICATION_MODE="slave"

    # Fetches current master's host and port
    REDIS_SENTINEL_INFO=($(get_sentinel_master_info))
    info "printing REDIS_SENTINEL_INFO=(${REDIS_SENTINEL_INFO[0]},${REDIS_SENTINEL_INFO[1]})"
    REDIS_MASTER_HOST=${REDIS_SENTINEL_INFO[0]}
    REDIS_MASTER_PORT_NUMBER=${REDIS_SENTINEL_INFO[1]}
fi

if [[ -n "$REDIS_EXTERNAL_MASTER_HOST" ]]; then
  REDIS_MASTER_HOST="$REDIS_EXTERNAL_MASTER_HOST"
  REDIS_MASTER_PORT_NUMBER="${REDIS_EXTERNAL_MASTER_PORT}"
fi

cp /opt/bitnami/redis-sentinel/mounted-etc/sentinel.conf /opt/bitnami/redis-sentinel/etc/sentinel.conf
printf "\nsentinel auth-pass %s %s" "redis-inst1-sentinel-master" "$REDIS_PASSWORD" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf
printf "\nrequirepass %s" "$REDIS_PASSWORD" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf
printf "\nsentinel myid %s" "$(host_id "$HOSTNAME")" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf

sentinel_conf_set "sentinel monitor" "redis-inst1-sentinel-master "$REDIS_MASTER_HOST" "$REDIS_MASTER_PORT_NUMBER" 2"

add_known_sentinel() {
    hostname="$1"
    ip="$2"

    if [[ -n "$hostname" && -n "$ip" && "$hostname" != "$HOSTNAME" ]]; then
        sentinel_conf_add "sentinel known-sentinel redis-inst1-sentinel-master $(get_full_hostname "$hostname") $(get_port "$hostname" "SENTINEL") $(host_id "$hostname")"
    fi
}
add_known_replica() {
    hostname="$1"
    ip="$2"

    if [[ -n "$ip" && "$(get_full_hostname "$hostname")" != "$REDIS_MASTER_HOST" ]]; then
        sentinel_conf_add "sentinel known-replica redis-inst1-sentinel-master $(get_full_hostname "$hostname") $(get_port "$hostname" "REDIS")"
    fi
}

# Add available hosts on the network as known replicas & sentinels
for node in $(seq 0 $((3-1))); do
    hostname="redis-inst1-node-$node"
    ip="$(getent hosts "$hostname.$HEADLESS_SERVICE" | awk '{ print $1 }')"
    add_known_sentinel "$hostname" "$ip"
    add_known_replica "$hostname" "$ip"
done

echo "" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf
echo "sentinel announce-hostnames yes" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf
echo "sentinel resolve-hostnames yes" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf
echo "sentinel announce-port $SERVPORT" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf
echo "sentinel announce-ip $(get_full_hostname "$HOSTNAME")" >> /opt/bitnami/redis-sentinel/etc/sentinel.conf
ARGS=("--port" "0")
ARGS+=("--tls-port" "${REDIS_SENTINEL_TLS_PORT_NUMBER}")
ARGS+=("--tls-cert-file" "${REDIS_SENTINEL_TLS_CERT_FILE}")
ARGS+=("--tls-key-file" "${REDIS_SENTINEL_TLS_KEY_FILE}")
ARGS+=("--tls-ca-cert-file" "${REDIS_SENTINEL_TLS_CA_FILE}")
ARGS+=("--tls-replication" "yes")
ARGS+=("--tls-auth-clients" "${REDIS_SENTINEL_TLS_AUTH_CLIENTS}")
exec redis-server /opt/bitnami/redis-sentinel/etc/sentinel.conf --sentinel "${ARGS[@]}"

meanwhile: in /opt/bitnami/redis-sentinel/etc, I can see the conf file:

$ cd /opt/bitnami/redis-sentinel/etc
$ ls
sentinel.conf
$ cat sentinel.conf
dir "/tmp"
port 26379
sentinel monitor redis-inst1-sentinel-master redis-inst1-node-0.redis-inst1-headless.shared-redis.svc.cluster.local 6379 2
sentinel down-after-milliseconds redis-inst1-sentinel-master 2000
sentinel failover-timeout redis-inst1-sentinel-master 18000
sentinel parallel-syncs redis-inst1-sentinel-master 1
# User-supplied sentinel configuration:
# End of sentinel configuration
sentinel auth-pass redis-inst1-sentinel-master ************
requirepass *********************
sentinel myid 2549ca9*************5f7d2a

after wait about 8mins, I see sentinel contianer give me fatal error:

 09:01:13.47 INFO  ==> about to run the command: REDISCLI_AUTH=$REDIS_PASSWORD timeout 220 redis-cli -h redis-inst1.shared-redis.svc.cluster.local -p 26379 --tls --cert /opt/bitnami/redis/certs/tls.crt --key /opt/bitnami/redis/certs/tls.key --cacert /opt/bitnami/redis/certs/ca.crt sentinel get-master-addr-by-name redis-inst1-sentinel-master
Could not connect to Redis at redis-inst1.shared-redis.svc.cluster.local:26379: Temporary failure in name resolution

*** FATAL CONFIG FILE ERROR (Redis 7.0.4) ***
Reading the configuration file, at line 18
>>> 'sentinel "--port" "0"'
Unrecognized sentinel configuration statement.

I think the --port 0 error was from script start-sentinel.sh

[rafariossaa]

Hi, Thanks for reporting this. It seems to be an issue when using Redis v7, I have run the same configuration but using v6 and it worked well.

helm install myredis bitnami/redis -f user.yaml --set image.tag=6.2.7 --set sentinel.image.tag=6.2.7

I am opening an internal task to look further into this. We will come back as soon as we have news.

[latituder]

thank you. confirm in image.tag=6.2.7,sentinel.image.tag=6.2.7, the issue doesn't occur; How would this issue be resolved in version 7.x?

[rafariossaa]

I am not sure on how that would be solved. That is why I opened an internal task, to study it and provide the proper fix.

[latituder]

thanks. By the way, if I use the Redis v7 + sentinel v6.2.x, the pods are still able to running; would sentinel v6.2 fit Redis v7? or better to use them all in v6.2x?

[simdevmat]

Changing the startup script exec command from

exec redis-server /opt/bitnami/redis-sentinel/etc/sentinel.conf --sentinel {{- if .Values.tls.enabled }} "${ARGS[@]}" {{- end }}

to

exec redis-server /opt/bitnami/redis-sentinel/etc/sentinel.conf {{- if .Values.tls.enabled }} "${ARGS[@]}" {{- end }} --sentinel

Works for me.

Have tested with 6.2.7 and 7.0.4

[latituder]

Thanks for this. change the --sentinel position also works for me!

[rafariossaa]

Hi, Thanks for providing the fix.

[rafariossaa]

Hi, The fix was merged. Could you give it a try ?