kubeblocks
kubeblocks copied to clipboard
apecloud
[BUG] redis cluster pod crash kill 1 in pod
Describe the bug
kbcli version
Kubernetes: v1.30.6
KubeBlocks: 0.9.2-beta.22
kbcli: 0.9.1
To Reproduce Steps to reproduce the behavior:
- create cluster
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
name: redis-ttznhh
namespace: default
spec:
terminationPolicy: Delete
componentSpecs:
- name: redis
componentDef: redis-7
replicas: 2
resources:
requests:
cpu: 100m
memory: 0.5Gi
limits:
cpu: 100m
memory: 0.5Gi
switchPolicy:
type: Noop
volumeClaimTemplates:
- name: data
spec:
storageClassName:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
- name: redis-sentinel
componentDef: redis-sentinel-7
replicas: 3
resources:
requests:
cpu: 100m
memory: 0.5Gi
limits:
cpu: 100m
memory: 0.5Gi
volumeClaimTemplates:
- name: data
spec:
storageClassName:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
- name: redis-twemproxy
componentDef: redis-twemproxy-0.5
replicas: 3
resources:
requests:
cpu: 100m
memory: 0.5Gi
limits:
cpu: 100m
memory: 0.5Gi
volumeClaimTemplates:
- name: data
spec:
storageClassName:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
- kiil 1
kubectl exec -it redis-ttznhh-redis-0 bash
kiil 1
- See error
kubectl get cluster
NAME CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS AGE
redis-ttznhh Delete Abnormal 5h30m
kubectl get pod
NAME READY STATUS RESTARTS AGE
redis-ttznhh-redis-0 2/3 CrashLoopBackOff 63 (71s ago) 5h
redis-ttznhh-redis-1 3/3 Running 0 5h
redis-ttznhh-redis-sentinel-0 2/2 Running 0 5h18m
redis-ttznhh-redis-sentinel-1 2/2 Running 0 5h30m
redis-ttznhh-redis-sentinel-2 2/2 Running 0 4h6m
redis-ttznhh-redis-twemproxy-0 1/1 Running 0 5h8m
redis-ttznhh-redis-twemproxy-1 1/1 Running 0 4h6m
describe pod
kubectl describe pod redis-ttznhh-redis-0
Name: redis-ttznhh-redis-0
Namespace: default
Priority: 0
Service Account: kb-redis-ttznhh
Node: aks-cicdamdpool-23244420-vmss000005/10.224.0.6
Start Time: Fri, 22 Nov 2024 11:14:22 +0800
Labels: app.kubernetes.io/component=redis-7
app.kubernetes.io/instance=redis-ttznhh
app.kubernetes.io/managed-by=kubeblocks
app.kubernetes.io/name=redis-7
app.kubernetes.io/version=redis-7
apps.kubeblocks.io/cluster-uid=894c9367-4ece-4540-912e-41b3b5e1ab17
apps.kubeblocks.io/component-name=redis
apps.kubeblocks.io/pod-name=redis-ttznhh-redis-0
componentdefinition.kubeblocks.io/name=redis-7
controller-revision-hash=5bdcd4697b
kubeblocks.io/role=primary
workloads.kubeblocks.io/access-mode=ReadWrite
workloads.kubeblocks.io/instance=redis-ttznhh-redis
workloads.kubeblocks.io/managed-by=InstanceSet
Annotations: apps.kubeblocks.io/component-replicas: 2
apps.kubeblocks.io/last-role-snapshot-version: 1732245306492198
config.kubeblocks.io/restart-redis-replication-config: 5b7b596b7f
Status: Running
IP: 10.244.1.23
IPs:
IP: 10.244.1.23
Controlled By: InstanceSet/redis-ttznhh-redis
Init Containers:
init-lorry:
Container ID: containerd://e73d5be318b68e292688d84131ce90ece3815cd703c567a183029f57ab526c31
Image: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools:0.9.2-beta.22
Image ID: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools@sha256:9f6e076fac2b480a49f4610a55d92dea3e11c74623f1d097da5d86a26264ac39
Port: <none>
Host Port: <none>
Command:
cp
-r
/bin/lorry
/config
/bin/curl
/kubeblocks/
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 22 Nov 2024 11:14:48 +0800
Finished: Fri, 22 Nov 2024 11:14:48 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Environment Variables from:
redis-ttznhh-redis-env ConfigMap Optional: false
Environment:
REDIS_DEFAULT_USER: <set to the key 'username' in secret 'redis-ttznhh-redis-account-default'> Optional: false
REDIS_DEFAULT_PASSWORD: <set to the key 'password' in secret 'redis-ttznhh-redis-account-default'> Optional: false
SENTINEL_USER: <set to the key 'username' in secret 'redis-ttznhh-redis-sentinel-account-default'> Optional: false
SENTINEL_PASSWORD: <set to the key 'password' in secret 'redis-ttznhh-redis-sentinel-account-default'> Optional: false
KB_POD_NAME: redis-ttznhh-redis-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_POD_FQDN: $(KB_POD_NAME).redis-ttznhh-redis-headless.$(KB_NAMESPACE).svc
Mounts:
/kubeblocks from kubeblocks (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dl5xk (ro)
Containers:
redis:
Container ID: containerd://3c8d5db460ec84d9faa3eb7860cc48af2de0a050f05b6de00f61dd64a7faab00
Image: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/redis-stack-server:7.2.0-v10
Image ID: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/redis-stack-server@sha256:f55712f253ffafedd9f403cee24bc6644a26e058d48a65037bc64b6d31a86349
Port: 6379/TCP
Host Port: 0/TCP
Command:
/scripts/redis-start.sh
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 22 Nov 2024 16:14:06 +0800
Finished: Fri, 22 Nov 2024 16:14:06 +0800
Ready: False
Restart Count: 63
Limits:
cpu: 100m
memory: 512Mi
Requests:
cpu: 100m
memory: 512Mi
Readiness: exec [sh -c /scripts/redis-ping.sh 1] delay=10s timeout=1s period=5s #success=1 #failure=5
Environment Variables from:
redis-ttznhh-redis-env ConfigMap Optional: false
redis-ttznhh-redis-rsm-env ConfigMap Optional: false
Environment:
REDIS_DEFAULT_USER: <set to the key 'username' in secret 'redis-ttznhh-redis-account-default'> Optional: false
REDIS_DEFAULT_PASSWORD: <set to the key 'password' in secret 'redis-ttznhh-redis-account-default'> Optional: false
SENTINEL_USER: <set to the key 'username' in secret 'redis-ttznhh-redis-sentinel-account-default'> Optional: false
SENTINEL_PASSWORD: <set to the key 'password' in secret 'redis-ttznhh-redis-sentinel-account-default'> Optional: false
KB_POD_NAME: redis-ttznhh-redis-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_POD_FQDN: $(KB_POD_NAME).redis-ttznhh-redis-headless.$(KB_NAMESPACE).svc
SERVICE_PORT: 6379
REDIS_REPL_USER: kbreplicator
REDIS_REPL_PASSWORD: $(REDIS_DEFAULT_PASSWORD)
REDIS_SENTINEL_USER: $(REDIS_REPL_USER)-sentinel
REDIS_SENTINEL_PASSWORD: $(REDIS_DEFAULT_PASSWORD)
Mounts:
/data from data (rw)
/etc/conf from redis-config (rw)
/etc/redis from redis-conf (rw)
/kb-podinfo from pod-info (rw)
/scripts from scripts (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dl5xk (ro)
metrics:
Container ID: containerd://3735a9337c963f1dd105404f1b0a4759741226d01e5357203f303c31650ec93b
Image: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1
Image ID: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto@sha256:01d06a16a2e96d0c5e1ac97db40353abad8ee73f34f9f7d148984e7de4ff0bd5
Port: 9121/TCP
Host Port: 0/TCP
Command:
/bin/agamotto
--config=/opt/conf/metrics-config.yaml
State: Running
Started: Fri, 22 Nov 2024 11:14:50 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Environment Variables from:
redis-ttznhh-redis-env ConfigMap Optional: false
redis-ttznhh-redis-rsm-env ConfigMap Optional: false
Environment:
REDIS_DEFAULT_USER: <set to the key 'username' in secret 'redis-ttznhh-redis-account-default'> Optional: false
REDIS_DEFAULT_PASSWORD: <set to the key 'password' in secret 'redis-ttznhh-redis-account-default'> Optional: false
SENTINEL_USER: <set to the key 'username' in secret 'redis-ttznhh-redis-sentinel-account-default'> Optional: false
SENTINEL_PASSWORD: <set to the key 'password' in secret 'redis-ttznhh-redis-sentinel-account-default'> Optional: false
KB_POD_NAME: redis-ttznhh-redis-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_POD_FQDN: $(KB_POD_NAME).redis-ttznhh-redis-headless.$(KB_NAMESPACE).svc
ENDPOINT: localhost:6379
REDIS_USER: $(REDIS_DEFAULT_USER)
REDIS_PASSWORD: $(REDIS_DEFAULT_PASSWORD)
Mounts:
/opt/conf from redis-metrics-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dl5xk (ro)
lorry:
Container ID: containerd://ac709cfbbb4edbc3392c8208274a95d1dbf29a926141e99aecdb95df129f6b63
Image: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/redis-stack-server:7.2.0-v10
Image ID: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/redis-stack-server@sha256:f55712f253ffafedd9f403cee24bc6644a26e058d48a65037bc64b6d31a86349
Ports: 3501/TCP, 50001/TCP
Host Ports: 0/TCP, 0/TCP
Command:
/kubeblocks/lorry
--port
3501
--grpcport
50001
--config-path
/kubeblocks/config/lorry/components/
State: Running
Started: Fri, 22 Nov 2024 11:14:50 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Readiness: http-get http://:3501/v1.0/checkrole delay=0s timeout=1s period=1s #success=1 #failure=3
Startup: tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
redis-ttznhh-redis-env ConfigMap Optional: false
redis-ttznhh-redis-rsm-env ConfigMap Optional: false
Environment:
REDIS_DEFAULT_USER: <set to the key 'username' in secret 'redis-ttznhh-redis-account-default'> Optional: false
REDIS_DEFAULT_PASSWORD: <set to the key 'password' in secret 'redis-ttznhh-redis-account-default'> Optional: false
SENTINEL_USER: <set to the key 'username' in secret 'redis-ttznhh-redis-sentinel-account-default'> Optional: false
SENTINEL_PASSWORD: <set to the key 'password' in secret 'redis-ttznhh-redis-sentinel-account-default'> Optional: false
KB_POD_NAME: redis-ttznhh-redis-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_POD_FQDN: $(KB_POD_NAME).redis-ttznhh-redis-headless.$(KB_NAMESPACE).svc
KB_RSM_ROLE_PROBE_PERIOD: 1
KB_BUILTIN_HANDLER: redis
KB_SERVICE_USER: <set to the key 'username' in secret 'redis-ttznhh-redis-account-default'> Optional: false
KB_SERVICE_PASSWORD: <set to the key 'password' in secret 'redis-ttznhh-redis-account-default'> Optional: false
KB_SERVICE_PORT: 6379
KB_DATA_PATH: /data
KB_ACTION_COMMANDS: {"accountProvision":["sh","-c","redis-cli -h $(KB_ACCOUNT_ENDPOINT) -a $REDIS_DEFAULT_PASSWORD $(KB_ACCOUNT_STATEMENT) \u0026\u0026 redis-cli -h $(KB_ACCOUNT_ENDPOINT) -a $REDIS_DEFAULT_PASSWORD acl save "],"postProvision":["/bin/bash","-c","/scripts/redis-register-to-sentinel.sh"]}
SERVICE_PORT: 6379
REDIS_REPL_USER: kbreplicator
REDIS_REPL_PASSWORD: $(REDIS_DEFAULT_PASSWORD)
REDIS_SENTINEL_USER: $(REDIS_REPL_USER)-sentinel
REDIS_SENTINEL_PASSWORD: $(REDIS_DEFAULT_PASSWORD)
KB_RSM_ACTION_SVC_LIST: null
KB_RSM_ROLE_UPDATE_MECHANISM: DirectAPIServerEventUpdate
KB_RSM_ROLE_PROBE_TIMEOUT: 1
KB_CLUSTER_NAME: (v1:metadata.labels['app.kubernetes.io/instance'])
KB_COMP_NAME: (v1:metadata.labels['apps.kubeblocks.io/component-name'])
KB_SERVICE_CHARACTER_TYPE: redis
Mounts:
/data from data (rw)
/etc/conf from redis-config (rw)
/etc/redis from redis-conf (rw)
/kb-podinfo from pod-info (rw)
/kubeblocks from kubeblocks (rw)
/scripts from scripts (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dl5xk (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
pod-info:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.annotations['rs.apps.kubeblocks.io/primary'] -> primary-pod
metadata.annotations['apps.kubeblocks.io/component-replicas'] -> component-replicas
redis-metrics-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: redis-ttznhh-redis-redis-metrics-config
Optional: false
redis-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: redis-ttznhh-redis-redis-replication-config
Optional: false
scripts:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: redis-ttznhh-redis-redis-scripts
Optional: false
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-redis-ttznhh-redis-0
ReadOnly: false
redis-conf:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kubeblocks:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kube-api-access-dl5xk:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: kb-data=true:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal checkRole 60m lorry {"operation":"checkRole","originalRole":"primary"}
Normal checkRole 55m lorry {"operation":"checkRole","originalRole":"primary"}
Normal checkRole 50m lorry {"operation":"checkRole","originalRole":"primary"}
Normal checkRole 45m lorry {"operation":"checkRole","originalRole":"primary"}
Normal checkRole 40m lorry {"operation":"checkRole","originalRole":"primary"}
Normal checkRole 35m lorry {"operation":"checkRole","originalRole":"primary"}
Normal checkRole 30m lorry {"operation":"checkRole","originalRole":"primary"}
Normal checkRole 25m lorry {"operation":"checkRole","originalRole":"primary"}
Normal checkRole 20m lorry {"operation":"checkRole","originalRole":"primary"}
Normal checkRole 15m lorry {"operation":"checkRole","originalRole":"primary"}
Normal checkRole 10m lorry {"operation":"checkRole","originalRole":"primary"}
Normal checkRole 5m20s lorry {"operation":"checkRole","originalRole":"primary"}
Warning BackOff 94s (x1423 over 5h) kubelet Back-off restarting failed container redis in pod redis-ttznhh-redis-0_default(4313e357-ce1f-495a-9b61-49486537a001)
Normal checkRole 20s lorry {"operation":"checkRole","originalRole":"primary"}
logs pod
kubectl logs redis-ttznhh-redis-0 -c redis --previous
+ declare -g primary
+ declare -g primary_port
+ declare -g default_initialize_pod_ordinal
+ declare -g headless_postfix=headless
+ parse_redis_advertised_svc_if_exist redis-ttznhh-redis-0
+ local pod_name=redis-ttznhh-redis-0
+ [[ -z '' ]]
+ echo 'Environment variable REDIS_ADVERTISED_PORT not found. Ignoring.'
+ return 0
+ build_redis_conf
+ load_redis_template_conf
+ echo 'include /etc/conf/redis.conf'
+ build_announce_ip_and_port
+ '[' -n '' ']'
+ kb_pod_fqdn=redis-ttznhh-redis-0.redis-ttznhh-redis-headless.default.svc
+ echo 'redis use kb pod fqdn redis-ttznhh-redis-0.redis-ttznhh-redis-headless.default.svc to announce'
+ echo 'replica-announce-ip redis-ttznhh-redis-0.redis-ttznhh-redis-headless.default.svc'
+ build_redis_service_port
+ service_port=6379
+ '[' -n 6379 ']'
+ service_port=6379
+ echo 'port 6379'
+ build_replicaof_config
+ init_or_get_primary_node
+ init_or_get_primary_from_redis_sentinel
+ '[' -z redis-ttznhh-redis-sentinel ']'
+ '[' -z redis-ttznhh-redis-sentinel-0,redis-ttznhh-redis-sentinel-1,redis-ttznhh-redis-sentinel-2 ']'
+ '[' -z redis-ttznhh-redis-sentinel-headless ']'
+ old_ifs='
'
+ IFS=,
+ set -f
+ read -ra sentinel_pod_list
+ set +f
+ IFS='
'
+ declare -A master_count_map
+ local first_redis_primary_host=
+ local first_redis_primary_port=
+ for sentinel_pod in "${sentinel_pod_list[@]}"
+ sentinel_pod_fqdn=redis-ttznhh-redis-sentinel-0.redis-ttznhh-redis-sentinel-headless
+ retry_get_master_addr_by_name_from_sentinel redis-ttznhh-redis-sentinel-0.redis-ttznhh-redis-sentinel-headless
+ local sentinel_pod_fqdn=redis-ttznhh-redis-sentinel-0.redis-ttznhh-redis-sentinel-headless
+ local retry_count=0
+ local max_retry=3
+ local retry_delay=2
+ local timeout_value=5
+ '[' 0 -lt 3 ']'
+ set +x
Environment variable REDIS_ADVERTISED_PORT not found. Ignoring.
redis use kb pod fqdn redis-ttznhh-redis-0.redis-ttznhh-redis-headless.default.svc to announce
execute command: timeout 5 redis-cli -h redis-ttznhh-redis-sentinel-0.redis-ttznhh-redis-sentinel-headless -p 26379 -a ******** sentinel get-master-addr-by-name redis-ttznhh-redis
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ exit_code=0
+ '[' 0 -eq 0 ']'
+ old_ifs='
'
+ IFS='
'
+ set -f
+ read -r -d '' -a REDIS_SENTINEL_PRIMARY_INFO
+ set +f
+ IFS='
'
+ '[' 2 -eq 2 ']'
+ '[' -n redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ']'
+ '[' -n 6379 ']'
+ echo 'Successfully retrieved primary info from sentinel: redis-ttznhh-redis-sentinel-0.redis-ttznhh-redis-sentinel-headless'
+ return 0
Successfully retrieved primary info from sentinel: redis-ttznhh-redis-sentinel-0.redis-ttznhh-redis-sentinel-headless
sentinel:redis-ttznhh-redis-sentinel-0.redis-ttznhh-redis-sentinel-headless has master info: redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc 6379
+ echo 'sentinel:redis-ttznhh-redis-sentinel-0.redis-ttznhh-redis-sentinel-headless has master info: redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc 6379'
+ '[' 2 -ne 2 ']'
+ '[' -z redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ']'
+ '[' -z 6379 ']'
+ host_port_key=redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc:6379
+ master_count_map[$host_port_key]=1
+ [[ -z '' ]]
+ [[ -z '' ]]
+ first_redis_primary_host=redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc
+ first_redis_primary_port=6379
+ '[' redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc '!=' redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ']'
+ '[' 6379 '!=' 6379 ']'
+ for sentinel_pod in "${sentinel_pod_list[@]}"
+ sentinel_pod_fqdn=redis-ttznhh-redis-sentinel-1.redis-ttznhh-redis-sentinel-headless
+ retry_get_master_addr_by_name_from_sentinel redis-ttznhh-redis-sentinel-1.redis-ttznhh-redis-sentinel-headless
+ local sentinel_pod_fqdn=redis-ttznhh-redis-sentinel-1.redis-ttznhh-redis-sentinel-headless
+ local retry_count=0
+ local max_retry=3
+ local retry_delay=2
+ local timeout_value=5
+ '[' 0 -lt 3 ']'
+ set +x
execute command: timeout 5 redis-cli -h redis-ttznhh-redis-sentinel-1.redis-ttznhh-redis-sentinel-headless -p 26379 -a ******** sentinel get-master-addr-by-name redis-ttznhh-redis
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ exit_code=0
+ '[' 0 -eq 0 ']'
+ old_ifs='
'
+ IFS='
'
+ set -f
+ read -r -d '' -a REDIS_SENTINEL_PRIMARY_INFO
+ set +f
+ IFS='
'
+ '[' 2 -eq 2 ']'
+ '[' -n redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ']'
+ '[' -n 6379 ']'
+ echo 'Successfully retrieved primary info from sentinel: redis-ttznhh-redis-sentinel-1.redis-ttznhh-redis-sentinel-headless'
+ return 0
Successfully retrieved primary info from sentinel: redis-ttznhh-redis-sentinel-1.redis-ttznhh-redis-sentinel-headless
+ echo 'sentinel:redis-ttznhh-redis-sentinel-1.redis-ttznhh-redis-sentinel-headless has master info: redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc 6379'
+ '[' 2 -ne 2 ']'
sentinel:redis-ttznhh-redis-sentinel-1.redis-ttznhh-redis-sentinel-headless has master info: redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc 6379
+ '[' -z redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ']'
+ '[' -z 6379 ']'
+ host_port_key=redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc:6379
+ master_count_map[$host_port_key]=2
+ [[ -z redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ]]
+ '[' redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc '!=' redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ']'
+ '[' 6379 '!=' 6379 ']'
+ for sentinel_pod in "${sentinel_pod_list[@]}"
+ sentinel_pod_fqdn=redis-ttznhh-redis-sentinel-2.redis-ttznhh-redis-sentinel-headless
+ retry_get_master_addr_by_name_from_sentinel redis-ttznhh-redis-sentinel-2.redis-ttznhh-redis-sentinel-headless
+ local sentinel_pod_fqdn=redis-ttznhh-redis-sentinel-2.redis-ttznhh-redis-sentinel-headless
+ local retry_count=0
+ local max_retry=3
+ local retry_delay=2
+ local timeout_value=5
+ '[' 0 -lt 3 ']'
+ set +x
execute command: timeout 5 redis-cli -h redis-ttznhh-redis-sentinel-2.redis-ttznhh-redis-sentinel-headless -p 26379 -a ******** sentinel get-master-addr-by-name redis-ttznhh-redis
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ exit_code=0
+ '[' 0 -eq 0 ']'
+ old_ifs='
'
+ IFS='
'
+ set -f
+ read -r -d '' -a REDIS_SENTINEL_PRIMARY_INFO
+ set +f
+ IFS='
'
+ '[' 2 -eq 2 ']'
+ '[' -n redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ']'
+ '[' -n 6379 ']'
+ echo 'Successfully retrieved primary info from sentinel: redis-ttznhh-redis-sentinel-2.redis-ttznhh-redis-sentinel-headless'
+ return 0
Successfully retrieved primary info from sentinel: redis-ttznhh-redis-sentinel-2.redis-ttznhh-redis-sentinel-headless
sentinel:redis-ttznhh-redis-sentinel-2.redis-ttznhh-redis-sentinel-headless has master info: redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc 6379
+ echo 'sentinel:redis-ttznhh-redis-sentinel-2.redis-ttznhh-redis-sentinel-headless has master info: redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc 6379'
+ '[' 2 -ne 2 ']'
+ '[' -z redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ']'
+ '[' -z 6379 ']'
+ host_port_key=redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc:6379
+ master_count_map[$host_port_key]=3
+ [[ -z redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ]]
+ '[' redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc '!=' redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc ']'
+ '[' 6379 '!=' 6379 ']'
get all primary info from redis sentinel master_count_map: 3
+ echo 'get all primary info from redis sentinel master_count_map: 3'
+ '[' 1 -eq 0 ']'
+ max_count=0
+ for host_port in "${!master_count_map[@]}"
+ (( 3 > max_count ))
+ max_count=3
++ cut -d: -f1
++ echo redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc:6379
+ primary=redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc
++ echo redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc:6379
++ cut -d: -f2
+ primary_port=6379
+ check_current_pod_is_primary
+ current_pod=redis-ttznhh-redis-0.redis-ttznhh-redis
+ [[ redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc == *\r\e\d\i\s\-\t\t\z\n\h\h\-\r\e\d\i\s\-\0\.\r\e\d\i\s\-\t\t\z\n\h\h\-\r\e\d\i\s* ]]
+ '[' -n '' ']'
+ [[ redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc == \1\0\.\2\4\4\.\1\.\2\3 ]]
+ return 1
+ echo 'replicaof redis-ttznhh-redis-1.redis-ttznhh-redis-headless.default.svc 6379'
+ rebuild_redis_acl_file
+ '[' -f /data/users.acl ']'
+ sed -i '/user default on/d' /data/users.acl
+ sed -i '/user kbreplicator on/d' /data/users.acl
+ sed -i '/user kbreplicator-sentinel on/d' /data/users.acl
+ build_redis_default_accounts
+ set +x
build default accounts succeeded!
+ echo 'aclfile /data/users.acl'
+ echo 'build default accounts succeeded!'
+ start_redis_server
+ exec_cmd='exec redis-server /etc/redis/redis.conf'
+ '[' -f /opt/redis-stack/lib/redisearch.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so '
+ '[' -f /opt/redis-stack/lib/redistimeseries.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so '
+ '[' -f /opt/redis-stack/lib/rejson.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so '
+ '[' -f /opt/redis-stack/lib/redisbloom.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so '
+ '[' -f /opt/redis-stack/lib/redisgraph.so ']'
+ '[' -f /opt/redis-stack/lib/rediscompat.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so'
+ '[' -f /opt/redis-stack/lib/redisgears.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so '
Starting redis server cmd: exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so
+ echo 'Starting redis server cmd: exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so '
+ eval 'exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so '
++ exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so
Expected behavior A clear and concise description of what you expected to happen.
Screenshots If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
Additional context Add any other context about the problem here.
This issue has been marked as stale because it has been open for 30 days with no activity
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}