bitnami
[bitnami/redis-cluster] redis cluster on k8s setup not working
Name and Version
docker.io/bitnami/redis-cluster:7.4.0-debian-12-r0
What architecture are you using?
amd64
What steps will reproduce the bug?
I am installing redis-cluster on on-prem k8s setup but the pods are not running and its crashing.
What is the expected behavior?
redis-cluster pod should run
What do you see instead?
I am seeing this in logs
14T14:58:28.331419724Z `-._ _.-' 2024-08-14T14:58:28.331422565Z `-.__.-' 2024-08-14T14:58:28.331425618Z 2024-08-14T14:58:28.331429938Z 1:M 14 Aug 2024 14:58:28.331 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
Additional information
No response
Hi,
Could you add BITNAMI_DEBUG=true to the containers? This could provide more insight in the logs
Hi, I can not login to container as its not in running state so I can not add debug=true. Below is my value.yaml, is there any way to add BITNAMI_DEBUG=true in this file and restart the deployment ?
global:
redis:
password: "xxxxxxxx" # Set a strong password for Redis
cluster:
enabled: true
slaveCount: 2 # Number of replicas; 3 nodes total (1 master + 2 replicas)
master:
persistence:
enabled: true
storageClass: "isilon" # Define your storage class
accessModes:
- ReadWriteOnce
size: 8Gi
replica:
persistence:
enabled: true
storageClass: "isilon" # Define your storage class
accessModes:
- ReadWriteOnce
size: 8Gi
service:
type: ClusterIP
```_
I can see this logs:
redis-cluster 11:34:22.62 INFO ==> ** Starting Redis setup ** 2024-08-15T11:34:22.813758472Z redis-cluster 11:34:22.81 INFO ==> Initializing Redis
--
Thu, Aug 15 2024 5:04:22 pm | redis-cluster 11:34:22.82 INFO ==> Setting Redis config file
Thu, Aug 15 2024 5:04:24 pm | redis-cluster 11:34:24.12 INFO ==> ** Redis setup finished! **
Thu, Aug 15 2024 5:04:24 pm |
Thu, Aug 15 2024 5:04:24 pm | 1:C 15 Aug 2024 11:34:24.416 # WARNING: Changing databases number from 16 to 1 since we are in cluster mode 2024-08-15T11:34:24.417142219Z 1:C 15 Aug 2024 11:34:24.417 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
Thu, Aug 15 2024 5:04:24 pm | 1:C 15 Aug 2024 11:34:24.417 * Redis version=7.4.0, bits=64, commit=00000000, modified=1, pid=1, just started
Thu, Aug 15 2024 5:04:24 pm | 1:C 15 Aug 2024 11:34:24.417 * Configuration loaded 2024-08-15T11:34:24.417360598Z 1:M 15 Aug 2024 11:34:24.417 * monotonic clock: POSIX clock_gettime
Thu, Aug 15 2024 5:04:24 pm | _._
Thu, Aug 15 2024 5:04:24 pm | _.-``__ ''-._
Thu, Aug 15 2024 5:04:24 pm | _.-`` `. `_. ''-._ Redis Community Edition
Thu, Aug 15 2024 5:04:24 pm | .-`` .-```. ```\/ _.,_ ''-._ 7.4.0 (00000000/1) 64 bit
Thu, Aug 15 2024 5:04:24 pm | ( ' , .-` \| `, ) Running in cluster mode
Thu, Aug 15 2024 5:04:24 pm | \|`-._`-...-` __...-.``-._\|'` _.-'\| Port: 6379 2024-08-15T11:34:24.418070137Z \| `-._ `._ / _.-' \| PID: 1
Thu, Aug 15 2024 5:04:24 pm | `-._ `-._ `-./ _.-' _.-' 2024-08-15T11:34:24.418076110Z \|`-._`-._ `-.__.-' _.-'_.-'\| 2024-08-15T11:34:24.418079063Z \| `-._`-._ _.-'_.-' \| https://redis.io 2024-08-15T11:34:24.418082281Z `-._ `-._`-.__.-'_.-' _.-'
Thu, Aug 15 2024 5:04:24 pm | \|`-._`-._ `-.__.-' _.-'_.-'\| 2024-08-15T11:34:24.418088052Z \| `-._`-._ _.-'_.-' \| 2024-08-15T11:34:24.418090939Z `-._ `-._`-.__.-'_.-' _.-' 2024-08-15T11:34:24.418093910Z `-._ `-.__.-' _.-' 2024-08-15T11:34:24.418096851Z `-._ _.-' 2024-08-15T11:34:24.418099748Z `-.__.-' 2024-08-15T11:34:24.418102666Z
Thu, Aug 15 2024 5:04:24 pm | 1:M 15 Aug 2024 11:34:24.418 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
_and when I describe one for the pods then I am seeing this:_
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 39m (x5708 over 28h) kubelet Readiness probe failed: Could not connect to Redis at localhost:6379: Connection refused
Warning Unhealthy 24m (x2413 over 28h) kubelet Liveness probe failed: Could not connect to Redis at localhost:6379: Connection refused
Warning BackOff 9m8s (x5961 over 28h) kubelet Back-off restarting failed container redis-redis-cluster in pod redis-redis-cluster-0_default(9bfc23eb-1780-43a8-951b-6b393d12f53f)
Normal Pulled 4m11s (x494 over 28h) kubelet Container image "docker.io/bitnami/redis-cluster:7.4.0-debian-12-r0" already present on machine
Hi!
If you are using the chart, you can add image.debug=true to see more information on the issue.
Hi,
I updated values.yaml with image.debug=true but I am still unable to see more logs
Below are the logs:
redis-cluster 08:13:32.74 INFO ==>
redis-cluster 08:13:32.74 INFO ==> Welcome to the Bitnami redis-cluster container
redis-cluster 08:13:32.74 INFO ==> Subscribe to project updates by watching https://github.com/bitnami/containers
redis-cluster 08:13:32.75 INFO ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
redis-cluster 08:13:32.75 INFO ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
redis-cluster 08:13:32.75 INFO ==>
redis-cluster 08:13:32.75 INFO ==> ** Starting Redis setup **
redis-cluster 08:13:32.94 INFO ==> Initializing Redis
redis-cluster 08:13:32.95 INFO ==> Setting Redis config file
redis-cluster 08:13:33.34 INFO ==> Changing old IP 10.42.7.11 by the new one 10.42.7.11
redis-cluster 08:13:33.44 INFO ==> Changing old IP 10.42.4.20 by the new one 10.42.4.20
redis-cluster 08:13:33.55 INFO ==> Changing old IP 10.42.7.12 by the new one 10.42.7.12
redis-cluster 08:13:33.74 INFO ==> Changing old IP 10.42.6.22 by the new one 10.42.6.22
redis-cluster 08:13:33.84 INFO ==> Changing old IP 10.42.5.19 by the new one 10.42.5.19
redis-cluster 08:13:33.94 INFO ==> Changing old IP 10.42.4.21 by the new one 10.42.4.21
redis-cluster 08:13:33.95 INFO ==> ** Redis setup finished! **
1:C 20 Aug 2024 08:13:34.149 # WARNING: Changing databases number from 16 to 1 since we are in cluster mode
1:C 20 Aug 2024 08:13:34.149 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 20 Aug 2024 08:13:34.149 * Redis version=7.4.0, bits=64, commit=00000000, modified=1, pid=1, just started
1:C 20 Aug 2024 08:13:34.149 * Configuration loaded
1:M 20 Aug 2024 08:13:34.149 * monotonic clock: POSIX clock_gettime
1:M 20 Aug 2024 08:13:34.150 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
and below are the describe of pods:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: redis-data-redis-redis-cluster-0
ReadOnly: false
scripts:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: redis-redis-cluster-scripts
Optional: false
default-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: redis-redis-cluster-default
Optional: false
empty-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 5m39s default-scheduler 0/9 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/9 nodes are available: 9 No preemption victims found for incoming pod..
Normal Scheduled 5m38s default-scheduler Successfully assigned default/redis-redis-cluster-0 to kubelet-ea2-worker-cnj04.eagleaccess.com
Normal SuccessfulAttachVolume 5m36s attachdetach-controller AttachVolume.Attach succeeded for volume "k8s-0ea1138e4e"
Warning BackOff 4m49s (x7 over 5m21s) kubelet Back-off restarting failed container redis-redis-cluster in pod redis-redis-cluster-0_default(f4f709aa-3b11-46fc-981f-144dbe83b564)
Normal Pulled 4m35s (x4 over 5m35s) kubelet Container image "docker.io/bitnami/redis-cluster:7.4.0-debian-12-r0" already present on machine
Normal Created 4m35s (x4 over 5m35s) kubelet Created container redis-redis-cluster
Normal Started 4m35s (x4 over 5m35s) kubelet Started container redis-redis-cluster
Warning Unhealthy 4m20s (x3 over 5m25s) kubelet Liveness probe failed: Could not connect to Redis at localhost:6379: Connection refused
Warning Unhealthy 30s (x33 over 5m25s) kubelet Readiness probe failed: Could not connect to Redis at localhost:6379: Connection refused
Hi,
This is very strange, because it seems that the process is running, so I don't understand why the port is not open. Could you confirm that this does not happen on a fresh installation? Without PVCs from other previous deployments
Hi ,
This is fresh installation. I also trued to uninstall using helm and removed all related pvc and then again install using helm but still showing this issue.
Thanks, Dinesh K.
Hi @dinesh-github-ops
I don't see the cluster.slaveCount in the bitnami/redis-cluster helm chart and I am not able to reproduce your issue with version 11.0.3 of that chart. Could you please share the instructions or the chart to reproduce the issue?
This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.