[charts/redis-ha]

[jb0071913]

Describe the bug We have a redis cluster of 5 nodes and partial syncs are occurring continuously between the master and slave nodes. Sometimes the syncs stop and only occur every minute but on occasion, they can run continuously for approx. 5 mins. The logs would indicate that the connection between master and slave nodes is lost and initiates the partial sync.

I have reported this as an issue rather than a bug as I am not sure if my configuration is wrong etc.

To Reproduce Configuration files

redis.conf:

dir "/data" port 6379 rename-command FLUSHDB "" rename-command FLUSHALL "" maxmemory 0 maxmemory-policy volatile-lru min-replicas-max-lag 5 min-replicas-to-write 1 rdbchecksum yes rdbcompression yes repl-diskless-sync yes save 900 1 slave-announce-port 6379 slave-announce-ip X.X.X.X

sentinel.conf:

sentinel myid cf1d3e4e80fc46ae8e7f9e76b7c1d566dac0d6d8 sentinel monitor my-master X.X.X.X 6379 2 dir "/data" port 26379 sentinel down-after-milliseconds my-master 10000 maxclients 10000 sentinel parallel-syncs my-master 5 sentinel announce-ip "X.X.X.X" sentinel announce-port 26379 latency-tracking-info-percentiles 50 99 99.9 user default on nopass ~* &* +@all sentinel config-epoch my-master 0 sentinel leader-epoch my-master 0 sentinel current-epoch 0 sentinel known-sentinel my-master X.X.X.X 26379 b6df0c1e8d43ec29ec6a09e2ea424f43288380bf sentinel known-replica my-master X.X.X.X 6379 sentinel known-sentinel my-master X.X.X.X 26379 920882492e76912cce8a77ea253b3abcba62509c sentinel known-replica my-master X.X.X.X 6379 sentinel known-sentinel my-master X.X.X.X 26379 2cdf5100fafb02f9afdad75f1739f7e93920dbff sentinel known-replica my-master X.X.X.X 6379 sentinel known-sentinel my-master X.X.X.X 26379 312133fe0f1b31f5bb1a291a2bcc948c030b735d sentinel known-replica my-master X.X.X.X 6379`

Log output (for master node):

24 Nov 2023 13:39:03.085 # Connection with replica X.X.X.X:X lost. 24 Nov 2023 13:39:03.085 # Client id=462995 addr=X.X.X.X:38584 laddr=X.X.X.X:6379 fd=17 name= age=61 idle=1 flags=S db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=1 omem=234881048 tot-mem=234903320 events=r cmd=replconf user=default redir=-1 resp=2 closed for overcoming of output buffer limits. 24 Nov 2023 13:39:03.155 * Replica X.X.X.X:X asks for synchronization 24 Nov 2023 13:39:03.155 * Partial resynchronization request from X.X.X.X:X accepted. Sending 0 bytes of backlog starting from offset 2010165167 24 Nov 2023 13:39:03.209 # Connection with replica X.X.X.X:6379 lost 24 Nov 2023 13:39:03.209 # Client id=462996 addr=X.X.X.X:41328 laddr=X.X.X.X:6379 fd=11 name= age=61 idle=1 flags=S db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=1 omem=234881048 tot-mem=234903320 events=r cmd=replconf user=default redir=-1 resp=2 closed for overcoming of output buffer limits. 24 Nov 2023 13:39:03.242 * Replica X.X.X.X:6379 asks for synchronization 24 Nov 2023 13:39:03.242 * Partial resynchronization request from X.X.X.X:6379 accepted. Sending 0 bytes of backlog starting from offset 2010165167. 24 Nov 2023 13:39:04.129 # Connection with replica X.X.X.X:6379 lost. 24 Nov 2023 13:39:04.129 # Client id=462994 addr=X.X.X.X:39922 laddr=X.X.X.X:6379 fd=13 name= age=62 idle=0 flags=S db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=1 omem=234881048 tot-mem=234903320 events=r cmd=replconf user=default redir=-1 resp=2 closed for overcoming of output buffer limits. 24 Nov 2023 13:39:04.163 * Replica X.X.X.X:6379 asks for synchronization 24 Nov 2023 13:39:04.163 * Partial resynchronization request from X.X.X.X:6379 accepted. Sending 0 bytes of backlog starting from offset 2010165181. 24 Nov 2023 13:39:13.941 # Connection with replica X.X.X.X:6379 lost. 24 Nov 2023 13:39:13.941 # Client id=462999 addr=X.X.X.X:35010 laddr=X.X.X.X:6379 fd=14 name= age=63 idle=0 flags=S db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=1 omem=234881048 tot-mem=234903320 events=r cmd=replconf user=default redir=-1 resp=2 closed for overcoming of output buffer limits 24 Nov 2023 13:39:14.007 * Replica X.X.X.X:6379 asks for synchronization 24 Nov 2023 13:39:14.007 * Partial resynchronization request from X.X.X.X:6379 accepted. Sending 0 bytes of backlog starting from offset 2010169351.

Expected behavior Partial syncs should not be continuously occurring.

Additional context K8s v1.27.5 Redis v7.0.9 charts/redis-ha v4.23.0