KeyDB

KeyDB copied to clipboard

Describe the bug

When nodes under heavy load - replication is not working

To reproduce

3 nodes:

client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 8129mb 8192mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
server-threads 14
active-replica yes
multi-master yes

node 1: 
replicaof km02 6389
replicaof km03 6389

node 2:
replicaof km01 6389
replicaof km03 6389

node 3:
replicaof km01 6389
replicaof km02 6389

haproxy is load-balance 6379/tcp on 3 nodes:

root@km01:~# cat /etc/haproxy/haproxy.d/6379.cfg 
listen keydb_6379
       bind *:6379
       mode tcp
       option dontlog-normal
       timeout connect 2s
       timeout client  600s
       timeout server  120s
       timeout client-fin 2s
       timeout server-fin 2s
#      option tcplog
#      balance leastconn
       balance roundrobin
       maxconn 1000000
       fullconn 1000000

       server km01 km01:6389 check 
       server km02 km02:6389 check
       server km03 km03:6389 check

other host simulate heavy load by haproxy port 6379:

# memtier_benchmark -s km01 -p6379 -Predis -c 1 -t 1000 -n 10000 --ratio 2:1 --data-size-range=4-512 --data-size-pattern=S --key-minimum=100 --key-maximum=300 --key-pattern=G:G --key-stddev=10 --key-median=200 --hide-histogram -x 5

other host check:

while (true); do for h in {01..03}; do echo "km$h: "; redis-cli -h km$h -p 6389 get test1; done; echo "------------"; sleep 1; done

other host - direct action on each keydb-server by port 6389:

redis-cli -h km01 -p 6389 set test1 "AAAAAA"
redis-cli -h km02 -p 6389 set test1 "BBBBBB"
redis-cli -h km03 -p 6389 set test1 "ZZZZZ"

In result: Data syncronization is slow and unpredicable:

km01: "BBBBBB"
km02: "ZZZZZ"
km03: "BBBBBB"

Without heavy load data syncronization is predicable and correct

Expected behavior Under heavy load data syncronization is fast and values is predicable and correct

Additional information

root@km01:~# egrep -v '^#|^$' /etc/keydb/keydb.conf 
bind 0.0.0.0
protected-mode yes
port 6389
tcp-backlog 4096
timeout 0
tcp-keepalive 300
daemonize yes
pidfile /var/run/keydb/keydb-server.pid
loglevel notice
logfile /var/log/keydb/keydb-server.log
databases 16
always-show-logo yes
set-proc-title yes
proc-title-template "{title} {listen-addr} {server-mode}"
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
rdb-del-sync-files no
dir /var/lib/keydb
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-diskless-load disabled
repl-disable-tcp-nodelay no
replica-priority 100
acllog-max-len 128
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
lazyfree-lazy-user-del no
lazyfree-lazy-user-flush no
oom-score-adj no
oom-score-adj-values 0 200 800
disable-thp yes
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 8129mb 8192mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes
jemalloc-bg-thread yes
server-threads 14
active-replica yes
multi-master yes

Hi @JohnSully ! Do you have any advice on this?

Hi @rnz does synchronization complete successfully after the heavy load subsides or does it remain out of sync?

@MalavanEQAlpha after the heavy load subsides it remain out of sync.

Is there any progress on this?

Additional info:

I try to migrate standalone node (6.0.18) to 5 nodes multi-master (6.2.2) Which configuration is right? I use next: https://paste.debian.net/plain/1264279 I asking because I can't make stable replication beetwen nodes. I tried multi-master-no-forward no|yes - both is not stable. What I did: I run 5 empty nodes in multi-master scheme and on one of them I add replicaof on standalone keydb (dataset ~4G, ~17mln keys, 100-150 kops). After few minutes nodes in multi-master randomly

"== WARNING == This replica is rejecting a command from its master: '-LOADING KeyDB is loading the dataset in memory' after processing the command 'KEYDB.MVCCRESTORE'"

In a few more minutes, all the masters in down, because the base is loaded into memory on variable nodes:

master_global_link_status:down
master_host:rdb05-prod  #<-- (standalone node)
master_link_status:down
master_1_host:kdb15-prod
master_1_link_status:down
master_2_host:kdb12-prod
master_2_link_status:down
master_3_host:kdb13-prod
master_3_link_status:up
master_4_host:kdb14-prod
master_4_link_status:down

HI, as discussed in another issue this is not a supported configuration, going to close this issue.

where is a link to other discussion?