KeyDB
KeyDB copied to clipboard
Published
20 hours ago •
Snapchat
Snapchat
[CRASH] MultiMaster Cluster cascade failure
trafficstars
Crash report
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.047 # Server initialized
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.047 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.048 * Loading RDB produced by version 6.3.1
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.048 * RDB age 2 seconds
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.048 * RDB memory usage when created 124.49 Mb
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 # Done loading RDB, keys loaded: 0, keys expired: 0.
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * DB loaded from disk: 0.062 seconds
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * RDB contains information on 21 masters
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * Cached master recovered from RDB for 172.28.69.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * Cached master recovered from RDB for 172.28.71.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * Cached master recovered from RDB for 172.28.70.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * Cached master recovered from RDB for 172.28.72.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * Cached master recovered from RDB for 172.28.39.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * Cached master recovered from RDB for 172.28.62.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * Cached master recovered from RDB for 172.28.47.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * Cached master recovered from RDB for 172.28.26.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * Cached master recovered from RDB for 172.28.27.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * Cached master recovered from RDB for 172.28.28.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * Cached master recovered from RDB for 172.28.53.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * Cached master recovered from RDB for 172.28.54.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * Cached master recovered from RDB for 172.28.55.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * Cached master recovered from RDB for 172.28.73.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * Cached master recovered from RDB for 172.28.74.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * Cached master recovered from RDB for 172.28.207.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * Cached master recovered from RDB for 172.28.208.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * Cached master recovered from RDB for 172.28.204.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * Cached master recovered from RDB for 172.28.205.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * Cached master recovered from RDB for 172.28.206.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 * Cached master recovered from RDB for 172.28.211.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:1:S 21 Oct 2022 12:39:19.110 # Warning: server-threads is set to 8. This is above the maximum recommend value of 4, please ensure you've verified this is actually faster on your machine.
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:23:S 21 Oct 2022 12:39:19.110 * Thread 1 alive.
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:19.110 * Thread 0 alive.
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:24:S 21 Oct 2022 12:39:19.110 * Thread 2 alive.
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:25:S 21 Oct 2022 12:39:19.110 * Thread 3 alive.
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:26:S 21 Oct 2022 12:39:19.110 * Thread 4 alive.
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:27:S 21 Oct 2022 12:39:19.110 * Thread 5 alive.
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:28:S 21 Oct 2022 12:39:19.110 * Thread 6 alive.
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:29:S 21 Oct 2022 12:39:19.110 * Thread 7 alive.
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:19.111 * Connecting to MASTER 172.28.69.1:6389
Oct 21 15:39:19 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:19.111 * MASTER <-> REPLICA sync started
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.113 * Non blocking connect for SYNC fired the event.
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.113 * Master replied to PING, replication can continue...
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.114 * Trying a partial resynchronization (request 05345317e97c9ef997fb52f334aff5a3dc2c644c:48870168333).
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.115 * Full resync from master: eff5b08d91b7d8fa3e5af917609c346d9509c378:49061727116
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.115 * Discarding previously cached master state.
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.269 * MASTER <-> REPLICA sync: receiving 11823499 bytes from master to disk
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.550 * MASTER <-> REPLICA sync: Loading DB in memory
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.551 * Loading RDB produced by version 6.3.1
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.551 * RDB age 0 seconds
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.551 * RDB memory usage when created 98.95 Mb
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 # Done loading RDB, keys loaded: 0, keys expired: 0.
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * RDB contains information on 21 masters
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * Cached master recovered from RDB for 172.28.71.1:6389
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * Cached master recovered from RDB for 172.28.70.1:6389
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * Cached master recovered from RDB for 172.28.72.1:6389
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * Cached master recovered from RDB for 172.28.39.1:6389
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * Cached master recovered from RDB for 172.28.62.1:6389
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * Cached master recovered from RDB for 172.28.47.1:6389
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * Cached master recovered from RDB for 172.28.26.1:6389
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * Cached master recovered from RDB for 172.28.27.1:6389
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * Cached master recovered from RDB for 172.28.28.1:6389
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * Cached master recovered from RDB for 172.28.53.1:6389
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * Cached master recovered from RDB for 172.28.54.1:6389
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * Cached master recovered from RDB for 172.28.55.1:6389
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * Cached master recovered from RDB for 172.28.73.1:6389
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * Cached master recovered from RDB for 172.28.74.1:6389
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * Cached master recovered from RDB for 172.28.207.1:6389
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * Cached master recovered from RDB for 172.28.208.1:6389
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * Cached master recovered from RDB for 172.28.204.1:6389
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * Cached master recovered from RDB for 172.28.205.1:6389
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * Cached master recovered from RDB for 172.28.206.1:6389
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * Cached master recovered from RDB for 172.28.211.1:6389
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.602 * Cached master recovered from RDB for 172.28.69.1:6389
Oct 21 15:39:20 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:20.604 * MASTER <-> REPLICA sync: Finished with success
Oct 21 15:39:21 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:21.024 * Connecting to MASTER 172.28.71.1:6389
Oct 21 15:39:21 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:21.024 * MASTER <-> REPLICA sync started
Oct 21 15:39:21 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:21.024 * Non blocking connect for SYNC fired the event.
Oct 21 15:39:21 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:21.025 * Master replied to PING, replication can continue...
Oct 21 15:39:21 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:21.026 * Trying a partial resynchronization (request 508f0762b487ac1e95472455aa9fe31593afb750:0).
Oct 21 15:39:21 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:21.026 * Full resync from master: 508f0762b487ac1e95472455aa9fe31593afb750:48924810898
Oct 21 15:39:21 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:21.026 * Discarding previously cached master state.
Oct 21 15:39:21 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:21.103 * MASTER <-> REPLICA sync: receiving 11824603 bytes from master to disk
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.344 * MASTER <-> REPLICA sync: Loading DB in memory
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.345 * Loading RDB produced by version 6.3.1
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.345 * RDB age 2 seconds
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.345 * RDB memory usage when created 83.52 Mb
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 # Done loading RDB, keys loaded: 0, keys expired: 0.
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 * RDB contains information on 21 masters
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 * Cached master recovered from RDB for 172.28.70.1:6389
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 * Cached master recovered from RDB for 172.28.72.1:6389
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 * Cached master recovered from RDB for 172.28.39.1:6389
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 * Cached master recovered from RDB for 172.28.62.1:6389
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 * Cached master recovered from RDB for 172.28.47.1:6389
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 * Cached master recovered from RDB for 172.28.26.1:6389
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 * Cached master recovered from RDB for 172.28.27.1:6389
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 * Cached master recovered from RDB for 172.28.28.1:6389
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 * Cached master recovered from RDB for 172.28.53.1:6389
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 * Cached master recovered from RDB for 172.28.54.1:6389
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 * Cached master recovered from RDB for 172.28.55.1:6389
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 * Cached master recovered from RDB for 172.28.73.1:6389
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 * Cached master recovered from RDB for 172.28.74.1:6389
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 * Cached master recovered from RDB for 172.28.207.1:6389
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 * Cached master recovered from RDB for 172.28.208.1:6389
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 * Cached master recovered from RDB for 172.28.204.1:6389
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 * Cached master recovered from RDB for 172.28.205.1:6389
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 * Cached master recovered from RDB for 172.28.206.1:6389
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 * Cached master recovered from RDB for 172.28.211.1:6389
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.389 * Cached master recovered from RDB for 172.28.71.1:6389
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.391 * MASTER <-> REPLICA sync: Finished with success
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.994 * Connecting to MASTER 172.28.70.1:6389
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.994 * MASTER <-> REPLICA sync started
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.996 * Non blocking connect for SYNC fired the event.
Oct 21 15:39:22 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:22.998 * Master replied to PING, replication can continue...
Oct 21 15:39:23 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:23.002 * Trying a partial resynchronization (request 19089be7dfcbd3a09a40dcc27fb9244db2374f65:62563356).
Oct 21 15:39:23 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:23.004 * Full resync from master: 8da4da7b8d135ed9a2e1d11196c8d03b16c9a106:54655558282
Oct 21 15:39:23 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:23.004 * Discarding previously cached master state.
Oct 21 15:39:23 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:23.135 * MASTER <-> REPLICA sync: receiving 11824599 bytes from master to disk
Oct 21 15:39:24 z27 docker/keydb-de[22018]: 1:23:S 21 Oct 2022 12:39:24.267 * Replica 172.28.28.6:6379 asks for synchronization
Oct 21 15:39:24 z27 docker/keydb-de[22018]: 1:23:S 21 Oct 2022 12:39:24.267 * Unable to partial resync with replica 172.28.28.6:6379 for lack of backlog (Replica request was: 1243856390).
Oct 21 15:39:24 z27 docker/keydb-de[22018]: 1:23:S 21 Oct 2022 12:39:24.267 * Starting BGSAVE for SYNC with target: disk
Oct 21 15:39:24 z27 docker/keydb-de[22018]: 1:23:S 21 Oct 2022 12:39:24.268 * Background saving started
Oct 21 15:39:24 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:24.299 # Replication backlog is too small, resizing to: 2097152 bytes
Oct 21 15:39:24 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:24.364 # Replication backlog is too small, resizing to: 4194304 bytes
Oct 21 15:39:24 z27 docker/keydb-de[22018]: 1:30:S 21 Oct 2022 12:39:24.393 * DB saved on disk
Oct 21 15:39:24 z27 docker/keydb-de[22018]: 1:30:S 21 Oct 2022 12:39:24.398 * RDB: 92 MB of memory used by copy-on-write
Oct 21 15:39:24 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:24.401 # Replication backlog is too small, resizing to: 8388608 bytes
Oct 21 15:39:24 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:24.435 * Background saving terminated with success
Oct 21 15:39:24 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:24.486 # Replication backlog is too small, resizing to: 16777216 bytes
Oct 21 15:39:24 z27 docker/keydb-de[22018]: 1:23:S 21 Oct 2022 12:39:24.559 * Synchronization with replica 172.28.28.6:6379 succeeded
Oct 21 15:39:24 z27 docker/keydb-de[22018]: 1:23:S 21 Oct 2022 12:39:24.604 * Replica 172.28.54.6:6379 asks for synchronization
Oct 21 15:39:24 z27 docker/keydb-de[22018]: 1:23:S 21 Oct 2022 12:39:24.604 * Unable to partial resync with replica 172.28.54.6:6379 for lack of backlog (Replica request was: 1243856390).
Oct 21 15:39:24 z27 docker/keydb-de[22018]: 1:23:S 21 Oct 2022 12:39:24.604 * Starting BGSAVE for SYNC with target: disk
Oct 21 15:39:24 z27 docker/keydb-de[22018]: 1:23:S 21 Oct 2022 12:39:24.604 * Background saving started
Oct 21 15:39:24 z27 docker/keydb-de[22018]: 1:31:S 21 Oct 2022 12:39:24.711 * DB saved on disk
Oct 21 15:39:24 z27 docker/keydb-de[22018]: 1:31:S 21 Oct 2022 12:39:24.714 * RDB: 105 MB of memory used by copy-on-write
Oct 21 15:39:24 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:24.737 * Background saving terminated with success
Oct 21 15:39:24 z27 docker/keydb-de[22018]: 1:23:S 21 Oct 2022 12:39:24.983 * Synchronization with replica 172.28.54.6:6379 succeeded
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.027 * MASTER <-> REPLICA sync: Loading DB in memory
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.028 * Loading RDB produced by version 6.3.1
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.028 * RDB age 3 seconds
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.028 * RDB memory usage when created 117.03 Mb
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.221 # Done loading RDB, keys loaded: 0, keys expired: 0.
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.221 * RDB contains information on 21 masters
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.221 * Cached master recovered from RDB for 172.28.72.1:6389
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.221 * Cached master recovered from RDB for 172.28.39.1:6389
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.221 * Cached master recovered from RDB for 172.28.62.1:6389
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.221 * Cached master recovered from RDB for 172.28.47.1:6389
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.221 * Cached master recovered from RDB for 172.28.26.1:6389
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.222 * Cached master recovered from RDB for 172.28.27.1:6389
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.222 * Cached master recovered from RDB for 172.28.28.1:6389
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.222 * Cached master recovered from RDB for 172.28.53.1:6389
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.222 * Cached master recovered from RDB for 172.28.54.1:6389
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.222 * Cached master recovered from RDB for 172.28.55.1:6389
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.222 * Cached master recovered from RDB for 172.28.73.1:6389
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.222 * Cached master recovered from RDB for 172.28.74.1:6389
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.222 * Cached master recovered from RDB for 172.28.207.1:6389
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.222 * Cached master recovered from RDB for 172.28.208.1:6389
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.222 * Cached master recovered from RDB for 172.28.204.1:6389
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.222 * Cached master recovered from RDB for 172.28.205.1:6389
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.222 * Cached master recovered from RDB for 172.28.206.1:6389
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.222 * Cached master recovered from RDB for 172.28.211.1:6389
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.222 * Cached master recovered from RDB for 172.28.70.1:6389
Oct 21 15:39:25 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:25.223 * MASTER <-> REPLICA sync: Finished with success
Oct 21 15:39:26 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:26.026 * Connecting to MASTER 172.28.72.1:6389
Oct 21 15:39:26 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:26.026 * MASTER <-> REPLICA sync started
Oct 21 15:39:26 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:26.026 * Reclaiming 15728640 replication backlog bytes
Oct 21 15:39:26 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:26.028 * Non blocking connect for SYNC fired the event.
Oct 21 15:39:26 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:26.030 * Master replied to PING, replication can continue...
Oct 21 15:39:26 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:26.032 * Trying a partial resynchronization (request a557cab1a2db8a2727bd09ee99b5ac048f78ed59:602099279).
Oct 21 15:39:26 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:26.034 * Full resync from master: bfa2e959907ee71a3701e14971786ff1d96504c6:61723621926
Oct 21 15:39:26 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:26.034 * Discarding previously cached master state.
Oct 21 15:39:26 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:26.172 * MASTER <-> REPLICA sync: receiving 11824837 bytes from master to disk
Oct 21 15:39:26 z27 docker/keydb-de[22018]: 1:24:S 21 Oct 2022 12:39:26.611 * Replica 172.28.73.7:6379 asks for synchronization
Oct 21 15:39:26 z27 docker/keydb-de[22018]: 1:24:S 21 Oct 2022 12:39:26.611 * Unable to partial resync with replica 172.28.73.7:6379 for lack of backlog (Replica request was: 1243856390).
Oct 21 15:39:26 z27 docker/keydb-de[22018]: 1:24:S 21 Oct 2022 12:39:26.611 * Starting BGSAVE for SYNC with target: disk
Oct 21 15:39:26 z27 docker/keydb-de[22018]: 1:24:S 21 Oct 2022 12:39:26.611 * Background saving started
Oct 21 15:39:26 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:26.678 # Replication backlog is too small, resizing to: 2097152 bytes
Oct 21 15:39:26 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:26.704 # Replication backlog is too small, resizing to: 4194304 bytes
Oct 21 15:39:26 z27 docker/keydb-de[22018]: 1:32:S 21 Oct 2022 12:39:26.729 * DB saved on disk
Oct 21 15:39:26 z27 docker/keydb-de[22018]: 1:32:S 21 Oct 2022 12:39:26.732 * RDB: 99 MB of memory used by copy-on-write
Oct 21 15:39:26 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:26.735 * Background saving terminated with success
Oct 21 15:39:26 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:26.771 # Replication backlog is too small, resizing to: 8388608 bytes
Oct 21 15:39:26 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:26.893 # Replication backlog is too small, resizing to: 16777216 bytes
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:24:S 21 Oct 2022 12:39:27.107 * Synchronization with replica 172.28.73.7:6379 succeeded
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.473 * MASTER <-> REPLICA sync: Loading DB in memory
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.474 * Loading RDB produced by version 6.3.1
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.474 * RDB age 1 seconds
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.474 * RDB memory usage when created 154.87 Mb
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.493 # == WARNING == This replica is rejecting a command from its master: '-LOADING KeyDB is loading the dataset in memory' after processing the command 'KEYDB.MVCCRESTORE'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.493 # Latest backlog is: '"\t\xa5\b\tteleorma\xe6 _\x006E\x19\x00\xc9@\xa6\x00T\xc0:\xe0\x1d\xa6\x00n\xe0\x03\xa6\x0690\xe8R\x88\x9d\xf5`\xa6\x04\b\xc435\xb7\xe2\x02\x90\x00\xe3\xe0\t\xa6\x03\x05ilf\xf4\x00\xe4\xe5\x18\xbc\x007E\x18\x00\xcc \xa2\x01\x05I@6\xe0\x1d\x9e\x00o\xe0\x03\x9e\x06:>\xb2D\x90D\xa1`\x9e\x05@7\xf2\xb5\x94\xa2\xe1\x01E\x00\xe7\xe0\t\x9e\x01\bh2\x06\x00h\xe1\"\xeb\x007E\x1a\x00\xcf \xa1\x01\bH\xa09\xe0\x1d\xa4\x00p\xe0\x03\xa4\x069\x9e\xbd\x9f(\xa7\x9d@\xa4\x06GRq\xb3\xa1\x82\x9c\xe0\x01\xa4\x00\xeb\xe0\x06\xa4\x01\x01\x06\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00k\xf0#\xb8t\xc2!^\r\n\r\n$1\r\n0\r\n$19\r\n1747300874968891392\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.493 # == CRITICAL == This replica is sending an error to its master: 'command did not execute' after processing the command 'rreplay'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.493 # Latest backlog is: '"\t\xa5\b\tteleorma\xe6 _\x006E\x19\x00\xc9@\xa6\x00T\xc0:\xe0\x1d\xa6\x00n\xe0\x03\xa6\x0690\xe8R\x88\x9d\xf5`\xa6\x04\b\xc435\xb7\xe2\x02\x90\x00\xe3\xe0\t\xa6\x03\x05ilf\xf4\x00\xe4\xe5\x18\xbc\x007E\x18\x00\xcc \xa2\x01\x05I@6\xe0\x1d\x9e\x00o\xe0\x03\x9e\x06:>\xb2D\x90D\xa1`\x9e\x05@7\xf2\xb5\x94\xa2\xe1\x01E\x00\xe7\xe0\t\x9e\x01\bh2\x06\x00h\xe1\"\xeb\x007E\x1a\x00\xcf \xa1\x01\bH\xa09\xe0\x1d\xa4\x00p\xe0\x03\xa4\x069\x9e\xbd\x9f(\xa7\x9d@\xa4\x06GRq\xb3\xa1\x82\x9c\xe0\x01\xa4\x00\xeb\xe0\x06\xa4\x01\x01\x06\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00k\xf0#\xb8t\xc2!^\r\n\r\n$1\r\n0\r\n$19\r\n1747300874968891392\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.493 # == WARNING == This replica is rejecting a command from its master: '-LOADING KeyDB is loading the dataset in memory' after processing the command 'KEYDB.MVCCRESTORE'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.493 # Latest backlog is: '"\t\xa5\b\tteleorma\xe6 _\x006E\x19\x00\xc9@\xa6\x00T\xc0:\xe0\x1d\xa6\x00n\xe0\x03\xa6\x0690\xe8R\x88\x9d\xf5`\xa6\x04\b\xc435\xb7\xe2\x02\x90\x00\xe3\xe0\t\xa6\x03\x05ilf\xf4\x00\xe4\xe5\x18\xbc\x007E\x18\x00\xcc \xa2\x01\x05I@6\xe0\x1d\x9e\x00o\xe0\x03\x9e\x06:>\xb2D\x90D\xa1`\x9e\x05@7\xf2\xb5\x94\xa2\xe1\x01E\x00\xe7\xe0\t\x9e\x01\bh2\x06\x00h\xe1\"\xeb\x007E\x1a\x00\xcf \xa1\x01\bH\xa09\xe0\x1d\xa4\x00p\xe0\x03\xa4\x069\x9e\xbd\x9f(\xa7\x9d@\xa4\x06GRq\xb3\xa1\x82\x9c\xe0\x01\xa4\x00\xeb\xe0\x06\xa4\x01\x01\x06\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00k\xf0#\xb8t\xc2!^\r\n\r\n$1\r\n0\r\n$19\r\n1747300874968891392\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.493 # == CRITICAL == This replica is sending an error to its master: 'command did not execute' after processing the command 'rreplay'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.493 # Latest backlog is: '"\t\xa5\b\tteleorma\xe6 _\x006E\x19\x00\xc9@\xa6\x00T\xc0:\xe0\x1d\xa6\x00n\xe0\x03\xa6\x0690\xe8R\x88\x9d\xf5`\xa6\x04\b\xc435\xb7\xe2\x02\x90\x00\xe3\xe0\t\xa6\x03\x05ilf\xf4\x00\xe4\xe5\x18\xbc\x007E\x18\x00\xcc \xa2\x01\x05I@6\xe0\x1d\x9e\x00o\xe0\x03\x9e\x06:>\xb2D\x90D\xa1`\x9e\x05@7\xf2\xb5\x94\xa2\xe1\x01E\x00\xe7\xe0\t\x9e\x01\bh2\x06\x00h\xe1\"\xeb\x007E\x1a\x00\xcf \xa1\x01\bH\xa09\xe0\x1d\xa4\x00p\xe0\x03\xa4\x069\x9e\xbd\x9f(\xa7\x9d@\xa4\x06GRq\xb3\xa1\x82\x9c\xe0\x01\xa4\x00\xeb\xe0\x06\xa4\x01\x01\x06\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00k\xf0#\xb8t\xc2!^\r\n\r\n$1\r\n0\r\n$19\r\n1747300874968891392\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.544 # Replication backlog is too small, resizing to: 33554432 bytes
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # == WARNING == This replica is rejecting a command from its master: '-LOADING KeyDB is loading the dataset in memory' after processing the command 'KEYDB.MVCCRESTORE'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # Latest backlog is: '"nor#l%\xc7\x05aegean\xe3\bp\x00\xb6 \xb4\x00\xee \xb4\x00\xb4\xe0\x1d\xb4\x00\x82\xe0\x03\xb4\x06:w\xa9\xcf\x92\xd9^`\xb4\x05\xd4\x9d7Hk\xaa@\xb4\n\x1b\x05\x0e\x17\x0e\x18\x0e\"\x02\x01\x01\xe0\x18\x9d\x02sou\xe0\x14\x9d\x00\x86 \x9d\x00\xef \x9d\x00\x84\xe0\x1d\x9d\x00\x83\xe0\x03\x9d\x069t\x7f\xeaH\xfb\x1b`\x9d\x05\x936\x94|#g\xe0\x05\x9d\x00\x05\xe0\t\x9d\x01\x05c4}\xc1\xd8\xec\x00\xbb4\x1b\x02\a\x0e\xf0 \x85\x006\xe0\x1d\x85\x00\x84\xe0\x03\x85\x068v\x9d\x9a(z\xc7@\x85\x06A\xa7|\xc5M|\x8b\xe0\x05\x85\x00\t\xe0\x06\x85\x01\x01\a\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00\xb1\x9e\x0bs\x06)\x0e\xbf\r\n\r\n$1\r\n0\r\n$19\r\n1747300875077943302\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: #024#002#006
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # == CRITICAL == This replica is sending an error to its master: 'command did not execute' after processing the command 'rreplay'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # Latest backlog is: '"nor#l%\xc7\x05aegean\xe3\bp\x00\xb6 \xb4\x00\xee \xb4\x00\xb4\xe0\x1d\xb4\x00\x82\xe0\x03\xb4\x06:w\xa9\xcf\x92\xd9^`\xb4\x05\xd4\x9d7Hk\xaa@\xb4\n\x1b\x05\x0e\x17\x0e\x18\x0e\"\x02\x01\x01\xe0\x18\x9d\x02sou\xe0\x14\x9d\x00\x86 \x9d\x00\xef \x9d\x00\x84\xe0\x1d\x9d\x00\x83\xe0\x03\x9d\x069t\x7f\xeaH\xfb\x1b`\x9d\x05\x936\x94|#g\xe0\x05\x9d\x00\x05\xe0\t\x9d\x01\x05c4}\xc1\xd8\xec\x00\xbb4\x1b\x02\a\x0e\xf0 \x85\x006\xe0\x1d\x85\x00\x84\xe0\x03\x85\x068v\x9d\x9a(z\xc7@\x85\x06A\xa7|\xc5M|\x8b\xe0\x05\x85\x00\t\xe0\x06\x85\x01\x01\a\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00\xb1\x9e\x0bs\x06)\x0e\xbf\r\n\r\n$1\r\n0\r\n$19\r\n1747300875077943302\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # == WARNING == This replica is rejecting a command from its master: '-LOADING KeyDB is loading the dataset in memory' after processing the command 'KEYDB.MVCCRESTORE'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # Latest backlog is: '"nor#l%\xc7\x05aegean\xe3\bp\x00\xb6 \xb4\x00\xee \xb4\x00\xb4\xe0\x1d\xb4\x00\x82\xe0\x03\xb4\x06:w\xa9\xcf\x92\xd9^`\xb4\x05\xd4\x9d7Hk\xaa@\xb4\n\x1b\x05\x0e\x17\x0e\x18\x0e\"\x02\x01\x01\xe0\x18\x9d\x02sou\xe0\x14\x9d\x00\x86 \x9d\x00\xef \x9d\x00\x84\xe0\x1d\x9d\x00\x83\xe0\x03\x9d\x069t\x7f\xeaH\xfb\x1b`\x9d\x05\x936\x94|#g\xe0\x05\x9d\x00\x05\xe0\t\x9d\x01\x05c4}\xc1\xd8\xec\x00\xbb4\x1b\x02\a\x0e\xf0 \x85\x006\xe0\x1d\x85\x00\x84\xe0\x03\x85\x068v\x9d\x9a(z\xc7@\x85\x06A\xa7|\xc5M|\x8b\xe0\x05\x85\x00\t\xe0\x06\x85\x01\x01\a\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00\xb1\x9e\x0bs\x06)\x0e\xbf\r\n\r\n$1\r\n0\r\n$19\r\n1747300875077943302\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: #024#002#006
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # == CRITICAL == This replica is sending an error to its master: 'command did not execute' after processing the command 'rreplay'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # Latest backlog is: '"nor#l%\xc7\x05aegean\xe3\bp\x00\xb6 \xb4\x00\xee \xb4\x00\xb4\xe0\x1d\xb4\x00\x82\xe0\x03\xb4\x06:w\xa9\xcf\x92\xd9^`\xb4\x05\xd4\x9d7Hk\xaa@\xb4\n\x1b\x05\x0e\x17\x0e\x18\x0e\"\x02\x01\x01\xe0\x18\x9d\x02sou\xe0\x14\x9d\x00\x86 \x9d\x00\xef \x9d\x00\x84\xe0\x1d\x9d\x00\x83\xe0\x03\x9d\x069t\x7f\xeaH\xfb\x1b`\x9d\x05\x936\x94|#g\xe0\x05\x9d\x00\x05\xe0\t\x9d\x01\x05c4}\xc1\xd8\xec\x00\xbb4\x1b\x02\a\x0e\xf0 \x85\x006\xe0\x1d\x85\x00\x84\xe0\x03\x85\x068v\x9d\x9a(z\xc7@\x85\x06A\xa7|\xc5M|\x8b\xe0\x05\x85\x00\t\xe0\x06\x85\x01\x01\a\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00\xb1\x9e\x0bs\x06)\x0e\xbf\r\n\r\n$1\r\n0\r\n$19\r\n1747300875077943302\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # == WARNING == This replica is rejecting a command from its master: '-LOADING KeyDB is loading the dataset in memory' after processing the command 'KEYDB.MVCCRESTORE'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # Latest backlog is: '"nor#l%\xc7\x05aegean\xe3\bp\x00\xb6 \xb4\x00\xee \xb4\x00\xb4\xe0\x1d\xb4\x00\x82\xe0\x03\xb4\x06:w\xa9\xcf\x92\xd9^`\xb4\x05\xd4\x9d7Hk\xaa@\xb4\n\x1b\x05\x0e\x17\x0e\x18\x0e\"\x02\x01\x01\xe0\x18\x9d\x02sou\xe0\x14\x9d\x00\x86 \x9d\x00\xef \x9d\x00\x84\xe0\x1d\x9d\x00\x83\xe0\x03\x9d\x069t\x7f\xeaH\xfb\x1b`\x9d\x05\x936\x94|#g\xe0\x05\x9d\x00\x05\xe0\t\x9d\x01\x05c4}\xc1\xd8\xec\x00\xbb4\x1b\x02\a\x0e\xf0 \x85\x006\xe0\x1d\x85\x00\x84\xe0\x03\x85\x068v\x9d\x9a(z\xc7@\x85\x06A\xa7|\xc5M|\x8b\xe0\x05\x85\x00\t\xe0\x06\x85\x01\x01\a\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00\xb1\x9e\x0bs\x06)\x0e\xbf\r\n\r\n$1\r\n0\r\n$19\r\n1747300875077943302\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: #024#002#006
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # == CRITICAL == This replica is sending an error to its master: 'command did not execute' after processing the command 'rreplay'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # Latest backlog is: '"nor#l%\xc7\x05aegean\xe3\bp\x00\xb6 \xb4\x00\xee \xb4\x00\xb4\xe0\x1d\xb4\x00\x82\xe0\x03\xb4\x06:w\xa9\xcf\x92\xd9^`\xb4\x05\xd4\x9d7Hk\xaa@\xb4\n\x1b\x05\x0e\x17\x0e\x18\x0e\"\x02\x01\x01\xe0\x18\x9d\x02sou\xe0\x14\x9d\x00\x86 \x9d\x00\xef \x9d\x00\x84\xe0\x1d\x9d\x00\x83\xe0\x03\x9d\x069t\x7f\xeaH\xfb\x1b`\x9d\x05\x936\x94|#g\xe0\x05\x9d\x00\x05\xe0\t\x9d\x01\x05c4}\xc1\xd8\xec\x00\xbb4\x1b\x02\a\x0e\xf0 \x85\x006\xe0\x1d\x85\x00\x84\xe0\x03\x85\x068v\x9d\x9a(z\xc7@\x85\x06A\xa7|\xc5M|\x8b\xe0\x05\x85\x00\t\xe0\x06\x85\x01\x01\a\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00\xb1\x9e\x0bs\x06)\x0e\xbf\r\n\r\n$1\r\n0\r\n$19\r\n1747300875077943302\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # == WARNING == This replica is rejecting a command from its master: '-LOADING KeyDB is loading the dataset in memory' after processing the command 'KEYDB.MVCCRESTORE'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # Latest backlog is: '"nor#l%\xc7\x05aegean\xe3\bp\x00\xb6 \xb4\x00\xee \xb4\x00\xb4\xe0\x1d\xb4\x00\x82\xe0\x03\xb4\x06:w\xa9\xcf\x92\xd9^`\xb4\x05\xd4\x9d7Hk\xaa@\xb4\n\x1b\x05\x0e\x17\x0e\x18\x0e\"\x02\x01\x01\xe0\x18\x9d\x02sou\xe0\x14\x9d\x00\x86 \x9d\x00\xef \x9d\x00\x84\xe0\x1d\x9d\x00\x83\xe0\x03\x9d\x069t\x7f\xeaH\xfb\x1b`\x9d\x05\x936\x94|#g\xe0\x05\x9d\x00\x05\xe0\t\x9d\x01\x05c4}\xc1\xd8\xec\x00\xbb4\x1b\x02\a\x0e\xf0 \x85\x006\xe0\x1d\x85\x00\x84\xe0\x03\x85\x068v\x9d\x9a(z\xc7@\x85\x06A\xa7|\xc5M|\x8b\xe0\x05\x85\x00\t\xe0\x06\x85\x01\x01\a\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00\xb1\x9e\x0bs\x06)\x0e\xbf\r\n\r\n$1\r\n0\r\n$19\r\n1747300875077943302\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: #024#002#006
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # == CRITICAL == This replica is sending an error to its master: 'command did not execute' after processing the command 'rreplay'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # Latest backlog is: '"nor#l%\xc7\x05aegean\xe3\bp\x00\xb6 \xb4\x00\xee \xb4\x00\xb4\xe0\x1d\xb4\x00\x82\xe0\x03\xb4\x06:w\xa9\xcf\x92\xd9^`\xb4\x05\xd4\x9d7Hk\xaa@\xb4\n\x1b\x05\x0e\x17\x0e\x18\x0e\"\x02\x01\x01\xe0\x18\x9d\x02sou\xe0\x14\x9d\x00\x86 \x9d\x00\xef \x9d\x00\x84\xe0\x1d\x9d\x00\x83\xe0\x03\x9d\x069t\x7f\xeaH\xfb\x1b`\x9d\x05\x936\x94|#g\xe0\x05\x9d\x00\x05\xe0\t\x9d\x01\x05c4}\xc1\xd8\xec\x00\xbb4\x1b\x02\a\x0e\xf0 \x85\x006\xe0\x1d\x85\x00\x84\xe0\x03\x85\x068v\x9d\x9a(z\xc7@\x85\x06A\xa7|\xc5M|\x8b\xe0\x05\x85\x00\t\xe0\x06\x85\x01\x01\a\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00\xb1\x9e\x0bs\x06)\x0e\xbf\r\n\r\n$1\r\n0\r\n$19\r\n1747300875077943302\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # == WARNING == This replica is rejecting a command from its master: '-LOADING KeyDB is loading the dataset in memory' after processing the command 'KEYDB.MVCCRESTORE'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # Latest backlog is: '"nor#l%\xc7\x05aegean\xe3\bp\x00\xb6 \xb4\x00\xee \xb4\x00\xb4\xe0\x1d\xb4\x00\x82\xe0\x03\xb4\x06:w\xa9\xcf\x92\xd9^`\xb4\x05\xd4\x9d7Hk\xaa@\xb4\n\x1b\x05\x0e\x17\x0e\x18\x0e\"\x02\x01\x01\xe0\x18\x9d\x02sou\xe0\x14\x9d\x00\x86 \x9d\x00\xef \x9d\x00\x84\xe0\x1d\x9d\x00\x83\xe0\x03\x9d\x069t\x7f\xeaH\xfb\x1b`\x9d\x05\x936\x94|#g\xe0\x05\x9d\x00\x05\xe0\t\x9d\x01\x05c4}\xc1\xd8\xec\x00\xbb4\x1b\x02\a\x0e\xf0 \x85\x006\xe0\x1d\x85\x00\x84\xe0\x03\x85\x068v\x9d\x9a(z\xc7@\x85\x06A\xa7|\xc5M|\x8b\xe0\x05\x85\x00\t\xe0\x06\x85\x01\x01\a\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00\xb1\x9e\x0bs\x06)\x0e\xbf\r\n\r\n$1\r\n0\r\n$19\r\n1747300875077943302\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: #024#002#006
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # == CRITICAL == This replica is sending an error to its master: 'command did not execute' after processing the command 'rreplay'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # Latest backlog is: '"nor#l%\xc7\x05aegean\xe3\bp\x00\xb6 \xb4\x00\xee \xb4\x00\xb4\xe0\x1d\xb4\x00\x82\xe0\x03\xb4\x06:w\xa9\xcf\x92\xd9^`\xb4\x05\xd4\x9d7Hk\xaa@\xb4\n\x1b\x05\x0e\x17\x0e\x18\x0e\"\x02\x01\x01\xe0\x18\x9d\x02sou\xe0\x14\x9d\x00\x86 \x9d\x00\xef \x9d\x00\x84\xe0\x1d\x9d\x00\x83\xe0\x03\x9d\x069t\x7f\xeaH\xfb\x1b`\x9d\x05\x936\x94|#g\xe0\x05\x9d\x00\x05\xe0\t\x9d\x01\x05c4}\xc1\xd8\xec\x00\xbb4\x1b\x02\a\x0e\xf0 \x85\x006\xe0\x1d\x85\x00\x84\xe0\x03\x85\x068v\x9d\x9a(z\xc7@\x85\x06A\xa7|\xc5M|\x8b\xe0\x05\x85\x00\t\xe0\x06\x85\x01\x01\a\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00\xb1\x9e\x0bs\x06)\x0e\xbf\r\n\r\n$1\r\n0\r\n$19\r\n1747300875077943302\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # == WARNING == This replica is rejecting a command from its master: '-LOADING KeyDB is loading the dataset in memory' after processing the command 'KEYDB.MVCCRESTORE'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # Latest backlog is: '"nor#l%\xc7\x05aegean\xe3\bp\x00\xb6 \xb4\x00\xee \xb4\x00\xb4\xe0\x1d\xb4\x00\x82\xe0\x03\xb4\x06:w\xa9\xcf\x92\xd9^`\xb4\x05\xd4\x9d7Hk\xaa@\xb4\n\x1b\x05\x0e\x17\x0e\x18\x0e\"\x02\x01\x01\xe0\x18\x9d\x02sou\xe0\x14\x9d\x00\x86 \x9d\x00\xef \x9d\x00\x84\xe0\x1d\x9d\x00\x83\xe0\x03\x9d\x069t\x7f\xeaH\xfb\x1b`\x9d\x05\x936\x94|#g\xe0\x05\x9d\x00\x05\xe0\t\x9d\x01\x05c4}\xc1\xd8\xec\x00\xbb4\x1b\x02\a\x0e\xf0 \x85\x006\xe0\x1d\x85\x00\x84\xe0\x03\x85\x068v\x9d\x9a(z\xc7@\x85\x06A\xa7|\xc5M|\x8b\xe0\x05\x85\x00\t\xe0\x06\x85\x01\x01\a\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00\xb1\x9e\x0bs\x06)\x0e\xbf\r\n\r\n$1\r\n0\r\n$19\r\n1747300875077943302\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: #024#002#006
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # == CRITICAL == This replica is sending an error to its master: 'command did not execute' after processing the command 'rreplay'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # Latest backlog is: '"nor#l%\xc7\x05aegean\xe3\bp\x00\xb6 \xb4\x00\xee \xb4\x00\xb4\xe0\x1d\xb4\x00\x82\xe0\x03\xb4\x06:w\xa9\xcf\x92\xd9^`\xb4\x05\xd4\x9d7Hk\xaa@\xb4\n\x1b\x05\x0e\x17\x0e\x18\x0e\"\x02\x01\x01\xe0\x18\x9d\x02sou\xe0\x14\x9d\x00\x86 \x9d\x00\xef \x9d\x00\x84\xe0\x1d\x9d\x00\x83\xe0\x03\x9d\x069t\x7f\xeaH\xfb\x1b`\x9d\x05\x936\x94|#g\xe0\x05\x9d\x00\x05\xe0\t\x9d\x01\x05c4}\xc1\xd8\xec\x00\xbb4\x1b\x02\a\x0e\xf0 \x85\x006\xe0\x1d\x85\x00\x84\xe0\x03\x85\x068v\x9d\x9a(z\xc7@\x85\x06A\xa7|\xc5M|\x8b\xe0\x05\x85\x00\t\xe0\x06\x85\x01\x01\a\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00\xb1\x9e\x0bs\x06)\x0e\xbf\r\n\r\n$1\r\n0\r\n$19\r\n1747300875077943302\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # == WARNING == This replica is rejecting a command from its master: '-LOADING KeyDB is loading the dataset in memory' after processing the command 'KEYDB.MVCCRESTORE'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # Latest backlog is: '"nor#l%\xc7\x05aegean\xe3\bp\x00\xb6 \xb4\x00\xee \xb4\x00\xb4\xe0\x1d\xb4\x00\x82\xe0\x03\xb4\x06:w\xa9\xcf\x92\xd9^`\xb4\x05\xd4\x9d7Hk\xaa@\xb4\n\x1b\x05\x0e\x17\x0e\x18\x0e\"\x02\x01\x01\xe0\x18\x9d\x02sou\xe0\x14\x9d\x00\x86 \x9d\x00\xef \x9d\x00\x84\xe0\x1d\x9d\x00\x83\xe0\x03\x9d\x069t\x7f\xeaH\xfb\x1b`\x9d\x05\x936\x94|#g\xe0\x05\x9d\x00\x05\xe0\t\x9d\x01\x05c4}\xc1\xd8\xec\x00\xbb4\x1b\x02\a\x0e\xf0 \x85\x006\xe0\x1d\x85\x00\x84\xe0\x03\x85\x068v\x9d\x9a(z\xc7@\x85\x06A\xa7|\xc5M|\x8b\xe0\x05\x85\x00\t\xe0\x06\x85\x01\x01\a\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00\xb1\x9e\x0bs\x06)\x0e\xbf\r\n\r\n$1\r\n0\r\n$19\r\n1747300875077943302\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: #024#002#006
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # == CRITICAL == This replica is sending an error to its master: 'command did not execute' after processing the command 'rreplay'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # Latest backlog is: '"nor#l%\xc7\x05aegean\xe3\bp\x00\xb6 \xb4\x00\xee \xb4\x00\xb4\xe0\x1d\xb4\x00\x82\xe0\x03\xb4\x06:w\xa9\xcf\x92\xd9^`\xb4\x05\xd4\x9d7Hk\xaa@\xb4\n\x1b\x05\x0e\x17\x0e\x18\x0e\"\x02\x01\x01\xe0\x18\x9d\x02sou\xe0\x14\x9d\x00\x86 \x9d\x00\xef \x9d\x00\x84\xe0\x1d\x9d\x00\x83\xe0\x03\x9d\x069t\x7f\xeaH\xfb\x1b`\x9d\x05\x936\x94|#g\xe0\x05\x9d\x00\x05\xe0\t\x9d\x01\x05c4}\xc1\xd8\xec\x00\xbb4\x1b\x02\a\x0e\xf0 \x85\x006\xe0\x1d\x85\x00\x84\xe0\x03\x85\x068v\x9d\x9a(z\xc7@\x85\x06A\xa7|\xc5M|\x8b\xe0\x05\x85\x00\t\xe0\x06\x85\x01\x01\a\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00\xb1\x9e\x0bs\x06)\x0e\xbf\r\n\r\n$1\r\n0\r\n$19\r\n1747300875077943302\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # == WARNING == This replica is rejecting a command from its master: '-LOADING KeyDB is loading the dataset in memory' after processing the command 'KEYDB.MVCCRESTORE'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # Latest backlog is: '"nor#l%\xc7\x05aegean\xe3\bp\x00\xb6 \xb4\x00\xee \xb4\x00\xb4\xe0\x1d\xb4\x00\x82\xe0\x03\xb4\x06:w\xa9\xcf\x92\xd9^`\xb4\x05\xd4\x9d7Hk\xaa@\xb4\n\x1b\x05\x0e\x17\x0e\x18\x0e\"\x02\x01\x01\xe0\x18\x9d\x02sou\xe0\x14\x9d\x00\x86 \x9d\x00\xef \x9d\x00\x84\xe0\x1d\x9d\x00\x83\xe0\x03\x9d\x069t\x7f\xeaH\xfb\x1b`\x9d\x05\x936\x94|#g\xe0\x05\x9d\x00\x05\xe0\t\x9d\x01\x05c4}\xc1\xd8\xec\x00\xbb4\x1b\x02\a\x0e\xf0 \x85\x006\xe0\x1d\x85\x00\x84\xe0\x03\x85\x068v\x9d\x9a(z\xc7@\x85\x06A\xa7|\xc5M|\x8b\xe0\x05\x85\x00\t\xe0\x06\x85\x01\x01\a\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00\xb1\x9e\x0bs\x06)\x0e\xbf\r\n\r\n$1\r\n0\r\n$19\r\n1747300875077943302\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: #024#002#006
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # == CRITICAL == This replica is sending an error to its master: 'command did not execute' after processing the command 'rreplay'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # Latest backlog is: '"nor#l%\xc7\x05aegean\xe3\bp\x00\xb6 \xb4\x00\xee \xb4\x00\xb4\xe0\x1d\xb4\x00\x82\xe0\x03\xb4\x06:w\xa9\xcf\x92\xd9^`\xb4\x05\xd4\x9d7Hk\xaa@\xb4\n\x1b\x05\x0e\x17\x0e\x18\x0e\"\x02\x01\x01\xe0\x18\x9d\x02sou\xe0\x14\x9d\x00\x86 \x9d\x00\xef \x9d\x00\x84\xe0\x1d\x9d\x00\x83\xe0\x03\x9d\x069t\x7f\xeaH\xfb\x1b`\x9d\x05\x936\x94|#g\xe0\x05\x9d\x00\x05\xe0\t\x9d\x01\x05c4}\xc1\xd8\xec\x00\xbb4\x1b\x02\a\x0e\xf0 \x85\x006\xe0\x1d\x85\x00\x84\xe0\x03\x85\x068v\x9d\x9a(z\xc7@\x85\x06A\xa7|\xc5M|\x8b\xe0\x05\x85\x00\t\xe0\x06\x85\x01\x01\a\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00\xb1\x9e\x0bs\x06)\x0e\xbf\r\n\r\n$1\r\n0\r\n$19\r\n1747300875077943302\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # == WARNING == This replica is rejecting a command from its master: '-LOADING KeyDB is loading the dataset in memory' after processing the command 'KEYDB.MVCCRESTORE'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # Latest backlog is: '"nor#l%\xc7\x05aegean\xe3\bp\x00\xb6 \xb4\x00\xee \xb4\x00\xb4\xe0\x1d\xb4\x00\x82\xe0\x03\xb4\x06:w\xa9\xcf\x92\xd9^`\xb4\x05\xd4\x9d7Hk\xaa@\xb4\n\x1b\x05\x0e\x17\x0e\x18\x0e\"\x02\x01\x01\xe0\x18\x9d\x02sou\xe0\x14\x9d\x00\x86 \x9d\x00\xef \x9d\x00\x84\xe0\x1d\x9d\x00\x83\xe0\x03\x9d\x069t\x7f\xeaH\xfb\x1b`\x9d\x05\x936\x94|#g\xe0\x05\x9d\x00\x05\xe0\t\x9d\x01\x05c4}\xc1\xd8\xec\x00\xbb4\x1b\x02\a\x0e\xf0 \x85\x006\xe0\x1d\x85\x00\x84\xe0\x03\x85\x068v\x9d\x9a(z\xc7@\x85\x06A\xa7|\xc5M|\x8b\xe0\x05\x85\x00\t\xe0\x06\x85\x01\x01\a\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00\xb1\x9e\x0bs\x06)\x0e\xbf\r\n\r\n$1\r\n0\r\n$19\r\n1747300875077943302\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: #024#002#006
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # == CRITICAL == This replica is sending an error to its master: 'command did not execute' after processing the command 'rreplay'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # Latest backlog is: '"nor#l%\xc7\x05aegean\xe3\bp\x00\xb6 \xb4\x00\xee \xb4\x00\xb4\xe0\x1d\xb4\x00\x82\xe0\x03\xb4\x06:w\xa9\xcf\x92\xd9^`\xb4\x05\xd4\x9d7Hk\xaa@\xb4\n\x1b\x05\x0e\x17\x0e\x18\x0e\"\x02\x01\x01\xe0\x18\x9d\x02sou\xe0\x14\x9d\x00\x86 \x9d\x00\xef \x9d\x00\x84\xe0\x1d\x9d\x00\x83\xe0\x03\x9d\x069t\x7f\xeaH\xfb\x1b`\x9d\x05\x936\x94|#g\xe0\x05\x9d\x00\x05\xe0\t\x9d\x01\x05c4}\xc1\xd8\xec\x00\xbb4\x1b\x02\a\x0e\xf0 \x85\x006\xe0\x1d\x85\x00\x84\xe0\x03\x85\x068v\x9d\x9a(z\xc7@\x85\x06A\xa7|\xc5M|\x8b\xe0\x05\x85\x00\t\xe0\x06\x85\x01\x01\a\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00\xb1\x9e\x0bs\x06)\x0e\xbf\r\n\r\n$1\r\n0\r\n$19\r\n1747300875077943302\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # == WARNING == This replica is rejecting a command from its master: '-LOADING KeyDB is loading the dataset in memory' after processing the command 'KEYDB.MVCCRESTORE'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # Latest backlog is: '"nor#l%\xc7\x05aegean\xe3\bp\x00\xb6 \xb4\x00\xee \xb4\x00\xb4\xe0\x1d\xb4\x00\x82\xe0\x03\xb4\x06:w\xa9\xcf\x92\xd9^`\xb4\x05\xd4\x9d7Hk\xaa@\xb4\n\x1b\x05\x0e\x17\x0e\x18\x0e\"\x02\x01\x01\xe0\x18\x9d\x02sou\xe0\x14\x9d\x00\x86 \x9d\x00\xef \x9d\x00\x84\xe0\x1d\x9d\x00\x83\xe0\x03\x9d\x069t\x7f\xeaH\xfb\x1b`\x9d\x05\x936\x94|#g\xe0\x05\x9d\x00\x05\xe0\t\x9d\x01\x05c4}\xc1\xd8\xec\x00\xbb4\x1b\x02\a\x0e\xf0 \x85\x006\xe0\x1d\x85\x00\x84\xe0\x03\x85\x068v\x9d\x9a(z\xc7@\x85\x06A\xa7|\xc5M|\x8b\xe0\x05\x85\x00\t\xe0\x06\x85\x01\x01\a\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00\xb1\x9e\x0bs\x06)\x0e\xbf\r\n\r\n$1\r\n0\r\n$19\r\n1747300875077943302\r\n"'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: #024#002#006
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # == CRITICAL == This replica is sending an error to its master: 'command did not execute' after processing the command 'rreplay'
Oct 21 15:39:27 z27 docker/keydb-de[22018]: 1:22:S 21 Oct 2022 12:39:27.597 # Latest backlog is: '"nor#l%\xc7\x05aegean\xe3\bp\x00\xb6 \xb4\x00\xee \xb4\x00\xb4\xe0\x1d\xb4\x00\x82\xe0\x03\xb4\x06:w\xa9\xcf\x92\xd9^`\xb4\x05\xd4\x9d7Hk\xaa@\xb4\n\x1b\x05\x0e\x17\x0e\x18\x0e\"\x02\x01\x01\xe0\x18\x9d\x02sou\xe0\x14\x9d\x00\x86 \x9d\x00\xef \x9d\x00\x84\xe0\x1d\x9d\x00\x83\xe0\x03\x9d\x069t\x7f\xeaH\xfb\x1b`\x9d\x05\x936\x94|#g\xe0\x05\x9d\x00\x05\xe0\t\x9d\x01\x05c4}\xc1\xd8\xec\x00\xbb4\x1b\x02\a\x0e\xf0 \x85\x006\xe0\x1d\x85\x00\x84\xe0\x03\x85\x068v\x9d\x9a(z\xc7@\x85\x06A\xa7|\xc5M|\x8b\xe0\x05\x85\x00\t\xe0\x06\x85\x01\x01\a\xfa\x0bmvcc-tstamp\x131747300865879834624\t\x00\xb1\x9e\x0bs\x06)\x0e\xbf\r\n\r\n$1\r\n0\r\n$19\r\n1747300875077943302\r\n"'
Aditional information
- OS distribution and version: docker image eqalpha/keydb:x86_64_v6.3.1
- Cluster was running for 5 days with no issues. Then rare errors appeared on one instance. Soon after all instances was affected and rebooting in loop. We tried full wipe of cluster and recreate it from scratch under load. But error appeared immediately. When we recreating cluster without requests then it working for some time.
Our config:
protected-mode no
tcp-backlog 511
timeout 600
tcp-keepalive 300
supervised no
pidfile /var/run/keydb_6382.pid
loglevel notice
databases 16
always-show-logo yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 0 0 0
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
dynamic-hz yes
dbfilename dump.rdb
dir /data
server-threads 8
maxclients 100000
maxmemory 2Gb
maxmemory-policy noeviction
replica-serve-stale-data yes
replica-read-only no
active-replica yes
multi-master yes
#admin
replicaof xxx.xxx.69.1 6389
replicaof xxx.xxx.71.1 6389
#bussiness
replicaof xxx.xxx.70.1 6389
replicaof xxx.xxx.72.1 6389
#cron
replicaof xxx.xxx.39.1 6389
replicaof xxx.xxx.62.1 6389
replicaof xxx.xxx.47.1 6389
#web
replicaof xxx.xxx.26.1 6389
replicaof xxx.xxx.27.1 6389
replicaof xxx.xxx.28.1 6389
replicaof xxx.xxx.53.1 6389
replicaof xxx.xxx.54.1 6389
replicaof xxx.xxx.55.1 6389
replicaof xxx.xxx.73.1 6389
replicaof xxx.xxx.74.1 6389
This can be easily reproduced as follows:
- Configure each instance to set itself as replica
- Start all instances
- Stop one instance
- Every other instance crashes with the same assertion (in particular,
cached_masteris not equal to NULL)
I've been struggling with multimaster configuration for like 2 months, not usable for a production ;)
Hi Snap Team,
Is this going to be attended? We are also experiencing such issues.
@Roycohen as @danog said we were able to fix this issue by excluding self replica ip from config on start like this:
#!/bin/sh
set -ex
if [ "$1" != "base" ]; then
sed "/^replicaof $DOCKER_HOST_IP /d" -i "/usr/local/etc/servers_$1.conf"
fi
keydb-server "/usr/local/etc/$1.conf"
@xtrime-ru can you please explain more about it? I don't see this comment BTW.
I need more wording to understand it please.
@Roycohen
All our instances had one same config. In this config we listed all instances IP addresses. So each instance was connected to other instances and itself. When instance reboots because of some other errors it tries to sync withitself and crashes.
To fix this loop you need to exclude on each instance its ip from its config.
UPDATE: By the way, we had serios issues with trafic. Replicas constantly was syncing with each other and utilizing 300-500 Mbits/s of network. So we decided to leave one server in replica list.
So our config now looks like this:
protected-mode no
tcp-backlog 511
timeout 600
tcp-keepalive 300
supervised no
pidfile /var/run/keydb.pid
loglevel notice
databases 16
always-show-logo yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 0 0 0
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
dynamic-hz yes
dbfilename dump.rdb
dir /data
rdbcompression yes
server-threads 4
maxclients 100000
maxmemory 2Gb
maxmemory-policy noeviction
replica-serve-stale-data yes
replica-read-only no
active-replica yes
multi-master yes
replicaof xxx.xxx.62.1 6389
And on server 62 there is no replicaof entry in config, because of entrypoint script i mentioned in previous message. This way we reduced network utilisation in 100 times!
Yeah, network utilization was obviously reduced because when a replica also has itself as master in the settings, upon startup it redownloads the entire database from itself, then drops the local copy and reinstalls the downloaded copy.
Generally I don't understand why doesn't keydb automatically exclude itself from the replica list, as this leads to a lot of unwanted behavior.
TL;DR @roycohen: make sure your replica IP address list doesn't list the IP address of the the current keydb instance.
Yeah, network utilization was obviously reduced because when a replica also has itself as master in the settings, upon startup it redownloads the entire database from itself, then drops the local copy and reinstalls the downloaded copy.
Well, but we had network issues even after we excluded replicas ip from itself. So its not the main/only cause.
Closing as it seems the crash issue is resolved(exclude own ip from replicaof).
