[BUG] Master - Master replication Command didn't execute

[Antiarchitect]

trafficstars

Describe the bug

keydb 1:14:S 06 Mar 2022 01:40:35.374 # == WARNING == This replica is rejecting a command from its master: '-LOADING KeyDB is loading the dataset in memory' after processing the command 'KEYDB.MVCCRESTORE'
keydb 1:14:S 06 Mar 2022 01:40:35.374 # Latest backlog is: '"ORE\r\n$39\r\nS:01FTZ2WW9E1ARYPGTZA3NAGH24:6759270384\r\n$19\r\n1726507402026549351\r\n$13\r\n1646612165231\r\n$116\r\n\x00@g\b\xb2\x95'\xe0\xbc4\xde\xc5*\x9a\xce\x1a\xb1\xf90\x8d\x8b4\xb2\xf90v\xa1\xc91\xa2\xc31\xe4\xc91\xe5\xbb4\xd6\xba4\x8c\x944\xdd\xba4v\a\xee\xb9\x82\xb7\xd2_\xa6\xb3#\xc0\xb0\xac\xc4\x80\x14\x82'\xa0\xb3#\x8a\x04\x88'\xa0\xb3#\xf8\x03\x88'\xa0\xb3#\xe2\x03\x88'\xa8\xe5$\xd4\x04\x88'\xe0\xe0$\x90\x04\x88'\xc8\xdf\x02\xfa\x03\t\x00\x0b\x9d\x94\x87\x04i\xf9\xa5\r\n\r\n$1\r\n0\r\n$19\r\n1726512717218447370\r\n"'
keydb 1:14:S 06 Mar 2022 01:40:35.374 # Command didn't execute: ����ۯ��֜��_��$������RН%��'
keydb 1:14:S 06 Mar 2022 01:40:35.374 # == CRITICAL == This replica is sending an error to its master: 'command did not execute' after processing the command 'rreplay'
keydb 1:14:S 06 Mar 2022 01:40:35.374 # Latest backlog is: '"ORE\r\n$39\r\nS:01FTZ2WW9E1ARYPGTZA3NAGH24:6759270384\r\n$19\r\n1726507402026549351\r\n$13\r\n1646612165231\r\n$116\r\n\x00@g\b\xb2\x95'\xe0\xbc4\xde\xc5*\x9a\xce\x1a\xb1\xf90\x8d\x8b4\xb2\xf90v\xa1\xc91\xa2\xc31\xe4\xc91\xe5\xbb4\xd6\xba4\x8c\x944\xdd\xba4v\a\xee\xb9\x82\xb7\xd2_\xa6\xb3#\xc0\xb0\xac\xc4\x80\x14\x82'\xa0\xb3#\x8a\x04\x88'\xa0\xb3#\xf8\x03\x88'\xa0\xb3#\xe2\x03\x88'\xa8\xe5$\xd4\x04\x88'\xe0\xe0$\x90\x04\x88'\xc8\xdf\x02\xfa\x03\t\x00\x0b\x9d\x94\x87\x04i\xf9\xa5\r\n\r\n$1\r\n0\r\n$19\r\n1726512717218447370\r\n"'

To reproduce

Tried to use 3 KeyDB nodes with master - master diskless replication as Thanos Cache (intense read-writes, Gbit speeds)

Extra args:

  configExtraArgs:
    - client-output-buffer-limit: ["normal", "0", "0", "0"]
    - client-output-buffer-limit: ["replica", "0", "0", "0"]
    - client-output-buffer-limit: ["pubsub", "0", "0", "0"]
    - repl-diskless-load: "on-empty-db"
    - repl-diskless-sync: "yes"
    - save: ""

Expected behavior

Replication goes normal

Additional information

eqalpha/keydb:x86_64_v6.2.2

[MalavanEQAlpha]

Hi @Antiarchitect, I am unable to reproduce this bug. Can you provide more details about your setup(i.e number of threads, are all the nodes on the same machine, what kind of load was each node under when this occured), as well as logs from all 3 nodes during the bug?

[Antiarchitect]

@MalavanEQAlpha So sorry, but we rejected the idea of Thanos cache with KeyDB for now. I cannot test now under the production load :(

[msotheeswaran-sc]

Closing as we were unable to reproduce and no other reports.