[BUG] Issue with consistency of KeyDB Stream data on active-active setup

[krunkosaurus]

In my testing of active-active KeyDB reading and writing to Streams there are data sync, data loss, and consistency issues.

The Setup:

• I have 2 KeyDB instances, one in Tokyo AWS and one in London AWS.
• I have the two instances in active-active sync with each other and confirmed if I set a simple key/value on one, the other will have it as well.
• I have also confirm broadcasting via pub/sub on one and the other will get the sub just fine.

The test:

• have a node.js script write to each's local KeyDB Stream called "action" once every 10-seconds with the following data:
  1. Tokyo or London
  2. Count (how many times this script has ran on this server)
  3. Current epoc time in microseconds (to compare with Stream key timestamp)
• I have a node.js process listening to events on each KeyDB "action" stream on each server to see if there are any differences.
• Lastly I have queried the last 20 items from each "action" stream on each KeyDB intances to check for differences.

The data:

Stream listener from Tokyo:

Id: 1631532820007-0. Data: [ 'action', 'tokyo1 1 1631532820008' ]
Id: 1631532820009-0. Data: [ 'action', 'london1 1 1631532820009' ]
Id: 1631532830003-0. Data: [ 'action', 'tokyo1 2 1631532830003' ]
Id: 1631532830011-0. Data: [ 'action', 'london1 2 1631532830011' ]
Id: 1631532840002-0. Data: [ 'action', 'tokyo1 3 1631532840003' ]
Id: 1631532840005-0. Data: [ 'action', 'london1 3 1631532840005' ]
Id: 1631532850005-0. Data: [ 'action', 'tokyo1 4 1631532850005' ]
Id: 1631532850008-0. Data: [ 'action', 'london1 4 1631532850008' ]
Id: 1631532860012-0. Data: [ 'action', 'tokyo1 5 1631532860011' ]
Id: 1631532870015-0. Data: [ 'action', 'tokyo1 6 1631532870003' ]
Id: 1631532880011-0. Data: [ 'action', 'tokyo1 7 1631532880011' ]
Id: 1631532890007-0. Data: [ 'action', 'tokyo1 8 1631532890007' ]
Id: 1631532890010-0. Data: [ 'action', 'london1 8 1631532890010' ]
Id: 1631532900002-0. Data: [ 'action', 'tokyo1 9 1631532900003' ]
Id: 1631532900010-0. Data: [ 'action', 'london1 9 1631532900010' ]
Id: 1631532910007-0. Data: [ 'action', 'tokyo1 10 1631532910007' ]
Id: 1631532910010-0. Data: [ 'action', 'london1 10 1631532910010' ]
Id: 1631532920010-0. Data: [ 'action', 'tokyo1 11 1631532920011' ]
Id: 1631532930005-0. Data: [ 'action', 'tokyo1 12 1631532930006' ]
Id: 1631532930009-0. Data: [ 'action', 'london1 12 1631532930008' ]
Id: 1631532940008-0. Data: [ 'action', 'tokyo1 13 1631532940008' ]
Id: 1631532950006-0. Data: [ 'action', 'tokyo1 14 1631532950007' ]
Id: 1631532950011-0. Data: [ 'action', 'london1 14 1631532950011' ]
Id: 1631532960004-0. Data: [ 'action', 'tokyo1 15 1631532960004' ]
Id: 1631532960012-0. Data: [ 'action', 'london1 15 1631532960011' ]
Id: 1631532970001-0. Data: [ 'action', 'tokyo1 16 1631532970001' ]
Id: 1631532970003-0. Data: [ 'action', 'london1 16 1631532970003' ]
Id: 1631532980003-0. Data: [ 'action', 'tokyo1 17 1631532980003' ]
Id: 1631532980012-0. Data: [ 'action', 'london1 17 1631532980011' ]
Id: 1631532990006-0. Data: [ 'action', 'tokyo1 18 1631532990007' ]
Id: 1631533000011-0. Data: [ 'action', 'tokyo1 19 1631533000011' ]

Stream listener in London:

Id: 1631532830011-0. Data: [ 'action', 'london1 2 1631532830011' ]
Id: 1631532840005-0. Data: [ 'action', 'london1 3 1631532840005' ]
Id: 1631532850008-0. Data: [ 'action', 'london1 4 1631532850008' ]
Id: 1631532860008-0. Data: [ 'action', 'london1 5 1631532860007' ]
Id: 1631532860012-0. Data: [ 'action', 'tokyo1 5 1631532860011' ]
Id: 1631532870006-0. Data: [ 'action', 'london1 6 1631532870006' ]
Id: 1631532870015-0. Data: [ 'action', 'tokyo1 6 1631532870003' ]
Id: 1631532880011-0. Data: [ 'action', 'london1 7 1631532880011' ]
Id: 1631532890010-0. Data: [ 'action', 'london1 8 1631532890010' ]
Id: 1631532900010-0. Data: [ 'action', 'london1 9 1631532900010' ]
Id: 1631532910010-0. Data: [ 'action', 'london1 10 1631532910010' ]
Id: 1631532920005-0. Data: [ 'action', 'london1 11 1631532920005' ]
Id: 1631532920010-0. Data: [ 'action', 'tokyo1 11 1631532920011' ]
Id: 1631532930009-0. Data: [ 'action', 'london1 12 1631532930008' ]
Id: 1631532940003-0. Data: [ 'action', 'london1 13 1631532940003' ]
Id: 1631532940008-0. Data: [ 'action', 'tokyo1 13 1631532940008' ]
Id: 1631532950011-0. Data: [ 'action', 'london1 14 1631532950011' ]
Id: 1631532960012-0. Data: [ 'action', 'london1 15 1631532960011' ]
Id: 1631532970003-0. Data: [ 'action', 'london1 16 1631532970003' ]
Id: 1631532980012-0. Data: [ 'action', 'london1 17 1631532980011' ]
Id: 1631532990005-0. Data: [ 'action', 'london1 18 1631532990004' ]
Id: 1631532990006-0. Data: [ 'action', 'tokyo1 18 1631532990007' ]
Id: 1631533000010-0. Data: [ 'action', 'london1 19 1631533000009' ]
Id: 1631533000011-0. Data: [ 'action', 'tokyo1 19 1631533000011' ]

Querying the last 20 items in "action" stream from Tokyo:

[
  [ '1631533000011-0', [ 'action', 'tokyo1 19 1631533000011' ] ],
  [ '1631532990006-0', [ 'action', 'tokyo1 18 1631532990007' ] ],
  [ '1631532980012-0', [ 'action', 'london1 17 1631532980011' ] ],
  [ '1631532980003-0', [ 'action', 'tokyo1 17 1631532980003' ] ],
  [ '1631532970003-0', [ 'action', 'london1 16 1631532970003' ] ],
  [ '1631532970001-0', [ 'action', 'tokyo1 16 1631532970001' ] ],
  [ '1631532960012-0', [ 'action', 'london1 15 1631532960011' ] ],
  [ '1631532960004-0', [ 'action', 'tokyo1 15 1631532960004' ] ],
  [ '1631532950011-0', [ 'action', 'london1 14 1631532950011' ] ],
  [ '1631532950006-0', [ 'action', 'tokyo1 14 1631532950007' ] ],
  [ '1631532940008-0', [ 'action', 'tokyo1 13 1631532940008' ] ],
  [ '1631532930009-0', [ 'action', 'london1 12 1631532930008' ] ],
  [ '1631532930005-0', [ 'action', 'tokyo1 12 1631532930006' ] ],
  [ '1631532920010-0', [ 'action', 'tokyo1 11 1631532920011' ] ],
  [ '1631532910010-0', [ 'action', 'london1 10 1631532910010' ] ],
  [ '1631532910007-0', [ 'action', 'tokyo1 10 1631532910007' ] ],
  [ '1631532900010-0', [ 'action', 'london1 9 1631532900010' ] ],
  [ '1631532900002-0', [ 'action', 'tokyo1 9 1631532900003' ] ],
  [ '1631532890010-0', [ 'action', 'london1 8 1631532890010' ] ],
  [ '1631532890007-0', [ 'action', 'tokyo1 8 1631532890007' ] ]
]

Querying the last 20 items from "actions" stream in London

[
  [ '1631533000011-0', [ 'action', 'tokyo1 19 1631533000011' ] ],
  [ '1631533000010-0', [ 'action', 'london1 19 1631533000009' ] ],
  [ '1631532990006-0', [ 'action', 'tokyo1 18 1631532990007' ] ],
  [ '1631532990005-0', [ 'action', 'london1 18 1631532990004' ] ],
  [ '1631532980012-0', [ 'action', 'london1 17 1631532980011' ] ],
  [ '1631532970003-0', [ 'action', 'london1 16 1631532970003' ] ],
  [ '1631532960012-0', [ 'action', 'london1 15 1631532960011' ] ],
  [ '1631532950011-0', [ 'action', 'london1 14 1631532950011' ] ],
  [ '1631532940008-0', [ 'action', 'tokyo1 13 1631532940008' ] ],
  [ '1631532940003-0', [ 'action', 'london1 13 1631532940003' ] ],
  [ '1631532930009-0', [ 'action', 'london1 12 1631532930008' ] ],
  [ '1631532920010-0', [ 'action', 'tokyo1 11 1631532920011' ] ],
  [ '1631532920005-0', [ 'action', 'london1 11 1631532920005' ] ],
  [ '1631532910010-0', [ 'action', 'london1 10 1631532910010' ] ],
  [ '1631532900010-0', [ 'action', 'london1 9 1631532900010' ] ],
  [ '1631532890010-0', [ 'action', 'london1 8 1631532890010' ] ],
  [ '1631532880011-0', [ 'action', 'london1 7 1631532880011' ] ],
  [ '1631532870015-0', [ 'action', 'tokyo1 6 1631532870003' ] ],
  [ '1631532870006-0', [ 'action', 'london1 6 1631532870006' ] ],
  [ '1631532860012-0', [ 'action', 'tokyo1 5 1631532860011' ] ]
]

Let me know if you would like my actual node.js test scripts. The conclusions so far (unless I'm doing something wrong) is that active-active KeyDB steams are unstable and not useable.