Multi mastering + Active replica for 3 nodes

[YuriyHab]

I Using config example from https://docs.keydb.dev/docs/multi-master#config-file I have Celery + HA-RabbitMQ and KeyDB (docker) as tool for sharing data between workers. In my project also uses RLock. My keydb entities uses persistence AOF (in always mode) + RDB

1. Up 3 keydb nodes. All node absolutely clear (no data)
2. I start my Celery workers, that lock and unlock some keys in keydb1 node, read & write new data for locked keys (simple set / get seems to work fine)
3. After first writing in keydb1, keydb2 and keydb3 start getting errors about lost connection bethween others replicas (including keydb1), start resync, and I get data loss with higly probability in this moments. Also sometimes i get some keydb entity freeze, that work, but on INFO on another entities it has "down" status.
4. After sometime all keydb entities come to a consensus for data

In step 2 keydb2, keydb3 have next logs output:

1:14:S 14 May 2021 15:09:05.200 # === ASSERTION FAILED === 1:14:S 14 May 2021 15:09:05.200 # ==> networking.cpp:268 'c->conn == nullptr || c->lock.fOwnLock()' is not true ... 1:14:S 14 May 2021 15:09:07.750 # Connection with replica client id #4 lost. .... restart keydb ... 1:1:S 14 May 2021 15:11:29.227 * Reading the remaining AOF tail... 1:1:S 14 May 2021 15:11:29.230 # == CRITICAL == This server is sending an error to its AOF-loading-client: 'Command must be sent from a master' after processing the command 'rreplay' 1:1:S 14 May 2021 15:11:29.232 # == CRITICAL == This server is sending an error to its AOF-loading-client: 'Command must be sent from a master' after processing the command 'rreplay' 1:1:S 14 May 2021 15:11:29.232 # == CRITICAL == This server is sending an error to its AOF-loading-client: 'Command must be sent from a master' after processing the command 'rreplay' .... here replication with keydb1 and keydb2/3 .... 1:16:S 14 May 2021 15:12:46.551 * Background saving started by pid 24 1:16:S 14 May 2021 15:12:47.416 # I/O error reading bulk count from MASTER: Resource temporarily unavailable ... 1:16:S 14 May 2021 15:12:47.964 * Non blocking connect for SYNC fired the event. 1:16:S 14 May 2021 15:12:47.965 * Master replied to PING, replication can continue... 1:16:S 14 May 2021 15:12:47.966 * Partial resynchronization not possible (no cached master)

After all i have some questions:

1. MultiMaster + active replica work fine for 2 nodes. Can i scaling it for N nodes in future?
2. If all of the above is a bug, then in what time frame can it be fixed?

[VivekSainiEQ]

Hi @YuriyHab,

This looks like bug and we're looking into it at the moment. When you say you're locking keys with RLock, are you referring to the in built Python multiprocessing.RLock, and if so how?

[YuriyHab]

Hi, @VivekSainiEQ !

I'm sorry if I misled you! By Rlock, I meant a redis distributed locks. I worked with keydb using redis

[VivekSainiEQ]

No worries!

I have a few more probing questions:

1. I'm assuming you're using RabbitMQ as the message broker and KeyDB as the backend, yes?
2. What library specifically are you using for your distributed locks? Or have you implemented your own distributed locks from scratch?

[YuriyHab]

1. Yes, I use RabbitMQ as message broker and KeyDB as backend.
2. What about distributed locks... I use python lib redis-py Perhaps I mistakenly believed that redis-py locks were implemented using the RedLock algorithm. In any case, there is no mention of this python library here The description on the github says that redis-py uses LUA-locks. However, the documentation for the Lock Class says the following: "A shared, distributed Lock. Using Redis for locking allows the Lock to be shared across processes and / or machines. "

[VivekSainiEQ]

I've setup a test environment with what you've described (3 nodes active replica KeyDB backend with persistence, RabbitMQ broker, Celery workers using Redis-py to lock keys before setting and getting values) and unfortunately can't seem to replicate the issue. Do you have a full crash log by any chance? I suspect the issue is related to Lua scripting (given the nature of the redis locks), but I can't be sure since I can't reproduce the issue locally.

[YuriyHab]

Unfortunately, there are no saved logs... In the near future I will try to roll back my project to using keyb and give you an answer

[VivekSainiEQ]

Sounds good, do let us know if you do manage to roll back your project and are able to replicate the issue.

[FallingSnow]

I'm running into a similar issue. Used to run redis with the time series module. Switched to keydb with the time series module and I see this spammed to the logs on startup.

1:1:M 14 Sep 2021 06:57:48.935 # == CRITICAL == This server is sending an error to its AOF-loading-client: '-ERR TSDB: Error at upsert, update is not supported in BLOCK mode' after processing the command 'ts.madd'
1:1:M 14 Sep 2021 06:57:48.935 # == CRITICAL == This server is sending an error to its AOF-loading-client: '-ERR TSDB: Error at upsert, update is not supported in BLOCK mode' after processing the command 'ts.madd'
1:1:M 14 Sep 2021 06:57:48.935 # == CRITICAL == This server is sending an error to its AOF-loading-client: '-ERR TSDB: Error at upsert, update is not supported in BLOCK mode' after processing the command 'ts.madd'
... More than 1000+ of the same line above
1:1:M 14 Sep 2021 06:57:48.935 * DB loaded from append only file: 4.150 seconds
1:26:M 14 Sep 2021 06:57:48.935   Thread 0 alive.
1:27:M 14 Sep 2021 06:57:48.935   Thread 1 alive.
1:28:M 14 Sep 2021 06:57:48.935   Thread 2 alive.
1:26:M 14 Sep 2021 06:58:45.011 * 10000 changes in 60 seconds. Saving...
1:26:M 14 Sep 2021 06:58:45.016 * Background saving started by pid 50
50:26:C 14 Sep 2021 06:58:45.376 * DB saved on disk
50:26:C 14 Sep 2021 06:58:45.378 * RDB: 0 MB of memory used by copy-on-write
1:26:M 14 Sep 2021 06:58:45.417 * Background saving terminated with success

[FallingSnow]

I solved the issue by adding DUPLICATE_POLICY LAST to the end of the module load in the config file.

loadmodule /usr/lib/redis/modules/redistimeseries.so DUPLICATE_POLICY LAST

Don't really want to do this since now any write to the same key will override rather than error. Seems to be an issue with keydb loading the AOF multiple times? I don't see this issue in redis.

[jgerry2002]

I have the same issue with a master / master replica pair (only two nodes) setup. I started using streams, not sure what set it off but the logs are filled with:

14794:14793:S 04 Nov 2021 02:57:56.363 # == CRITICAL == This server is sending an error to its AOF-loading-client: 'Command must be sent from a master' after processing the command 'rreplay' 14794:14793:S 04 Nov 2021 02:57:56.363 # == CRITICAL == This server is sending an error to its AOF-loading-client: 'Command must be sent from a master' after processing the command 'rreplay' 14794:14793:S 04 Nov 2021 02:57:56.363 # == CRITICAL == This server is sending an error to its AOF-loading-client: 'Command must be sent from a master' after processing the command 'rreplay' 14794:14793:S 04 Nov 2021 02:57:56.363 # == CRITICAL == This server is sending an error to its AOF-loading-client: 'Command must be sent from a master' after processing the command 'rreplay' 14794:14793:S 04 Nov 2021 02:57:56.363 # == CRITICAL == This server is sending an error to its AOF-loading-client: 'Command must be sent from a master' after processing the command 'rreplay' 14794:14793:S 04 Nov 2021 02:57:56.363 # == CRITICAL == This server is sending an error to its AOF-loading-client: 'Command must be sent from a master' after processing the command 'rreplay' 14794:14793:S 04 Nov 2021 02:57:56.363 # == CRITICAL == This server is sending an error to its AOF-loading-client: 'Command must be sent from a master' after processing the command 'rreplay'

Streams had no data. So something clearly broke. Had to restart keydb several times in order for data to be inserted again.

[msotheeswaran-sc]

Is there anyone still experiencing this issue, will be helpful to understand how to prioritize this.

[msotheeswaran-sc]

Closing as there has been no response in 30 days

[dbotwinick]

I came across the same problem, but unfortunately, I don't have a good handle on what caused it and I can't easily reproduce it--so I am going to ramble a bit and see if it gives anyone who is more knowledgeable about keydb internals any ideas. I am testing active replication between 2 nodes--and 1 node is frequently accessed (A) while the other was essentially not accessed (B). B is acting as a backup to A that could be promoted at any time. I was observing this issue on B after problems with the AOF growing out of control despite AOF-rewriting enabled (which I suspect was related to the lack of client traffic on node B?).

Anyway, after trying to reset B due to running out of disk space from a 30G AOF file for a 250MB database, I ended up in a "replication loop on startup" that kept driving up the AOF file size (by 100% each time) but this time, the AOF rewriting on growth was working, but that meant the server was rewriting the AOF files after each replication (so continuously in a loop...). I shut down the keydb-server process, increased the size of the replication backlog (which I suspected was related to the replication loop problem), and then (finally) I encountered the error as described in this issue--which I assume is tied to some sort of truncation of the AOF file.

For me, the problem was solved by removing the AOF+RDB files from the server and starting it fresh from its replication partner. I am not entirely sure if the replication loop was caused solely by the replication backlog size, but upon a fresh start, I no longer had the problem with the replication AOF writing loop on startup--so I think the replication backlog size was too small given that I was trying to force a full database sync of a 250MB database. I don't think that should cause such an issue, but I haven't finished digging into it. Perhaps while increasing the size of the backlog, it timed out the replication process and then that kept happening each time it tried again? (pure speculation)

So, with all that said, I think the problem above comes with some sort of specific truncation of the AOF file content? Sorry for just rambling... I'm reasonably new to keydb/redis (and have limited time to play with it), so I just figured I'd throw this out there hoping that it provides someone with a hint/lead.

[msotheeswaran-sc]

Hi @dbotwinick was this on version 6.3.1 or 6.3.2, what you described sounds like a combination of bugs that should be fixed with 6.3.3, if the problem persists with 6.3.3 please re-open this issue.

[dbotwinick]

Hi @dbotwinick was this on version 6.3.1 or 6.3.2, what you described sounds like a combination of bugs that should be fixed with 6.3.3, if the problem persists with 6.3.3 please re-open this issue.

6.3.2. I'll keep you posted if any of these things pop back. It's the 2nd time I've encountered uninhibited AOF file growth with v6.3.2. For now, I'm planning to have a cron job to run BGREWRITEAOF weekly to be safe.