Snapchat
[BUG] BGSAVE | Assertion Failed: rdb.cpp:1372 'ckeysExpired == db->expireSize()' is not true
Because of this bug, BGSAVE failed (stuck) at 99.91% (current_fork_perc: 99.91). As a result, I can't take a backup and move to another server.
What should I do? What is the reason? keydb-server.log
Hi @waska14 have a read on this thread, hopefully you have a backup from an older version:
https://github.com/Snapchat/KeyDB/issues/743#issuecomment-1888538934
Hi @keithchew .
Returned to Redis :) We don’t have time to deal with these kinds of issues in production, where there are over 30k requests per second.
Anyway, thanks. I saw that issue, but I was hoping it had been fixed by now.
I just found out we're experiencing this in prod after 6.3.4 was pushed out with Rocky Linux patches. I'm probably going to revert everything to 6.2.2 in containers and then work out what to do with Redis instead. We have a lot of very important environments running as active-active across 2 datacenters.
This bug is so painful in production. I can't restart server because I will lost data. I can't migrate to another version KeyDB (or Redis) because you cant add slave if BGSAVE doesn't work. The only solution a could do is to dump and restore all keys to another server.
So if you on a same boat and you need to migrate your keys from server with broken BGSAVE you can use my python script. It transfers keys with TTL (It uses commands SCAN, TTL, DUMP, RESTORE) https://github.com/anadion/redis-keys-migrate-tool/
