[CRASH] Race condition between prefetchKeysAsync, freeMemoryGetNotCountedMemory and beforeSleep

[msotheeswaran-sc]

Crash report

Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.

------ STACK TRACE ------
Backtrace:
../../src//keydb-server *:30005 [cluster](+0x358f79) [0x560a46b61f79]
../../src//keydb-server *:30005 [cluster](freeMemoryGetNotCountedMemory()+0xf4) [0x560a46b20b54]
../../src//keydb-server *:30005 [cluster](getMaxmemoryState(unsigned long*, unsigned long*, unsigned long*, float*, bool, bool)+0xea) [0x560a46b20d5a]
../../src//keydb-server *:30005 [cluster](performEvictions(bool)+0x180) [0x560a46b21640]
../../src//keydb-server *:30005 [cluster](serverCronLite(aeEventLoop*, long long, void*)+0x31) [0x560a46a2cb41]
../../src//keydb-server *:30005 [cluster](aeProcessEvents+0x22c) [0x560a46a234fc]
../../src//keydb-server *:30005 [cluster](aeMain+0x3e) [0x560a46a23c5e]
../../src//keydb-server *:30005 [cluster](workerThreadMain(void*)+0x1b0) [0x560a46a3b4d0]
/lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f816b9d6b43]
/lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f816ba68a00]
------ STACK TRACE ------
Backtrace:
../../src//keydb-server *:30005 [cluster](+0x358f79) [0x560a46b61f79]
../../src//keydb-server *:30005 [cluster](aeAcquireLock+0x61) [0x560a46a1f9c1]
../../src//keydb-server *:30005 [cluster](beforeSleep(aeEventLoop*)+0x412) [0x560a46a2d272]
../../src//keydb-server *:30005 [cluster](aeProcessEvents+0x400) [0x560a46a236d0]
../../src//keydb-server *:30005 [cluster](aeMain+0x3e) [0x560a46a23c5e]
../../src//keydb-server *:30005 [cluster](workerThreadMain(void*)+0x1b0) [0x560a46a3b4d0]
/lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f816b9d6b43]
/lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f816ba68a00]
------ STACK TRACE ------
Backtrace:
../../src//keydb-server *:30005 [cluster](+0x358f79) [0x560a46b61f79]
../../src//keydb-server *:30005 [cluster](aeAcquireLock+0x61) [0x560a46a1f9c1]
../../src//keydb-server *:30005 [cluster](redisDbPersistentData::prefetchKeysAsync(client*, parsed_command&)+0x77f) [0x560a46a72b2f]
../../src//keydb-server *:30005 [cluster](+0x2535aa) [0x560a46a5c5aa]
../../src//keydb-server *:30005 [cluster](readQueryFromClient(connection*)+0x397) [0x560a46a52b97]
../../src//keydb-server *:30005 [cluster](connSocketEventHandler(aeEventLoop*, int, void*, int)+0x1e0) [0x560a46b4a4c0]
../../src//keydb-server *:30005 [cluster](ProcessEventCore+0x104) [0x560a46a1f484]
../../src//keydb-server *:30005 [cluster](aeProcessEvents+0x175) [0x560a46a23445]
../../src//keydb-server *:30005 [cluster](aeMain+0x3e) [0x560a46a23c5e]
../../src//keydb-server *:30005 [cluster](workerThreadMain(void*)+0x1b0) [0x560a46a3b4d0]
/lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f816b9d6b43]
/lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f816ba68a00]
------ STACK TRACE ------
Backtrace:
../../src//keydb-server *:30005 [cluster](+0x358f79) [0x560a46b61f79]
../../src//keydb-server *:30005 [cluster](aeAcquireLock+0x61) [0x560a46a1f9c1]
../../src//keydb-server *:30005 [cluster](redisDbPersistentData::prefetchKeysAsync(client*, parsed_command&)+0xcf) [0x560a46a7247f]
../../src//keydb-server *:30005 [cluster](+0x2535aa) [0x560a46a5c5aa]
../../src//keydb-server *:30005 [cluster](readQueryFromClient(connection*)+0x397) [0x560a46a52b97]
../../src//keydb-server *:30005 [cluster](connSocketEventHandler(aeEventLoop*, int, void*, int)+0x1e0) [0x560a46b4a4c0]
../../src//keydb-server *:30005 [cluster](ProcessEventCore+0x104) [0x560a46a1f484]
../../src//keydb-server *:30005 [cluster](aeProcessEvents+0x175) [0x560a46a23445]
../../src//keydb-server *:30005 [cluster](aeMain+0x3e) [0x560a46a23c5e]
../../src//keydb-server *:30005 [cluster](workerThreadMain(void*)+0x1b0) [0x560a46a3b4d0]
/lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f816b9d6b43]
/lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f816ba68a00]

Aditional information Race condition, those are the relevant stack traces

1. OS distribution and version
2. Steps to reproduce (if any) happened when running memtier_benchmark -s 127.0.0.1 -p 30001 -n 2000000 --cluster --key-minimum=100000000 --key-maximum=999000000 --expiry-range 0-66000 on cluster created with utils/create-cluster script