apache
Memory issue of kvrocks2redis detected by ASan/TSan
Search before asking
- [X] I had searched in the issues and found no similar issues.
Motivation
Currently, the Kvrocks2redis CI fails with TSAN/ASAN or in macos env. An example for this: https://github.com/Zakelly/kvrocks/actions/runs/8344973371/job/22838786340 The value mismatch in destination redis server. More investigation is needed.
Solution
No response
Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Thank you @Zakelly ! The action log will be deleted by GitHub after some days. Can you download the log file and upload?
IMHO we should spend some time on investigating this issue and find a solution before next release. cc @git-hulk
@PragmaTwice I may take some time to investigate this after resolving #2253, and it'd be great if other guys would like to dive deep into this issue.
@PragmaTwice I may take some time to investigate this after resolving #2253, and it'd be great if other guys would like to dive deep into this issue.
I think this issue is related to https://github.com/apache/kvrocks/issues/1988 which makes kvrocks2redis crash. It is important to keep kvrocks2redis usable, without crashing.
@PragmaTwice I may take some time to investigate this after resolving #2253, and it'd be great if other guys would like to dive deep into this issue.
@git-hulk Hi, have you started studying this core? If not I kind of want to try it.
@Zakelly Hi, I've investigated this issue and successfully reproduced it on my computer.
I suspect it's not a memory issue, but rather that compiling with ASAN and TSAN significantly impacts the performance of kvrocks2redis. This leads to slower data synchronization in kvrocks2redis.
And a simple sleep(0.02)(url) in the check_consistency.py test script might not give Redis enough time to sync the corresponding results, resulting in execution errors.
As shown in the diagram below, although the test failed, the corresponding keys and values can still be found in Redis afterward.
Or we can try to make the test script sleep longer?
@Zakelly Hi, I've investigated this issue and successfully reproduced it on my computer.
I suspect it's not a memory issue, but rather that compiling with ASAN and TSAN significantly impacts the performance of
kvrocks2redis. This leads to slower data synchronization inkvrocks2redis.And a simple
sleep(0.02)(url) in thecheck_consistency.pytest script might not give Redis enough time to sync the corresponding results, resulting in execution errors.As shown in the diagram below, although the test failed, the corresponding keys and values can still be found in Redis afterward.
Or we can try to make the test script sleep longer?
cc @git-hulk
@AntiTopQuark Thanks for your great investigation. Can we relax the check condition? like check it N times with a fixed interval and return once it's passed. cc @PragmaTwice What do you think?