valkey icon indicating copy to clipboard operation
valkey copied to clipboard

[NEW] Offload corruption check to a separate thread during external data load

Open ashtul opened this issue 5 months ago • 0 comments

Following the approval of #399, which eliminates redundant corruption checks for listpacks on every read, verification will now be mandatory when the listpack is loaded into the Valkey server. This change optimizes read operations by removing the check from the critical read path, allowing for better performance during runtime.

To further enhance server startup speed, an option should be introduced to offload the corruption verification process to a separate thread. By moving verification to another thread, Valkey can proceed with the startup sequence while simultaneously ensuring data integrity. This parallelization would reduce startup time, particularly when loading large datasets or handling multiple keys.

The dedicated thread would be responsible for processing multiple keys at once. Once a batch of keys is verified, the thread would acquire the global lock, insert the sanitized keys into the main hash table, and then release the lock. This approach minimizes contention for the global lock, ensuring that the main thread remains unaffected by verification delays, ultimately improving overall efficiency during data loading.

ashtul avatar Sep 19 '24 07:09 ashtul