okaywal
okaywal copied to clipboard
Add a synchronous "before checkpoint" hook?
For some use cases of a write ahead log, I'd like to keep an in-memory representation (which gets recovered) of the contents of the log. When a checkpoint is made, I can drop the check-pointed information from memory since it is now part of the checkpoint.
One difficulty, as far as I can tell, is that the checkpoint method is called in a separate thread -- this means the following could happen:
- Commit some data
Ato the log - Initiate a checkpoint
- Commit some data
Bto the log - Actually perform the checkpoint
The problem I have is that at that point, the contents of A and B are in the in-memory representaiton, while I only want to checkpoint (and potentially drop) A. For my use case, I'm fine keeping the in-memory representation behind a Mutex (there should be only one writer to the log), which also simplifies a lot of the logic.
There are a few things that could help me:
- If the result of committing data indicated that a checkpoint would be taken. I think this would only help if there was a lock, but then I could detect that a checkpoint would be taken after
A, and arrange to have a separate in-memory representation started for data added after that. - If there was a "pre-checkpoint" hook that was called before other commits were allowed. Again, this would allow me to split the in-memory representation for B.
Thank you for the idea for the additional use case for this crate. I think the easiest thing to offer would be your second suggestion.
If you need this quickly, I would love a pull request. Otherwise I will eventually be getting back onto this project, but right now it's not my primary focus.