speedb
speedb copied to clipboard
speedb-io
Add a backdoor for running compaction method on demand
Provide a backdoor for running the compaction method. This can be used to schedule the compaction method periodically by external application.
@bosmatt - Could you elaborate on the motivation for this feature? There is manual compaction (now both blocking and non-blocking). Is it not sufficient? Why?
One of our Kafka Streams users would like to schedule periodic compaction based on calendar dates and times. For example, they want to schedule compactions on the weekends and/or in the night hours when their system has less load. With manual compaction, we could theoretically also implement this calendar based compaction triggers in Kafka Streams, I guess. However, as @Guyme pointed out it might also be interesting for other users outside of Kafka Streams.
Ah, wait. Reading again the title of the issue, I am not sure if we are on the same page. I assumed this feature is about RocksDB/Speedb triggering the compaction based on a calendar. However, the title suggest adding a method for runnig compactions. If it is about the latter than -- as @udi-speedb -- I am also wondering whether manual compaction might be sufficient.
Once there is a backdoor for running compaction you can run it periodically using external tools/scripts/code. @cadonna This indeed should give an answer to your request, and help others as well. Please share your thoughts on this solution.
@bosmatt Do I understand you correctly that this backdoor would allow -- for example -- a operation system cron job to trigger the compaction?
That would be different from running manual compaction by calling compactRange() from within Kafka Streams.
Originally, I envisioned this feature as a config in Speedb/RocksDB that allows to pass in a schedule (for example in cron format) that is used to trigger compactions at the specified times. WDYT?