microsoft
Mark auto-expired key-value pairs in the mutable log as tombstoned
Feature request type
enhancement
Is your feature request related to a problem? Please describe
When an expiration time is attached to entries in the store, after expiration, they are not removed from the log until (and if) they tier down to storage and get compacted.
We would like to add some way to tombstone these records so that (1) the heap objects are freed in case of the object store; and (2) the log space can be reused for new objects using revivification, in case of the main store.
Describe the solution you'd like
One way to accomplish this is to have a periodic background scanner thread that scans (only the mutable region of) the log, and tombstones any expired entries.
Describe alternatives you've considered
No response
Additional context
See https://github.com/microsoft/garnet/issues/1097 for a more detailed discussion of this.
@badrishc can this be assigned to me, if someone has not actively picked this up yet, since this will be something ARG will be needing.
Working on this here: https://github.com/microsoft/garnet/pull/1180
Will probably flush this out tomorrow.
I noticed that PR #1180 was closed with unmerged commits. Is there any update on when this issue will be resolved or if there's an alternative approach being considered?
Working on this here: #1180
Will probably flush this out tomorrow.
I noticed that PR #1180 was closed with unmerged commits. Is there any update on when this issue will be resolved or if there's an alternative approach being considered?
Better approach is being worked on. You really should help us with a stand alone repro if possible, not just code snippets. Program.cs that we can just dotnet run. Otherwise we are making improvements in a vacuum. Thanks!
Hi Badrish,
Please refer to the GitHub repository below to access my sample source code, which demonstrates the use of Hangfire recurring jobs. In this real-time scenario, all job logs are stored in the Garnet database with 3 minute auto-expiration. However, the database continues to grow indefinitely due to this functionality.
Git Path: https://github.com/wesleyisaacd/GarnetRepro.git
Code: https://github.com/wesleyisaacd/GarnetRepro/tree/master/WebScheduler