dragonfly icon indicating copy to clipboard operation
dragonfly copied to clipboard

feat (tiering): implementing periodic defragmentation for second tier

Open theyueli opened this issue 1 year ago • 1 comments

fixes https://github.com/dragonflydb/dragonfly/issues/2433

This update introduces a new feature that regularly performs a defragmentation process by scanning "external keys". If a key's SSD page has a bin utilization lower than a certain threshold, all the keys' values will be loaded back into memory, and offloading will be rescheduled for each. This process aims to consolidate fragmented keys into a new block, which will be better utilized.

Todo: will move the defrag into a different place rather than abusing the heart beat function.

theyueli avatar Feb 15 '24 03:02 theyueli

@adiholden For now, I did not scan the reference count table as that table doesn't provide the bin size ... of course, one can always grab a hash, get the iterator, then get the length, and then recalculate bin_size ... but I felt it is just a bit easier to directly obtain it from the PrimeIterator passed to the defrag function from db_slice.

theyueli avatar Feb 15 '24 03:02 theyueli

please add unit tests to your code

adiholden avatar Mar 05 '24 08:03 adiholden