cortex icon indicating copy to clipboard operation
cortex copied to clipboard

Add badger index cache

Open SungJin1212 opened this issue 1 year ago • 4 comments

Introduce a badgerDB (https://github.com/dgraph-io/badger) as a disk index cache. It could be a middle layer cache between in-memory and remote cache (in-memory -> (badger) -> Memcached or Redis).

There are two loop in the badger index cache, retentionLoop and diskStatUpdateLoop. retentionLoop: periodically run badgerDB GC to sink disk space. diskStatUpdateLoop: exports metric badger_value_log_space_available_bytes and badger_value_log_space_available_bytes track amount of disk space left on the value and key log used in Badger at mount point in bytes.

Which issue(s) this PR fixes: Fixes #6241

Checklist

  • [X] Tests updated
  • [ ] Documentation added
  • [x] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

SungJin1212 avatar Oct 10 '24 11:10 SungJin1212

@yeya24 Using Size, we can track bytes usage of vlog, lsm. But, I should do more digging to delete keys when some threshold reached.

SungJin1212 avatar Oct 14 '24 06:10 SungJin1212

This is a very interesting PR.. at some point i was thinking on doing something similar.

@SungJin1212 were you able to do some load test on this to see how it performs compared with the other caching solutions?

alanprot avatar Oct 24 '24 23:10 alanprot

@alanprot Thank you for your interest in my work. I haven't done the load test yet, but I was thinking that the load tests are necessary. Which performance aspects are essential in the load test do you think?

SungJin1212 avatar Oct 25 '24 01:10 SungJin1212

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Apr 26 '25 17:04 stale[bot]