server icon indicating copy to clipboard operation
server copied to clipboard

[badgerdb] vlog growing unbounded - consider adding GC and exposing options

Open woodzy opened this issue 4 months ago • 6 comments

I noticed that BadgerDB storage started growing unbounded (it was many files and several gigabytes quite quickly). Looks like the Badgerhold/Badger Storage example doesn't provide a way for the user to utilize GC correctly per Badgers docs. https://github.com/dgraph-io/badger/blob/main/db.go#L1225

Fairly sure the option tuning is probably useful to expose as well somehow -- for example ValueLogFileSize has a pretty large default

I didn't plumb up a full fix, just copied the default and started experimenting, but this showed that something like option tuning and GC can resolve the infinite growth modified a copy of the default to include some signalling and a regular GC call.

I would recommend at least adding the GC as a default. I would also suggest lowering some of the default or exposing them.

        go func() {
                ticker := time.NewTicker(5 * time.Minute)
                defer ticker.Stop()
                for { 
                        select {
                                case <- ticker.C:
                                        _ = h.db.Badger().RunValueLogGC(0.7)
                                        continue
                                case <- h.doneCh:
                                        return
                        }
                }
        }()

woodzy avatar Feb 19 '24 05:02 woodzy