badger icon indicating copy to clipboard operation
badger copied to clipboard

GC doesn't seem to run

Open lkp-k opened this issue 1 year ago • 6 comments

What version of Badger are you using?

Latest v4

What version of Go are you using?

1.20.3

Have you tried reproducing the issue with the latest release?

Yes

What is the hardware spec (RAM, CPU, OS)?

16gb ram, i5 intel, mac os

What steps will reproduce the bug?

package main

import (
	"fmt"
	"log"
	"math/rand"
	"strconv"
	"time"

	"github.com/dgraph-io/badger/v4"
)

var db *badger.DB

const charset = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"

func generateRandomString(length int) string {
	b := make([]byte, length)
	for i := range b {
		b[i] = charset[rand.Intn(len(charset))]
	}
	return string(b)
}

func main() {
	opts := badger.DefaultOptions("badger")
	opts.ValueLogFileSize = 1 << 20
	var err error
	db, err = badger.Open(opts)
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()

	fmt.Println("Start inserting 5 million items.")

	for i := 0; i < 5_000_000; i++ {
		err := db.Update(func(txn *badger.Txn) error {
			key := strconv.Itoa(i)
			value := []byte(generateRandomString(300))

			e := badger.NewEntry([]byte(key), value).WithTTL(30 * time.Second)

			err := txn.SetEntry(e)
			if err != nil {
				return err
			}
			return nil
		})

		if err != nil {
			log.Fatal("Error while inserting:", err)
		}
	}

	fmt.Println("Successfully inserted 5 million items.")

	startGarbageCollection()
}

func startGarbageCollection() {
	ticker := time.NewTicker(10 * time.Second)
	defer ticker.Stop()
	for range ticker.C {
	again:
		err := db.RunValueLogGC(0.01)
		fmt.Println(err)
		if err == nil {
			goto again
		}
	}
}

Expected behavior and actual result.

Since the keys are expiring in 30 seconds, I thought that the files in badger/ would get removed.

However, heres what I see:

image image

Additional information

No response

lkp-k avatar Sep 06 '23 17:09 lkp-k

may be related to #1995. Thanks for filling an issue with code to reproduce. I will try to find some time to look into it.

mangalaman93 avatar Sep 06 '23 17:09 mangalaman93

Maybe the compaction is not triggered. Similar to https://discuss.dgraph.io/t/gc-may-not-work-in-some-cases/17197

wangyang0918 avatar Sep 17 '23 07:09 wangyang0918

Hi Friends. Any updates on this? Ty

lkp-k avatar Sep 29 '23 23:09 lkp-k

@mangalaman93 Would it possible for you to take a look when you have some time ? Thank you !

GraphR00t avatar May 18 '24 13:05 GraphR00t

This issue has been stale for 60 days and will be closed automatically in 7 days. Comment to keep it open.

github-actions[bot] avatar Jul 18 '24 14:07 github-actions[bot]

Any updates ? Thanks !

GraphR00t avatar Jul 18 '24 14:07 GraphR00t