Badger2 integration

[gammazero]

This is a provisional PR and should not be merged until dependencies are merged into their master branches.

• Add badger2 plugin
• Update docs to describe badger2
• Add tests for badger2
• Set dependencies to pending PRs with necessary updates

[welcome[bot]]

Thank you for submitting this PR! A maintainer will be here shortly to review it. We are super grateful, but we are also overloaded! Help us by making sure that:

• The context for this PR is clear, with relevant discussion, decisions and stakeholders linked/mentioned.

• Your contribution itself is clear (code comments, self-review for the rest) and in its best form. Follow the code contribution guidelines if they apply.

Getting other community members to do a review would be great help too on complex PRs (you can ask in the chats/forums). If you are unsure about something, just leave us a comment. Next steps:

• A maintainer will triage and assign priority to this PR, commenting on any missing things and potentially assigning a reviewer for high priority items.

• The PR gets reviews, discussed and approvals as needed.

• The PR is merged by maintainers when it has been approved and comments addressed.

We currently aim to provide initial feedback/triaging within two business days. Please keep an eye on any labelling actions, as these will indicate priorities and status of your contribution. We are very grateful for your contribution!

[aschmahmann]

@gammazero your go.mod files are conflicting with master. Could you rebase when you get a chance?

[aschmahmann]

This is blocked by the badger folks figuring out if there latest breaking changes will be handled within badger v2 or whether there will be a badger v3. The go-ipfs-config PR adding badger 2 support has been reverted until this is resolved.

[dokterbob]

@aschmahmann There's now a Badger v3 https://github.com/dgraph-io/badger/releases/tag/v3.2011.1

However, go-ds-badger2 is actively maintained. Requirements mentioned by you have been met a long time ago and, even as an experimental feature, it could gain great benefit and insight to have something less broken to test with. This might also yield useful feedback while a move towards badger 3 is made.

We currently have a badgerdb in IPFS which is broken in obvious ways making it unsuitable for production.

[dokterbob]

In short; all that's necessary to have a badger v2, based on actively maintained code, in master, is to resolve these merge conflicts. Getting v3 support is probably a fair amount of work and might delay having something working by several months/up to a year.

It would be great to have this in user's hands!

[godcong]

I tried to use V3, but it will panic in some cases.

[dokterbob]

@aschmahmann Poke!

[dokterbob]

At ipfs-search.com we currently have a script running from a cron job which stops IPFS, flattens the datastore using the badger binary and then restarts IPFS.

I know you've got a lot on your hands, but it would be great to see some progress on this one!

[dokterbob]

Poke!

[feld]

flatfs is not good when you've got 10s to 100s of gigs in IPFS. One of my use cases can survive with a manual flatten as suggested by @dokterbob but it would be nice if we had badger v2 or v3 to work with

edit: i may just rebase this on kubo v14.0 and badger v2.2007.4 and see what happens...

edit2: also learned that flatten is not the same as garbage collection; flatten is a different thing in Badger where it consolidates storage layers but has nothing to do with freeing up space. But freeing up some space could be a side effect.

[feld]

@dokterbob I have a branch with Badger2 working on IPFS v0.14.0 and garbage collection does work properly

https://github.com/feld/kubo/tree/badger2-rebase-v0.14.0