btcd
btcd copied to clipboard
btcsuite
database: investigate replacing leveldb with pebble
Today we use a pure Go implementation of leveldb for the main database and relevant indexes. The DB has been able to keep up pretty well over the years, but it's maintained by more or less a single person, and hasn't received any significant commits over the past year or so. As a result, we should consider moving to a better maintained database that can be a drop in replacement for our current usage.
Enter pebble as pure Go KV database inspired by rocksdb (which was inspired by leveldb). One can view this as the 3rd generation of LSM (Log-Structured-Merge) Tree based KV stores. Compared to goleveldb pebble is actively maintained, and has been in used by Cockroachdb in production for ~3 years. The API is very similar to goleveldb, which means we won't need a significant overhaul.
Rather than replace it all at once, we should first add it as a new database option, with no migration path. After enough time, we can add a migration path, t hen eventually deprecate the old database type. At a glance, pebble also appears to be more performant than goleveldb.
FWIW leveldb itself hasn't had that much development either. I don't think it necessarily means lack of maintenance.
That being said, I'm not opposed to having pebble as an option. If it ends up being more performant, that's a win. Though maybe have a big warning that pebble itself may not be consensus compatible with leveldb.
If no enough performance improvement, i think there is no need to replace leveldb with pebble. To replace, we must do a lot of test for pebble to check whether there are internal bugs like missing data or incorrect values when insert or get.
Another point, maybe keepping building pebble means that there are bugs. leveldb is enough stable and safe.
@Roasbeef Can I proceed with this task? using Pebble enabled synchronization more than 3~5x faster in my benchmark.
@Roasbeef Can I proceed with this task? using Pebble enabled synchronization more than 3~5x faster in my benchmark.
Saw that you have code already. You should push it and we'd be happy to take a look.
@kcalvinalvin Awesome! However, ffldb and leveldb are currently coupled tightly, making it hard to replace pebble. I`m working on additional interface between ffldb and leveldb(db engine). Any feedback or reviews are always welcome!
Did a quick test up to around block 400,000 with the following command:
./btcd --datadir=. --logdir=. --connect=192.168.0.215 --utxocachemaxsize=2000 --cpuprofile=cpuprof
With binaries built on master and https://github.com/rabbitprincess/btcd/commit/af784fb2cb080630af1b5aa740d920d45389207b
On master:
On https://github.com/rabbitprincess/btcd/commit/af784fb2cb080630af1b5aa740d920d45389207b
We can see that the db.View and db.Update are taking up much less space on the flamegraph with pebble. Seems promising enough.
Great work @rabbitprincess, this is indeed very promising!
We should also heed the early comment from @Eoous:
To replace, we must do a lot of test for pebble to check whether there are internal bugs like missing data or incorrect values when insert or get.
Another point, maybe keepping building pebble means that there are bugs. leveldb is enough stable and safe.
Thank you @Roasbeef! As you mentioned, additional verifications are necessary.
Requirement
- sync with leveldb should remain unchanged.
- sync with pebbledb can work all networks.
Tasks
- Make db test suite for db engines ( leveldb, pebbledb )
- compatible test for leveldb (include running existing node for new branch)
- full-sync test for pebbledb node ( almost done; the sync time for bitcoin mainnet is ~7 days with new db )
Please let me know if you have any requests or suggestions!