flow-go icon indicating copy to clipboard operation
flow-go copied to clipboard

[Storage] Upgrade to Pebble 2

Open zhangchiqing opened this issue 5 months ago • 2 comments
trafficstars

Work towards: https://github.com/onflow/flow-go/issues/7265

This PR upgrades:

  • pebble from v1.1.2 to v2.0.3
  • github.com/ipfs/go-ds-pebble from v0.3.1 to v0.8.2 (which depends on pebble v2.0.3)

Tested that the upgrade is backward compatible, which means the following process works:

  • running execution node with protocol state and execution data in badger
  • running db-migration to copy all key-value pairs from badger to pebble for protocol state and execution data in pebble 1.0
  • start up the execution node with pebble 2, and the node should be able execute blocks, this means pebble 2 is able to recognize the sstable created by pebble 1 in previous step.
  • also verified that chunk data pack database stored in pebble 1 format can be read by pebble 2 (backward compatible)

zhangchiqing avatar Jun 03 '25 18:06 zhangchiqing

Codecov Report

Attention: Patch coverage is 66.66667% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
storage/migration/sstables.go 70.00% 3 Missing :warning:
...cutiondatasync/storage/pebble_datastore_manager.go 0.00% 1 Missing :warning:

:loudspeaker: Thoughts on this report? Let us know!

codecov-commenter avatar Jun 03 '25 18:06 codecov-commenter

The TestRemoveDiskUsage/PebbleStorage test case is failing, seems without additional config pebble2 can not reclaim disk space after removing data and triggering compaction. Will have to give another look.

zhangchiqing avatar Jun 23 '25 21:06 zhangchiqing