celestia-node
celestia-node copied to clipboard
feat(pruner): Implement `full` and `bridge` node pruning
This PR introduces full and bridge node pruning via the --experimental-pruning flag. Support is included for nodes that start from scratch with pruning enabled and also for archival (nodes retaining all historical blocks) that enable the --experimental-pruning flag.
Note that this PR does not support the conversion of a pruned node into an archival one explicitly (it would not support re-syncing deleted blocks).
With pruning enabled, full and bridge nodes' block stores can be expected not to exceed ~4TB (as the upper bound).
In follow-up PRs (hardening), the following features can be expected:
- [ ] discovery for archival nodes for archival sync
- [ ] inverted_index / light node pruning
- [ ] include more metrics for errors
TODO:
- [x] clean up some TODOs
- [x] fix one flakey unit test
- [x] change values back to the actual (GC cycle, sampling window, pruning window, etc).
- [x] figure out whether to store error in pruner checkpoint
- [x] fix issue with pruning genesis block via findPruneableHeaders
- [x] metrics for failed prunes
- [x] set a sane default for max pruneable / consider removing
MaxPruneablePerGCas now context timeouts are on a per block basis - [ ] dedup findPruneableHeader test utility
- [x] badger dep
Codecov Report
Attention: Patch coverage is 60.08584% with 93 lines in your changes are missing coverage. Please review.
Project coverage is 52.89%. Comparing base (
1809680) to head (5918f6d). Report is 6 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@
## main #3150 +/- ##
==========================================
+ Coverage 51.92% 52.89% +0.97%
==========================================
Files 178 183 +5
Lines 11316 9539 -1777
==========================================
- Hits 5876 5046 -830
+ Misses 4942 3980 -962
- Partials 498 513 +15
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
TODO @renaynay:
add some protection in finder.go to prevent
2024-02-28T16:10:59.333+0100 ERROR pruner/service pruner/service.go:129 failed to find prune-able blocks {"error": "header/store: invalid range(26625,26624)"}
(can sometimes happen but is non-critical error, just ugly)
@renaynay, there is a little conflict fyi