namada
namada copied to clipboard
anoma
Update to CometBFT v1
After the storage fix, RocksDB growth is quite demure, but now CometBFT is coming for our poor hard drives.
While the system requirements for running a full node recommend 1 TB drives, we experienced how unwieldy 200 GB databases are (see post-mortem). For non-archive nodes we should aim to stay well below that, balancing storage growth, performance and maintenance against a high level of integrity and security granted by retaining a large fraction of the chain.
Currently Namada mainnet is in phase 1 with I assume minimal network activity and participants. With those variables CometBFT already grows at about 0.5 GB per day. At this rate it will take under a year, 200 days, to reach 100 GB. I expect database growth to increase dramatically once IBC transactions are enabled in phase 3, and more and more integrations and participants are active. Our (drives') days are counted!
Namada runs on CometBFT v0.37, which does not have fully fleshed out pruning features, if at all. Recently CometBFT v1.0 RC2 has been published, which adds more capable pruning.
Considering the necessary work and time involved to upgrade Namada to CometBFT v1.0 and the opportune time window where the database is still manageable closing in under a year, I suggest to start looking into updating CometBFT not long after Namada phase 5 and the stable release of CometBFT v1.0.
Thank you for the suggestion.
I think CometBFT v1.0 will offer the pruning API. CometBFT v0.37 has the pruning function and can prune the old block data.
Currently, Namada returns 0 as retain_height in the commit response to CometBFT not to prune blocks in CometBFT.
https://github.com/anoma/namada/blob/08c16b74c0345542c5a89f5b4d60223ec47a10d9/crates/node/src/shell/mod.rs#L804-L806
In my experiment, the block data in CometBFT could be pruned with CometBFT v0.37. I tried modifying the logic to return the snapshot height where the Namada RocksDB snapshot is taken: yuji/prune-cometbft-blocks
Please note that the pruned CometBFT data can't be restored without a backup of CometBFT or resync from other nodes which have all data of CometBFT (The above Namada's snapshot is for state sync and has only Namada's RocksDB data in this case), i.e. some nodes have to keep all data of CometBFT for block sync (sync from genesis.
CometBFT v1 has additional gRPC service that allows to control pruning (https://docs.cometbft.com/v1.0/explanation/data-companion/pruning) that allows to prune block results as well as blocks. The retain_height parameter in ABCI Commit response only controls block pruning. I think we could make this configurable, but let's open a different issue for it and keep this issue just for tracking CometBFT v1 upgrade.
