Receipt Compression Strategy

[non-fungible-nelson]

Description

The largest part of the database storage is the receipts. As a Besu user, I want the lowest storage size possible on my disk.

Some clients have a notion of "ancient" receipts that they compress or move to a freezer to keep storage small. For more recent data, Besu can provide non-compressed receipts/data. For older data, decompression will happen, latency may be introduced, which is a tradeoff for lower overall storage use.

We need to determine what "ancient" (user configurable) barriers are appropriate, what compression strategy we may use for these ancient receipts, and if we want to combine it with the freezer approach.

Some references & implementations: https://github.com/ethereum/go-ethereum/pull/19345 https://github.com/NethermindEth/nethermind/pull/5531

We may also be able to prune receipts over the historical-block-limit for BONSAI, reducing data further (is this covered by --Xchain-pruning? ).

Notes

• may also mean we can improve sync because there's less data to write?
• storing bloom filter is useless because you can recreate it?
  • maybe rocksdb is already compressing all the leading zeros?
  • https://github.com/hyperledger/besu/pull/5392 - not great results but Karim didn't spend much time on it
• Could also compress other parts of the body?
• Could we push to modify snapsync spec to avoid needing to download the receipt
  • because it can be recreated anyway
  • then maybe don't need to store the receipt at all
• often addresses are stored multiple times in the receipt?
  • related to https://github.com/ethereum/go-ethereum/pull/19345#issue-426054825 ? -> "The big wins come from not storing the transaction hash and contract address since it is not amenable to the snappy compression used by LevelDB"
• Are receipts stored for forks as well? c.f. trie log storage pruning #5390
• Avoiding a "pruning"-style operation by always storing in a compressed format would have some benefits
• Maybe split out some improvements we can do for all blocks, including near-head VS some we can only apply to "ancient data"

Tasks

• [x] Determine if receipts are used for protocol operation or for anything other than RPCs and snapsync.
  • Consider EIP-6110? https://github.com/hyperledger/besu/blob/1c261db30f5cb9504e86b6b24eb99502d17b6336/ethereum/blockcreation/src/main/java/org/hyperledger/besu/ethereum/blockcreation/AbstractBlockCreator.java#L298-L304
• [x] Prototype the other ELs receipt encoding and spike on a similar approach.
• [x] Investigate the possibility of not storing bloom filter and generating that on demand.
• [x] Have a chat with Karim to try understand what the results were for the attempt on this PR: https://github.com/hyperledger/besu/pull/5392 - not great results but Karim didn't spend much time on it. maybe rocksdb is already compressing all the leading zeros?
• [x] Investigate and (possibly) spike on optimise the address stored multiple times in the receipts - related to https://github.com/ethereum/go-ethereum/pull/19345#issue-426054825 ? -> "The big wins come from not storing the transaction hash and contract address since it is not amenable to the snappy compression used by LevelDB"
• [x] Investigate if we're storing receipts for forks and how much storage pruning those would save - did a rough calculation based on some nodes' number of forks and size of a receipt and decided it wasn't worth the effort so closed this: https://github.com/hyperledger/besu/issues/6424
• [x] Test RPCs work correctly with receipt changes

[non-fungible-nelson]

we need to engage the community at large around needing receipts - in the DB and in the snap sync protocol.

Create a discussion in Eth R&D

[non-fungible-nelson]

@jframe what is the status on this epic?

[jframe]

I have a draft PR with the changes for receipt compression. Currently doing some additional testing before I open the PR for review.