Feature/Era1 import/export

[ak88]

Introduces a new archive format called Era1, which is based on the Era spec made by nimbus, and this proposal from arnetheduck.

The purpose of Era1 is to provide the same functionality for EL, that Era provides for the CL. An easily verifiable and shareable format for pre-merge historic EL data, which is considered a prerequisite for eip-4444 . Era1 only concerns pre-merge data, since post-merge EL payload is contained in the beacon chain.

In the long term the idea is to distribute historic data through the portal network.

Underneath Era1 is based on e2store which is a simple type-length-value scheme.

Specification

An Era1 archive can be expressed like so

   era1 := Version | block-tuple* | other-entries* | Accumulator | BlockIndex
   block-tuple :=  CompressedHeader | CompressedBody | CompressedReceipts | TotalDifficulty
   block-index := starting-number | index | index | index ... | count

Headers, bodies and receipts are compressed using the snappy framing format. Additionally each file contains a block index for fast lookup, and an Epoch accumulator for verification.

The epoch accumulator can be used to verify the entire archive with accumulators from a trusted source. Additionally it allows a node to download a header with a merkle-proof, that proves it belongs to a certain epoch.

The Epoch accumulator is defined in the portal network spec here.

Changes

• File reader and writer for Era1 archives
• Era1 export from the admin RPC
• Era1 accumulator verification from admin RPC
• Import Era1 in full archive mode from directory defined in sync config

Types of changes

What types of changes does your code introduce?

• [ ] Bugfix (a non-breaking change that fixes an issue)
• [x] New feature (a non-breaking change that adds functionality)
• [ ] Breaking change (a change that causes existing functionality not to work as expected)
• [ ] Optimization
• [ ] Refactoring
• [ ] Documentation update
• [ ] Build-related changes
• [ ] Other: Description

Testing

Requires testing

• [x] Yes
• [ ] No

If yes, did you write tests?

• [x] Yes
• [ ] No

Documentation

Requires documentation update

• [x] Yes
• [ ] No

If yes, link the PR to the docs update or the issue with the details labeled docs. Remove if not applicable. TODO

Requires explanation in Release Notes

• [x] Yes
• [ ] No

If yes, fill in the details here. Remove if not applicable.

Remarks

I placed the export and verification functionality in admin RPC, which you have to enable before it can be called. I am open to suggestions for better places for it.

Era1 as lookup

During project discussions we also wanted to use Era1 archives as a freezer for old data. This would allow nethermind to serve bodies and receipts from beyond the barriers, and in addition it would allow a user to store "cold" data on a cheap storage device, since the archives can be placed in a separate location. This functionality has been cut for now.