client: save preimages feature

[gabrocheleau]

This PR will add a savePreimages option to the client, which will save a key-value map of keyHashes to their preimages (pre-hashed keys). This is a required preliminary step to eventually implement the merkle->verkle transition logic.

Could also be somewhat related to #614

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 70.91837% with 57 lines in your changes are missing coverage. Please review.

Project coverage is 86.88%. Comparing base (b8f5b6d) to head (7a0842d).

Additional details and impacted files

Flag	Coverage Δ
block	88.34% <ø> (ø)
blockchain	91.61% <ø> (ø)
client	84.77% <84.84%> (+0.04%)	:arrow_up:
common	98.25% <100.00%> (+<0.01%)	:arrow_up:
devp2p	82.12% <ø> (ø)
ethash	∅ <ø> (∅)
evm	74.25% <50.00%> (-0.09%)	:arrow_down:
genesis	99.98% <ø> (ø)
rlp	∅ <ø> (∅)
statemanager	77.00% <75.00%> (-0.02%)	:arrow_down:
trie	89.24% <ø> (ø)
tx	95.45% <ø> (ø)
util	89.13% <ø> (ø)
vm	79.62% <50.00%> (-0.58%)	:arrow_down:
wallet	88.35% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[jochem-brouwer]

I would actually add this flag into Trie. Then, before you hash the key, save it into another DB (as preimage). This would only require an extra option for the DB to write too, though. But adding it into Trie means that we capture all trie items which are ever put, and we can be 100% (?) sure that we capture all preimages.

[gabrocheleau]

I would actually add this flag into Trie. Then, before you hash the key, save it into another DB (as preimage). This would only require an extra option for the DB to write too, though. But adding it into Trie means that we capture all trie items which are ever put, and we can be 100% (?) sure that we capture all preimages.

So I haven't pushed the whole WIP, as I was still playing around with some lower-level (vm/statemanager/trie) handling. I don't think we can strictly have that at the trie level, as it needs to be at least one-level higher up to handle the transition logic from trie->verkle-trie. Also, after discussing with gajinder, we though that the metaDB from the client seemed like a good place to put this. Nothing set in stone though, happy to hear your thoughts/ideas

My current plan is to have the following structure:

• the stateManager stores every preimage in memory. If the option is activated at the stateManager level, then anytime there is a putAccount operation, we add the preimage to the stateManager key/value map. I think that should cover all "touched accounts", right?
• then, at the runBlock level, once all the txs from the block have been executed, we retrieve those stored preimages from the statemanager and dump them. We would return the array of hashedKeys->preimages in RunBlockResult
• RunBlockResult can then be accessed by the client, and the preimages can be added to the metadb.

[jochem-brouwer]

the stateManager stores every preimage in memory. If the option is activated at the stateManager level, then anytime there is a putAccount operation, we add the preimage to the stateManager key/value map. I think that should cover all "touched accounts", right?

Yes, upon first thoughts I think this is right. What about the idea of Verkle where touched-but-deleted is different than in Merkle where touched-but-deleted does not change the trie? I think I recall this is dropped?

To test if all preimages are indeed stored in the StateManager I think we can add an extra test to the blockchain/state tests, what we do is we indeed copy all the MetaDB pre-image items, then after runBlock we copy the Trie and set the root to the root before the runBlock was ran. Then, we re-dump all these preimages in the copied trie and check if the root is the same reported root after runBlock.

[holgerd77]

Updated this via UI

[holgerd77]

Seems browser tests are not going through here?