Add `EthersStateManager`

[acolytec3]

Fixes #2019

This is preliminary but takes the first step towards an EthersStateManager.

• [x] Use @ethersproject/provider to source data for getAccount/getContractCode/getContractStorage
• [x] Add support for setting the block number to reference when retrieving state from the provider
• [x] Leverage getProof (where available from the provider) to validate account existence
• [x] Add support for getStateRoot methods
• [x] Handle getContractStorage correctly
• [x] Add more integration tests with runBlock/runTx to verify that data sourced from the Ethers provider is being handled correctly
• [x] Add some tests that recreate a transaction and block from Mainnet using data sourced from Ethers provider
• [x] Add better unit tests with truly fake provider rather than relying on cloudflare
• [x] Fix browser tests (I fixed them by skipping all the API tests :grimacing: )
• [x] Validate that putContractStorage works right

[codecov[bot]]

Codecov Report

Merging #2059 (9037bb8) into master (3f7594a) will decrease coverage by 0.03%. The diff coverage is 85.17%.

Additional details and impacted files

Flag	Coverage Δ
block	92.86% <85.71%> (+0.04%)	:arrow_up:
blockchain	88.47% <ø> (ø)
client	86.95% <ø> (ø)
common	98.09% <ø> (ø)
devp2p	92.37% <ø> (-0.03%)	:arrow_down:
ethash	∅ <ø> (∅)
evm	79.23% <ø> (ø)
rlp	∅ <ø> (∅)
statemanager	87.22% <85.16%> (-1.24%)	:arrow_down:
trie	90.36% <ø> (ø)
tx	97.98% <ø> (ø)
util	91.67% <ø> (ø)
vm	85.31% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[acolytec3]

I'm sure there's more work to be done but this should be a good first step. Will work on adding some more tests to try and recreate real transactions and blocks using data sourced from mainnet.

[acolytec3]

I'm not sure why the karma tests are failing but will look at that once I get some comprehensive tests.

[acolytec3]

I'm going to pause work on this. We now have a reasonably complete proof of concept where a user can leverage any ethers.jsonRpcProvider to source data and run at least simple tests with the vm without data preconditioning. My initial tests are just a simple runTx with a transfer of some ETH to Vitalik and then running a block that has no transactions in it from the ancient chain history, but I think this demonstrates the concept is sound.

Next steps on this are to decide if we want to integrate this as an alternative state manager implementation exposed by our statemanager class or just split it out as a standalone representation of what's possible.

If we want to integrate it in statemanager, I will make the tests truly unit tests with mocked responses from the provider and add some better integration tests using the cloudflareprovider (which is a subclass of ethers.jsonrpcprovider) so we can get a sense for how it works.

@holgerd77 and @ethereumjs/committers, over to you for opinions at this point.

[holgerd77]

Great stuff, cool! ❤️

Just a general note since I stumbled upon this now a couple of times already: we might want to split our StateManager interfaces in a slightly different/evolved way and do a StatelessStateManager interface and then a full (Statefull)StateManager interface, with the first one taking out the stateful operations (checkpoint/commit/revert/setStateRoot?).

Eventually the BaseStateManager needs some alignment (split ?) along this as well?

This needs some expansion respectively some deeper thinking I guess.

[holgerd77]

(updated the last post on site)

[acolytec3]

Alright all, I think this is ready for a semi-final round of review. I'd like for folks to build and run it locally and see if you're satisfied with what it offers or let me know if there are things we should change. At the moment, it supports retrieving state from the provider and running "most" blocks, unless you happen to be one of those edge cases where a storage slot is zeroed or an account self-destruct occurs and that results in a branch node having only a single leaf node that gets deleted (which also means the branch node gets deleted). In this edge case, we don't have a way via the web3Provider interface to retrieve the surrounding inner nodes we need to update the state trie so we wouldn't be able to compute the proper state root for those blocks. Beyond that edge case, it seems to work pretty reliably as far as running blocks, running transactions, etc.

One thing I haven't done yet is implemente a "reset trie" method which would let you completely reset the trie, start the provider on a different block, and then start running things again, but that should be trivial.

Look at the tests I've written and see if that covers all the ground you'd like to cover in web3provider driven State Manager and if it looks good, I'll rework the tests with mocked provider responses so we have proper unit tests and then this might be about merge ready.

[acolytec3]

Alright team, this one is now ready for a first/final round of review. It does the following:

1. Fixes the fromRpc methods and test data in block to work with our current code base and the data provided by ethers providers
2. Adds a new EthersStateManager class that implements the StateManager interface.

• This state manager can execute transactions and blocks in the vm package using data sourced from any ethers JsonRpcProvider - works best with archive nodes so ideally alchemy or infura (though will work with CloudFlare as long as you aren't looking at ancient chain history).
• Adds a new mockProvider class for use in unit testing the new state manager

cc: @ricmoo would be happy for any additional input you have on this.

Thanks to @jochem-brouwer for many hours of help troubleshooting the work to allow us to run blocks (or at least most blocks) with data only sourced from a provider (and not devp2p)

[g11tech]

Wow! really amazing PR! @acolytec3, @jochem-brouwer :heart: :fire: :rocket:

successfully able to hookup the jsonrpc to my local mainnet node and able to run this block successfully: https://etherscan.io/block/15496421

Feels really close

[gabrocheleau]

General comment that the new deps increase the npm audit issues, before:

after:

the additional high severity issue is used in production:

I checked some of these but am not clear on if they are problematic for us or not.

[acolytec3]

I think this is ready for final review and/or merging at this point. I made some last edits to the readme to note the gotchas related to slot deletions/account self-destruct. It doesn't really get into the nuts & bolts of how it works but hopefully it at least alerts folks to the potential outcomes.

[holgerd77]

Superseeded by #2315, will close.