Implement t8ntool to use for execution-spec-tests

[jochem-brouwer]

Resolves #3602

Super WIP

Running with fill --fork Prague tests/prague/eip7702_eoa_code_tx/ --evm-bin=../ethereumjs-monorepo/packages/vm/test/t8n/start.sh --evm-dump-dir=tmp/

(does not work currently)

Instructions on how to run/test this:

• Clone https://github.com/ethereum/execution-spec-tests
• Checkout this PR: https://github.com/ethereum/execution-spec-tests/pull/752
• Install execution-spec-tests https://github.com/ethereum/execution-spec-tests?tab=readme-ov-file#quick-start
• Now it is possible to fill tests!

To fill tests, and verify this works, run for instance:

fill -vv -x --fork Cancun tests/cancun/eip1153_tstore/ --evm-bin=../ethereumjs-monorepo/packages/vm/test/t8n/start.sh

(-x: quits the filler once an error is encountered)

To debug t8n, one can add the --evm-dump-dir flag, this will dump all inputs/outputs into the dir set (which is very helpful). Also, -k will filter out any tests (to focus on certain tests which fail)

NOTE: currently points to the 7702-update branch. Do not merge this one before that one is merged!

TODOs

• [ ] Cleanup t8ntool script
• [ ] Add test to verify t8ntool works (sample inputs from execution-spec-tests, with expected output)
• [ ] Discuss changes in the vm/evm
• [ ] Make t8ntool hot-load server to skip the warmup time of the VM for each test (can be done in a separate PR)

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Please upload report for BASE (7702-update@ec85a45). Learn more about missing BASE report.

Additional details and impacted files

Flag	Coverage Δ
blockchain	84.92% <ø> (?)
common	89.68% <ø> (?)
util	70.20% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

[jochem-brouwer]

Do not spend time to review this, the branch is extremely messy

[gabrocheleau]

I am not too familiar with our testing architecture for consuming external tests like the execution-spec-tests, but I was on a verkle call this week, and learnt that there are now tests available for the Verkle EIPs. Link: https://github.com/ethereum/execution-spec-tests/tree/verkle/main/tests/verkle

Would this PR allow us to also consume those? This would be excellent for Verkle progress, and also great for us as a client if we can stand out and be early to get tests to pass.

@jochem-brouwer

[jochem-brouwer]

@gabrocheleau Yes, this tool allows us to "fill" those tests. This includes checks which are in those tests, for instance it can test if the post-state of something (for instance a storage slot, or a balance) matches the expected case. The output is the fixtures format, which we can also directly consume using blockchain/state tests.

I also noticed there are verkle releases of those tests, these can be directly consumed by our blockchain/state test runner. The only thing we have to do there is to ensure we are on the right fork (i.e. Verkle). Those tests can be found here: https://github.com/ethereum/execution-spec-tests/releases/tag/verkle%40v0.0.3

[jochem-brouwer]

Thanks a lot all, this was a huge process, but super happy to get this merged now :smile:

Together with https://github.com/ethereum/execution-spec-tests/pull/752 we are now ready "out-of-the-box" to fill tests :smile: :rocket:

[holgerd77]

🎉🎉🎉

[jochem-brouwer]

Hi @holgerd77, the slow stuff of t8n has not much to do with using MCL. It is simply the load time of all our libraries. It is done each time t8n is called, which can be done multiple times per test (in fact, for each block t8n is called).

I have on the agenda a separate PR to open up a server daemon which will then remove this "cold load" time and instead "hot load" the VM. This will drastically speed up the runtime of this tool (for multiple tests).

You could think of this when running the Blockchain tests, except now you dont run it with the default settings but instead you run npm run test:blockchain -- --test=TEST. This is essentially what is happening right now with t8n, which is obviously very slow.

Note that for BLS tests MCL is indeed somewhat faster (we have seen in the Blockchain tests a decrease from ~7 seconds to something like ~6 seconds). However that is for all tests, this ~1 second change is already what would be saved if we would "hot load" the libraries.

I plan to add this server daemon asap!

[holgerd77]

Hi @holgerd77, the slow stuff of t8n has not much to do with using MCL. It is simply the load time of all our libraries. It is done each time t8n is called, which can be done multiple times per test (in fact, for each block t8n is called).

I have on the agenda a separate PR to open up a server daemon which will then remove this "cold load" time and instead "hot load" the VM. This will drastically speed up the runtime of this tool (for multiple tests).

You could think of this when running the Blockchain tests, except now you dont run it with the default settings but instead you run npm run test:blockchain -- --test=TEST. This is essentially what is happening right now with t8n, which is obviously very slow.

Note that for BLS tests MCL is indeed somewhat faster (we have seen in the Blockchain tests a decrease from ~7 seconds to something like ~6 seconds). However that is for all tests, this ~1 second change is already what would be saved if we would "hot load" the libraries.

I plan to add this server daemon asap!

Ah ok, makes a lot of sense, thanks for the insight! 🤯

Also thanks for planning ahead on this already! 🙏