Migrate to benchmarks in JSON format

[chfast]

The existing benchmark suite has been converted to JSON State Test format and is available in https://github.com/ipsilon/evm-benchmarks.

• [ ] Add evm-benchmarks as git submodule replacing test/benchmarks.
• [ ] Update evmone-bench tool to load benchmarks from JSON files. You can reuse StateTransitionTest and load_state_test() from https://github.com/ethereum/evmone/blob/master/test/statetest/statetest_loader.cpp#L198.

[miles170]

Could you mind helping clarify where the name and expected_output are in the evm-benchmarks JSON State Test format?

[chfast]

The names are in the labels section and they should match the elements of transaction.data. https://github.com/ipsilon/evm-benchmarks/blob/main/benchmarks/main/sha1_shifts.json#L8-L14 Our loader does not parse labels yet.

The output is simply not there. We proposed to include it but turned out this is substantial change in many tools. https://github.com/ethereum/tests/issues/998

The good news is we have a tool evmone-statetest (not main branch yet) which can execute and verify these files. Later we can reuse its implementation to verify benchmark files before execution.

For now you should check if a benchmark returns EVMC_SUCCESS. You can also check the status and gas used (if the same) in every benchmark loop iteration.