ethereum
Deprecate StateTest format
The StateTest format has many disadvantages for users and is very close to the single-transaction BlockchainTest.
What is blocking us from using only BlockchainTests?
Disadvantages
- Weird multi-transaction definition. It does not make sense for generating tests. Potentially, it can save some disk space, but in practice both ethereum/tests and execution-spec-tests generate the same tests in BlockchainTest and StateTest formats. This is also bad for users because they need special transaction representation only for StateTests.
- Hashed post state. Users are not able to tell what's wrong with a failing test because they only can see that the state hash mismatch. Moreover, computing the state hash is not trivial (e.g. evmone implemented MPT just for that).
Yes, definitely agree on 1, it's not good and it makes the definition of our types way more convoluted than it has to be, and this is just the result of us trying to adapt the existing fixture format without breaking anything.
The use I see of the state test format is that you can test for rejected transactions more easily than in the blockchain test format:
In the blockchain test format, if the cause of invalidation is an invalid transaction, you get a passing test if the client rejects the block, but if you want to make sure that it was rejected because the transaction was deemed as invalid (and not some other block validation error) you need to check the error string.
In the case of the state test you check that the state hash did not change and then you can be sure the transaction was the cause of the rejection.
IMO I think we need to standardize the block invalidation errors before phasing out the state test format.
The t8n has the "rejected" array where it puts the invalid transactions with error messages.
The multy transaction types are fillers only. When you parse the generated test you don't need to support it. Use txbytes field
The state hash could be provided with full state no problem.
Looks like we all still love and use state test format, and it comes in handy. Especially with fuzzing.