hive icon indicating copy to clipboard operation
hive copied to clipboard

rpc-compat: Run `speccheck` on non-deterministic tests

Open lightclient opened this issue 3 years ago • 4 comments
trafficstars

Some methods are not deterministic across clients. For example, eth_gasPrice returns as gasPrice in wei that users could use for a transaction to be included quickly. This is subjective based on client so a static test case doesn't make sense. However, we know what the shape the request and response should look like. So it would make sense to make the request anyways and verify the round trip against the spec.

I've written a tool speccheck which more-or-less does this. However it doesn't yet have the ability to verify round-trips that aren't written to file in the format of the test fixtures. So it would need to be librarified to a degree and then imported into rpc-compat and run against certain non-deterministic endpoints (maybe even all of them!).

lightclient avatar Aug 22 '22 16:08 lightclient

I'm adding a speconly flag in ethereum/execution-apis#520 so we can at least mark the tests where this kind of checking should occur.

fjl avatar Jan 30 '24 12:01 fjl

This is a really nice idea, I am happy to begin using it.

lightclient avatar Jan 30 '24 15:01 lightclient

I'm working on it right now!

fjl avatar Jan 30 '24 18:01 fjl

Parsing of the flag is now implemented in the simulator, but it's unclear to me how to proceed with adding speccheck. I don't want to depend on rpctestgen from the simulator, it defeats the purpose of having the generator and test output separated.

fjl avatar Feb 01 '24 20:02 fjl