ethereum
feat(tests): port ethereum/tests test cases (tracking issue)
This is a top-level issue for tracking the manual porting of ethereum/tests test cases to execution-spec-tests:
Contributions to manually port individual test cases are very welcome!
If you're already familiar with ethereum/tests and execution-spec-tests, jump to the issue tracker below!
Background
ethereum/tests was the repository used to define EVM test cases from Frontier up to and including The Merge. The test cases are defined as YAML (and sometimes JSON) files in the ./src/ sub-directory and the JSON "test fixtures" (fully-populated tests that can be executed against clients) are generated using ethereum/retesteth. These JSON artifacts are regenerated when required and added to the repository, for example, in the ./GeneralStateTests/ sub-directory.
From Shanghai on, new test cases (in particular for new features included in hard-forks) have been defined in Python in this repository (execution-spec-tests). The existing test cases are still important to execute against clients, however, and porting ethereum/tests to execution-spec-tests ensures that the test cases can be continually maintained and filled (generated) for newer forks. It also has the advantage that client teams will only need to obtain test fixture releases from one source.
Automated Port to Python
Although there will be an effort to automate ("transpile") the remaining test cases from YAML (respectively JSON) to Python; contributions to manually port individual test cases can be very beneficial and are welcome!
The benefits of manual porting are:
- Reduce the number of test cases by combing multiple YAML (resp. JSON) test cases in one Python test function (via parametrization).
- Potentially improve coverage by parametrizing the Python version.
- Code/docstrings are likely higher quality than an automated conversion.
- The test can be better organized within the
./testsfolder in execution-spec-tests by the fork and the EIP in which the functionality was introduced.
Process
- Pick a single or a set of YAML test cases in ethereum/tests to port and create an issue in this repository - optional, but definitely recommended if porting tests from the tracker below to avoid duplicate work.
- Create a PR with the ported tests: a. Add the list of ported YAML files to converted-ethereum-tests.txt. b. If the tests can't currently be filled, please explain the issue (feel free to also open a Discussion).
Help with ethereum/tests
The test formats are described in the ethereum/tests online docs. Note that state filler test cases in YAML can be parametrized, see the docs here.
Hints to Get Started with ethereum/execution-spec-tests
- Start here: Getting Started → Installation.
- Use
uv run et make testto create a template test module (currently a WIP in #950). - Where to put the test? See how tests are organized in Writing Tests → Adding a New Test.
- State vs Blockchain test? If in doubt try to write a state test, see Writing Tests → Types of Tests → Deciding on a Test Type.
- Tests are filled via EELS by default (no further installation necessary; it's a Python dependency and got installed in Step 0), see Filling → Getting Started.
- Parametrizing test functions is encouraged, see Writing Tests → Writing a New Test → Parametrizing Tests.
- The Opcode mini-lang defined in this repository is the preferred way to define bytecode in tests (if feasible). For a very simple example, see
test_chainid()- there are other examples in the tests.
Feel free to reach out for help or guidance in the Ethereum R&D discord or via DM; see Getting Help in the docs.
### Test Cases
- [ ] Add test cases for any single opcode (porting an entire test case from ethereum/tests can be overwhelming; instead add your own test cases for a single opcode from https://www.evm.codes/).
- [x] [src/GeneralStateTestsFiller/VMTests/vmTests](https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/VMTests/vmTests) - swapFiller.yml - WIP: ethereum/execution-spec-tests#1012
- [x] [src/GeneralStateTestsFiller/VMTests/vmTests](https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/VMTests/vmTests) - dupFiller.yml.
- [x] [src/GeneralStateTestsFiller/VMTests/vmTests](https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/VMTests/vmTests) - pushFiller.yml - WIP: ethereum/execution-spec-tests#974
- [ ] [src/GeneralStateTestsFiller/VMTests/vmTests](https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/VMTests/vmTests) (Note: DUP already ported).
- [ ] [src/GeneralStateTestsFiller/stLogTests/](https://github.com/ethereum/tests/tree/4f65a0a7cbecf4442415c226c65e089acaaf6a8b/src/GeneralStateTestsFiller/stLogTests).
- [ ] [src/GeneralStateTestsFiller/stRandom](https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/stRandom) and [src/GeneralStateTestsFiller/stRandom2](https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/stRandom2) (see #1016 for details)
- [ ] [src/GeneralStateTestsFiller/stZeroKnowledge](https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/stZeroKnowledge) and [src/GeneralStateTestsFiller/stZeroKnowledge2](https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/stZeroKnowledge2) (see #1016 for details)
- [ ] [src/GeneralStateTestsFiller/stPreCompiledContracts1](https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/stPreCompiledContracts1) and [src/GeneralStateTestsFiller/stPreCompiledContracts2](https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/stPreCompiledContracts2) (see #1016 for details)
- [ ] [src/GeneralStateTestsFiller/stQuadraticComplexityTest](https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/stQuadraticComplexityTest) (see #1016 for details)
- [ ] [src/GeneralStateTestsFiller/stSStoreTest](https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/stSStoreTest) (see #1016 for details)
- [ ] [src/GeneralStateTestsFiller/stTimeConsuming](https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/stTimeConsuming) (see #1016 for details)
- [ ] [src/GeneralStateTestsFiller/stWalletTest](https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/stWalletTest) (see #1016 for details)
- [ ] [src/GeneralStateTestsFiller/stCallCodes](https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/stCallCodes), [src/GeneralStateTestsFiller/stCallDelegateCodesCallCodeHomestead](https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/stCallDelegateCodesCallCodeHomestead) and [src/GeneralStateTestsFiller/stCallDelegateCodesHomestead](https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/stCallDelegateCodesHomestead) (see #1016 for details)
- [ ] [src/GeneralStateTestsFiller/stEIP150singleCodeGasPrices](https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/stEIP150singleCodeGasPrices) (see #1016 for details)
Some Examples of Ported Tests
I still remain quite skeptical about automated port. I would rather focus on rewriting the tests. there are not so many if you take combinations and vector tests (precompile inputs) and random tests out. (which are easy to do in python too) then remaining are either simple cases or heavy cases.
~~I'll take porting swapFiller.yml if nobody's working on it yet.~~ NM, Looking for a better option :grin:
OK, how about the 3 calldata tests - copy, load, and size? @danceratopz
Please can I take on swapFiller.yml? @danceratopz
Yes, @bomanaps! Please make an issue when you get started and I can link it in the tracker here to avoid anyone else working on it :)
Hi @winsvega Sorry I haven't written in so long, but I got sick and then Christmas..As I see from docs the best idea will be to start with tag: 'good first issue'. In this case can I take for example: https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/stEIP150singleCodeGasPrices or maybe rc/GeneralStateTestsFiller/stRandom and src/GeneralStateTestsFiller/stRandom2? Or maybe there is something more important ? ps. Happy New Year ;)
That seems to only check the gas price of individual opcodes. That should be already covered in pyspec.
Take a look at https://github.com/ethereum/execution-specs/issues/1505
That seems to only check the gas price of individual opcodes. That should be already covered in pyspec.
Take a look at ethereum/execution-specs#1505
So maybe this one stWalletTest ?
@danceratopz @winsvega If the ~Zero Knowledge and/or~ Precompiled Contracts are available, I'll dig into those.
Edit: Focusing solely on precompiled contracts. ZK tests can be worked by someone else or I can circle back once I've nailed things down.
That seems to only check the gas price of individual opcodes. That should be already covered in pyspec.
Take a look at ethereum/execution-specs#1505
So maybe this one stWalletTest ?
stWallet test is not an evm test. This is a legacy test for a smart contract, I just kept it as a legacy, and it probably doesn't cover anything evm related.
hm so in this case @winsvega can you suggest which one I can take? maybe it will be easier to choose correct one ;)
- [ ] stCreateTest/CREATE_ContractSuicideDuringInit_ThenStoreThenReturnFiller.json, stCreateTest/CREATE_ContractSuicideDuringInit_WithValueFiller.json, stCreateTest/CREATE_ContractSuicideDuringInit_WithValueToItselfFiller.json, stCreateTest/CREATE_ContractSuicideDuringInitFiller.json https://github.com/ethereum/execution-spec-tests/pull/1871
- [ ] stCreateTest/CREATE_AcreateB_BSuicide_BStoreFiller.json https://github.com/ethereum/execution-spec-tests/pull/1867
I think we should close this now - it is not currently planned to port the remaining yml,json fillers. These test cases are now filled by execution-spec-tests and included in EEST releases:
- https://github.com/ethereum/execution-spec-tests/tree/10b02048237f917c8ae33b827ba92fca39d48c4a/tests/static