ethereumjs-monorepo icon indicating copy to clipboard operation
ethereumjs-monorepo copied to clipboard

EVM: bump opcode coverage + add performance tester

Open jochem-brouwer opened this issue 1 year ago • 4 comments

The goal of this PR is to:

  • Add basic test coverage of most opcodes
  • Use the code ran in these tests to generate a profile report of the EVM

The goal of this PR is to create both simple opcode tests (compare stack/memory/storage output to expected code), with transparant bytecode generation. Some (all?) of these tests are also used in the profiler report, which tries edge cases (adding maximal numbers together, exponentation to the max, etc.) and reports MGas/S output of those. This can be used to spot slow/fast variants of certain opcodes, or identify which opcodes need some optimization.

Sample profile report output:

image

To run the tests, run npm run profiler:report. You need to have tsx installed for this, so npm i -g tsx. Note: the default gas limit for each test is 30M, and currently all opcodes keep looping until this gas limit is reached (this is slow). There is a progress report though so one can track how much time it should take to finish.

WIP

jochem-brouwer avatar Dec 15 '23 02:12 jochem-brouwer

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (3981bca) 88.99% compared to head (9d7b806) 89.64%.

Additional details and impacted files

Impacted file tree graph

Flag Coverage Δ
block ?
blockchain ?
client 84.63% <ø> (?)
common ?
devp2p ?
ethash ?
evm ?
genesis 99.98% <ø> (ø)
statemanager ?
trie ?
util ?
vm ?
wallet 91.00% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

codecov[bot] avatar Dec 15 '23 02:12 codecov[bot]

This looks great but we should merge #3198 before merging this one. It's not obvious to me but are these opcode tests also run by CI?

acolytec3 avatar Dec 18 '23 19:12 acolytec3

@jochem-brouwer What is the status of this PR?

holgerd77 avatar Feb 14 '24 12:02 holgerd77

I intend to finish this one this month. The goal is to write basic tests for all opcodes and provide a test framework.

jochem-brouwer avatar Feb 14 '24 12:02 jochem-brouwer