lighthouse icon indicating copy to clipboard operation
lighthouse copied to clipboard

Add assertoor tests to CI

Open eserilev opened this issue 1 year ago • 4 comments
trafficstars

Add assertoor tests to CI

This PR adds a new step to the local_testnet CI. This new step spins up a kurtosis testnet (defined in assertoor_network_params.yml) and runs assertoor tests as a github action via kurtosis-assertoor-github-action

Issues w/ assertoor github action

Note that the repository ppa:rmescandon/yq is no longer maintained and does not contain the most up to date version of yq

https://github.com/mikefarah/yq/issues/1308

Installation recommendations for yq can be found here: https://github.com/mikefarah/yq?tab=readme-ov-file#install

I've updated the local_testnet.yml script to install yq via snap

Choosing which tests to run

The list of assertoor tests can be found here: https://github.com/ethpandaops/assertoor-test/tree/master/assertoor-tests

Theres a few tests that are broken at the momemnt (mev-block-proposal as an example) and a few that take 48 hours to complete. I've picked the most relevant tests that finish running at a reasonable time.

Theres some electra tests that we may want to add in the future

eserilev avatar Jul 26 '24 17:07 eserilev

I don't think we need opcodes tests, those sound like they'd be EVM specific

big-calldata-tx-test might be useful for us as it effects blocks size

stability-check and synchronized-check also sound interesting but not sure what they are

realbigsean avatar Jul 31 '24 22:07 realbigsean

the mev-block-proposal-check test fails for both lighthouse and prysm across multiple EL combos. not sure whats up with that, I've left that test out for now

eserilev avatar Oct 16 '24 21:10 eserilev

Can you upload the logs to the CI run artifacts? To debug failures

dapplion avatar Oct 18 '24 14:10 dapplion

logs should now be uploaded with a 3 day retention window

eserilev avatar Oct 18 '24 17:10 eserilev