builder icon indicating copy to clipboard operation
builder copied to clipboard

Testing suite for builder

Open bertmiller opened this issue 3 years ago • 1 comments

Rationale

To allow anyone to easily make changes to the builder, measure their impact, and ensure that they are safe.

Implementation

  • Provide standardized tools and data (sample bundles)for testing
  • Provide a framework for testing (e.g. performance metrics)
  • Create guide to deploying and testing the builder

One way to approach this is to create a mass conflicting bundles at random and test merging against those. But thinking it through we might want a standardized dataset that allows for more precise measurement of performance (e.g. variance in the randomized generation of bundles doesn't affect performance)

bertmiller avatar Nov 23 '22 11:11 bertmiller

I was thinking we need these things:

  • provide a guide to setting up the builder for local development
  • provide a test script to run
  • provide testing data (bundles and txs)

the output should be something like:

  • hey, the code all works as intended and these X number of checks were met!
  • XX bundles were merged
  • XX bundles were simulated
  • in XX time XX block templates were generated
  • the most profitable block was XY ETH with XX bundles merged alongside XX transactions

ideally this is somehow standardized with the same inputs so that you can directly compare 2 versions of the code and say "hey, this version performed better than the other, which performs better than the main branch"

bertmiller avatar Nov 23 '22 11:11 bertmiller