[Macro] Automated benchmarking and infrastructure

[rob-maron]

What is this task and why do we need to work on it?

This task is to define and implement an automated benchmarking framework and the corresponding infrastructure to support it. We need these updates to allow for more efficient and detailed benchmarking. The main theme is "automated", but we need some benchmark changes as well.

What work will need to be done to complete this task?

Child tasks:

• [x] Add new stats https://github.com/EspressoSystems/HotShot/issues/2341
  • [x] Define which stats we want EspressoSystems/HotShot#2343
  • [x] Add stat collection EspressoSystems/HotShot#2345
  • [x] Update orchestrator to collect stats EspressoSystems/HotShot#2346
• [ ] Update examples EspressoSystems/HotShot#2348
• [ ] Allow for running multiple runs simultaneously or sequentially EspressoSystems/HotShot#2350
• [ ] Clean up config file EspressoSystems/espresso-sequencer#2569
• [ ] Health check for consensus nodes EspressoSystems/HotShot#2352
• [ ] CI to run benchmarks automatically EspressoSystems/HotShot#2353
  • [x] https://github.com/EspressoSystems/HotShot/issues/2723
  • [x] https://github.com/EspressoSystems/HotShot/issues/2826
  • [x] https://github.com/EspressoSystems/HotShot/issues/2827
  • [x] Store previous benchmark run outputs for fast comparison EspressoSystems/HotShot#2354
• [x] EspressoSystems/HotShot#2719
  • [x] https://github.com/EspressoSystems/HotShot/issues/2857

Are there any other details to include?

No response

What are the acceptance criteria to close this issue?

The benchmarking infrastructure is fully automated. Approval from other consensus team members that it meets the needs met.

Branch work will be merged to (if not the default branch)

No response

[dailinsubjam]

• Priorities
  • Adding new statistics
  • Having things in CI run the benchmarks automatically
  • Updating examples so that we use the builder
  • Other tasks are stretch goals