Add hosted benchmark runners

[epompeii]

These hosted testbeds would make it easy to create and deploy continuous benchmarking workflows that have much less noise than shared CI runners.

Something like AWS Bare Metal for the most extreme cases.

Communication between the runner and the API server should have the runner acting as the client and HTTPS long polling to the API acting as the server, similar to GitHub Actions runners: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners#communication-between-self-hosted-runners-and-github

Using something like a Raspberry Pi 8GB plus the new docker build kit that allows cross compilation of images to ARM. This would require QEMU for the Docker images: https://github.com/docker/setup-qemu-action Usage is per concurrent build.

Testbed M-1 Platform 1-M Runner

To start out the platforms should be created at the server level, ie only admins can CRUD them. Next would likely be organization levels, and eventually maybe project level.

[rukai]

This article has some interesting points https://www.memorysafety.org/blog/rustls-performance/#automated-feedback-on-pull-requests In particular they mention using ovh cloud bare metal which would be much more cost effective than AWS bare metal

https://kobzol.github.io/rust/rustc/2023/08/18/rustc-benchmark-suite.html is also a very good read

[epompeii]

Thanks for sharing those links!

https://www.memorysafety.org/blog/rustls-performance/#automated-feedback-on-pull-requests

I actually worked with @aochagavia to add Bencher to the rustls continuous benchmarking suite (rustls Bencher page) 😃

https://kobzol.github.io/rust/rustc/2023/08/18/rustc-benchmark-suite.html

And yes, a very good read! I have it and a few others on my prior art page if you're interested: https://bencher.dev/docs/reference/prior-art/

[epompeii]

It may be worth adding address space randomization as an option for the bare metal runners: https://github.com/ccurtsinger/stabilizer

[epompeii]

For dedicated runners, this is not as much of an issue, but on shared runners, there needs to be strong sandboxing.

So either light weight VMs: https://github.com/firecracker-microvm/firecracker Or WASM + x86 emulation: https://github.com/copy/v86

Workloads could then share a host sequentially. That is never in parallel.