Setup CI pipeline for benchmarking performance

[jzelinskie]

This requires a few steps:

• Converting our existing ad-hoc process for performance testing into a GitHub Action
• Connecting GitHub Actions to dedicated hardware that can measure performance without noisy-neighbor issues
• Formalizing the output of performance tests
• Determining how and when these tests run
• Determining how to produce action items from tests that are ran

[jzelinskie]

We should setup a dedicated worker for benchmarks or explore using a service like bencher to get reliable benchmarks built into our CI workflow.

[kirbyquerby]

Hi, I work on bencher! If you wanted to try it out (it's free!), it should just take a few clicks to install from the GitHub marketplace We've got some more-detailed installation instructions (with pictures!) as well.

Let me know if I can help with anything :)

[jzelinskie]

@kirbyquerby I tried enabling it, but we use docker for integration testing in our benchmarks -- this is because we benchmark against various databases.

I see in the bencher configuration you can depend on services based on docker, but can we actually just get access to a docker socket?

[jzelinskie]

@kirbyquerby @odeke-em Are there any rate limits on bencher? I've been playing with things over at https://github.com/jzelinskie/benchpress and half of my pushes don't run. I was pulling my hair out thinking I had an invalid config, but it's not starting builds even without a config.

I'm experimenting with using docker-in-docker so that our test suite can run unmodified.

[odeke-em]

Hey @jzelinskie, Bencher runs only when there is a code change in .go files or when there is a configuration update. The reason for that is that in a PR, lots of diverse changes can happen but shouldn't be wasting precious machine time and CPU re-benchmarking. We shall document this on https://bencher.orijtech.com/

[jzelinskie]

@odeke-em Thanks for the response -- that makes perfect sense.

Do you have any recommendations on how we could get access to the Docker daemon? Our tests use the Docker API to spin up/down images it needs for end-to-end tests. I've been playing with docker-in-docker, but that hasn't seemed to work.

[kirbyquerby]

I'm hesitant to provide direct access to the docker daemon because we run benchmarks on bare metal. This makes it tricky to ensure the state of the machine is reset after a benchmark is run. The current setup provides a intentionally trimmed-down set of docker features to try and minimize the attack surface -- I'm not well-versed enough to understand all the potential pitfalls of allowing arbitrary access to all of docker.

docker-in-docker seems promising, though -- I'll take a look at what it would take to make it work with bencher.