Benchmarking for potential regression

[aminnj]

We should make a benchmark suite to make sure we don't regress...

I don't think CI would be very happy to pull in a 2GB file for benchmarking ;) To start off with, it could even be just a local script to run.

[tamasgal]

Yes, regression is also on my todo list (not only for this project) but it's hard to realise a replicable environment. But yes, we can of course start with a simple script which prints a summary and then every contributor can at least run it on master and their PR as an additional input for a PR ;)

[Moelf]

I like https://github.com/JuliaCI/Nanosoldier.jl

I don't know how we can set up this, but I have a list of things I want to benchmark once we know "how".

it could even be just a local script to run.

this is hard because we need to record the "last commit" number and that would be different for each machine we run benchmark on.

[tamasgal]

I don't find any devops guide to set this up. I definitely have the "power" to set up something like this but I think this is only documented internally 😕

I'll have a look tomorrow. In the worst case scenario I'll simply set up a server and we use a self-hosted GitHub-action instance there. I'll limit the number of jobs to 1 so that it's "reproducible".

[tamasgal]

Btw. once we have a reference runner, we can store the benchmark results in the repository and simply compare the values in the CI. That would do it...

[Moelf]

yeah we can have a CI_result branch and mimic nanosoldier (which is to have some hash_directory for PR and commit etc.)

[aminnj]

What about https://github.com/JuliaCI/PkgBenchmark.jl?

[Moelf]

https://github.com/tkf/BenchmarkCI.jl

this is based on PkgBenchmark and is better wrapped in a CI except we now need a place to run CI haha