sentry-dotnet
sentry-dotnet copied to clipboard
getsentry
Investigate running benchmark in CI and post results
To investigate: https://github.com/marketplace/actions/continuous-benchmark-net
Looks really shaky. It compares benchmarks against previous (cached) results instead of running another benchmark on HEAD/parent commit. The action expects 1 output file, but Benchmark.NET produces several (one per each benchmark). The PR/fork does not suggest a way to deal with this, the included sample only has one benchmark.
I also looked for other ready solutions to help us with this, but it seems that there's nothing good for .NET. I think we're better off making our own action.
It would work by doing the following:
Job 1:
- Checkout repo at current commit
- Run benchmarks with JSON formatter
- Upload benchmark results as an artifact
Job 2 (parallel with 1):
- Checkout repo at base commit
- Run benchmarks with JSON formatter
- Upload benchmark results as an artifact
Job 3 (after 1 and 2):
- Retrieve artifacts from Job 1
- Retrieve artifacts from Job 2
- Parse the JSON results, compare benchmark performance
- If the mean goes up by 50% (or whatever), report
::error:
The heavy lifting in Job 3 can of course be done in Powershell to avoid having to create a separate action repository, but I think it will be easier to write in Javascript. One way or another, this will probably take a few days, so I wanted to confirm if it's something we want to commit to.
It compares benchmarks against previous (cached) results instead of running another benchmark on HEAD/parent commit.
Sounds like a good idea to me though instead of having to rerun it on each PR, just relying on the previous results?
I think we're better off making our own action.
It might be the way to go but it's possibly something we'll kick further down the road as it'll involve non trivial effort on our end.
Sounds like a good idea to me though instead of having to rerun it on each PR, just relying on the previous results?
It may be a good idea in theory but caching is very unreliable and you also don't get the option to have separate caches per branch. So if someone submits a PR it might cache those results and then compare incorrectly.
I wasn't necessarily thinking of GItHub cache. We might rely on another service such as GCS