qdk icon indicating copy to clipboard operation
qdk copied to clipboard

Remove benchmarking comment bot; Run benches weekly and save results as an artifact

Open sezna opened this issue 1 year ago • 1 comments

The benchmark comments have generally been pretty noisy and I think unpopular, especially when iterating on one open PR.

This PR removes the current benchmark commenting functionality and instead runs a once-weekly (Monday morning at midnight) benchmark on main and saves the results as JSON. These results can then be easily interpreted by a separate tool that I'm putting together. This will help us catch any performance regressions or trends, but in a once-weekly retrospective report of all merges into main, instead of every single commit.

The key thing here is that after this PR goes in, we will have a mechanism to retrospect over performance data without having it commented on every commit of every PR.

sezna avatar Aug 21 '24 20:08 sezna

I agree with removing the current benchmarks from the pipeline, and having a trend of noisy benchmarks over time is more useful than having no data.

orpuente-MS avatar Aug 21 '24 22:08 orpuente-MS