rustc-perf icon indicating copy to clipboard operation
rustc-perf copied to clipboard

Consider running new benchmarks on old artifacts

Open rylev opened this issue 3 years ago • 1 comments

When we add new benchmarks, we typically don't run them on old artifacts, but we could (at least to a certain point in history). While this might not bring a ton of insight when adding regression focused benchmarks (e.g., stress tests), it would likely be very useful when adding new "real world" crates.

rylev avatar Oct 02 '21 11:10 rylev

The collector and frontend are both almost setup for this. The collector already supports partial resumption, mostly intended for restarting it without losing data. That should work fine for recollecting commits though.

Generally I think the only thing needed is to alter our sense of "what needs benchmarking" to include artifacts which are missing a benchmark from their results. We'll want to make sure to timebox that -- the collector will currently get stuck if we try to benchmark an artifact it can't download, so avoiding that is critical.

Mark-Simulacrum avatar Oct 02 '21 14:10 Mark-Simulacrum