rustc-perf
rustc-perf copied to clipboard
Consider running new benchmarks on old artifacts
When we add new benchmarks, we typically don't run them on old artifacts, but we could (at least to a certain point in history). While this might not bring a ton of insight when adding regression focused benchmarks (e.g., stress tests), it would likely be very useful when adding new "real world" crates.
The collector and frontend are both almost setup for this. The collector already supports partial resumption, mostly intended for restarting it without losing data. That should work fine for recollecting commits though.
Generally I think the only thing needed is to alter our sense of "what needs benchmarking" to include artifacts which are missing a benchmark from their results. We'll want to make sure to timebox that -- the collector will currently get stuck if we try to benchmark an artifact it can't download, so avoiding that is critical.