rustc-perf
rustc-perf copied to clipboard
Show changes over time to compiler performance in the triage report
The triage report has improved, but its focus is largely on highlighting single changes impacts on the performance of the Rust compiler. We should think more holistically about what we want to get from the triage report.
Specfically we should add more detail about change over time. For instance, we can show change over the last 1 week/over the period in the triage report, 1 month, 6 months, 1 year, etc.
To do this we can:
- Link to comparison pages that show this difference
- Summarize the difference by showing average percent changes per profile and per scenario (over all benchmarks). Finding the right data to expose here will be a challenge.
- Furthermore, we can highlight changes specifically in real-world benchmarks as this might provide clearer insight over stress tests (which are generally there to prevent regressions)
One small step may be to auto-generate a partial summary (to be edited by the triage report author, perhaps) that gives an indication of performance over that week. I often end up writing something sort of hinting at that, but typically without any hard numbers.
I also think it's worthwhile to ask why we care about change over time / what the purpose of these longer term summaries would be. I think some possible goals might be (with slightly different audiences and "purposes" in mind):
- Users continue to be optimistic about rustc compile times, and believe we're investing in them (i.e., selling the slow compiles of today as "they're actually getting much better over time").
- Increasing interest in performance work on the compiler through story telling about investigation & fix work
- Motivating rustc maintainers and contributors to care about compiler performance
IMO, it could be a good idea to either file a separate ticket or just change the purpose of this one to answering this question -- it feels a little like the specific "change over time" is a goal without rationale just yet, and perhaps taking a step back to look at what our goals (as the compiler performance group, for example) are, so that we can then design solutions to fit them. (Ideally with a finer grain than "faster rustc").