rust-lang
rust-timer contradicts itself about whether there were perf changes
Example: https://github.com/rust-lang/rust/pull/90411#issuecomment-955130324
rust-timer says both:
This benchmark run did not return any relevant changes.
and:
While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR led to changes in compiler perf.
(When I opened the comparison page, there were very slight changes, but as rust-timer itself said, nothing very significant.)
Hm, the latter message I think used to be less confident, but this still seems correct to me. The key point is that while there are no relevant changes (i.e., the PR author/review should likely not take any action to resolve regressions etc), there are still changes, and so we shouldn't roll this PR up as it may mask regressions (or, less importantly, improvements) in other PRs.
In general we take perf invocation as a proxy for "someone thought this perf sensitive", and that's usually enough that rolling up is probably at least =iffy, and =never is not that comparatively more expensive.
We've not yet found good wording for the "relevant changes" messaging though, suggestions welcome there -- ideally we'd concisely indicate that there potentially are changes but that you probably shouldn't care.
We've not yet found good wording for the "relevant changes" messaging though, suggestions welcome there -- ideally we'd concisely indicate that there potentially are changes but that you probably shouldn't care.
It'd be less concise, but something along these lines could work:
This benchmark run did not return any relevant changes. While some changes may have occurred, they are insignificant, although they could mask other changes if included in a rollup.
The messages have been quite overhauled since this issue has been created, and in this specific case, the text has been "softened":
While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this
PR may lead to changes in compiler perf.
So I think that this can be closed.
