Ryan Levick

@Mark-Simulacrum what's the latest on this?

Obviously the best thing to do here is to find the source of the noise, but should we consider not running this benchmark in the meantime (at least full builds)?...

Eventually, this might happen if the benchmark consistently bounces back and forth. However, it does take time for that noise to distribute through the historical data. I'm not sure when...

> I would have expected the average to be used, rather than the minimum, personally. But either way, I also wonder if the benefits of multiple collections outweigh the costs....

> without the collection table there's no way to get the statistics for all the metrics we collect from a specific run This is correct though there is not code...

Thanks for the feedback! Overall, these definitely seems like a good proposal. I'm just not sure how difficult it would be to implement. Some thoughts and questions: I suppose every...

This should probably build on top of [this doc](https://github.com/rust-lang/rustc-perf/blob/master/docs/comparison-analysis.md) which is a bit too "in the weeds" for your casual viewer of perf results (the doc was created with maintainers...

@Mark-Simulacrum perhaps we could remove the `ON CONFLICT DO NOTHING` to see in a conflict is occurring (which would then cause a panic).

> Do we want to "double up" statistics I'd like to see this land before we get too deep into discussing strategies of when to run the Windows tests. I...

We may want to consider having the [windows](https://github.com/microsoft/windows-rs) crate in addition to winapi. It's also a large code base, but it's sufficiently different in design that I think having it...