rust-lang
Add endpoint for viewing performance change from one nightly to another
Following a tweet by @nrc, there are now a number of Twitter bots tweeting out information about the latest nightly. While I was writing one myself, I realized that it'd be really cool to also integrate it with perf.rust-lang.org.
A good first step would be the ability to look up the perf change for a given nightly. Something like http://perf.rust-lang.org/compare.html, except where you can provide ?nightly=<date> in the URL to show the diff between that nightly and the previous benchmark result. I initially thought that compare.html could already do this, but it appears that that's not the case (changing the query parameters does not do anything). It'd also be fine to use revision instead of date in the URL, though it would be slightly more painful to work with.
If we wanted to be even fancier, we could also expose a JSON API endpoint that gives summary information about the performance change. That way it could be directly included in these announcements. Though I realize that that'd be a bunch more work.
Following some digging around for such an endpoint, it appears I may have crashed perf.r-l.o :/ I did find http://perf.rust-lang.org/perf/get in the process though, which looks like it is a little bit like one of these endpoints. But it's not clear how to generate the right arguments to it?
The query parameters should work. I thought they did; that's a separate issue -- I'll look into it. /perf/get will take the arguments given here: https://github.com/rust-lang-nursery/rustc-perf/blob/844995579d0c3023f5b7754f3f0d58ba57756132/src/api.rs#L153-L170; let me know if there's anything wrong with that. Don't worry if you crash perf.rlo, it's easy to restart it. Just ping acrichto or myself in IRC (#rust-internals or #rust-infra is fine).
Ouch, that's not a great endpoint. Having to JSON encode every parameter :/ But I guess it's fine as a starting point, thanks!
Yeah, it's possible that we'll soon/eventually restructure this into using non-JSON parameters, but I'm uncertain if/when that'll happen.
I ended up just directly using rustc_perf, which works pretty well: https://github.com/jonhoo/rust-at-sunrise/blob/master/src/main.rs#L164. Won't be quite as easy for non-Rust clients though.
This seems like it might be resolved now?
At least for me, a URL like: https://perf.rust-lang.org/compare.html?start=2019-08-01&end=2019-08-02 seems to work as you would expect?
- (I'm guessing, from a quick skim of the
git blame, that this fix was introduced by commit b628a00d30786ce102959728cd355501e94ea38f )
I think this is somewhat of an issue as the link above does not necessarily guarantee that you're comparing the nightly builds for those dates just the last and first artifacts for the start and end date respectively (thought I'm not 100% sure, and I'd need to investigate).
