Nanosoldier.jl icon indicating copy to clipboard operation
Nanosoldier.jl copied to clipboard

Reports we should generate for periodic benchmark jobs

Open jrevels opened this issue 9 years ago • 1 comments

From discussion here:

Does it make sense to run against previous release as well. I'm thinking else, regressions could be introduced bit by bit were each part is small enough to dissappear in the noise.

After we gather enough nightly data, we'll be able to start looking at trends over longer time intervals, which helps with this. My idea was to have @nanosoldier generate a weekly/monthly report using the existing benchmark data, in addition to the daily reports. Then we'd have daily comparisons, weekly comparisons, and monthly comparisons, as well as the data to do more advanced trend analysis later on.

We can simply run the benchmarks for the current stable release once, and reuse that data when doing comparisons. Then we could just generate report_vs_release.md in addition to our current report.md.

jrevels avatar Jun 11 '16 18:06 jrevels

Just a small comment. Does it make sense to put all the daily reports under a common daily folder in BaseBenchmarkReports? I know that using Github as data storage is a temporary solution but at this rate, the BaseBenchmarkReports landing page will soon be filled with folders.

KristofferC avatar Jun 12 '16 18:06 KristofferC