lefticus
Possible Regression Problem
So we started seeing our regression results giving funny results. Basically it appeared like decent_ci was always comparing to some old baseline, instead of the current develop. We hadn't changed the yaml files to point the branch to a different ref.
Through some fun investigation, I figured out it was deciding to use an old pull request as the baseline. I think it's because of a combination of two things:
- the develop branch didn't move for a couple weeks after the 8.9.0 release (so it was no longer a potential build, or it was skipped, or whatever)
- the pull request came from a fork where the user made the changes on the develop branch itself
So in this block it looks like it tries to match up the nearest potential build coming from a develop branch. Normally this would be our own develop branch, but in this case it isn't. This should probably be protected to just use the root repo baseline branch, right?
I'm still confirming this, but it's all looking this way.
It also really quietly just doesn't run regressions if it can't find a matching branch. I'm not sure if I feel strongly about that or not. I think it would be nice if it at least threw a warning to the CI dashboard to say "I didn't run regressions!".
It should always use develop if it's not currently testing the develop branch. The checks previous to the ones that you reference start by loading the baseline from https://github.com/NREL/EnergyPlus/blob/develop/.decent_ci.yaml#L6 so I don't personally see a pass through the code where it would not get the develop branch. Unless there's something going on with PR's coming from outside of the NREL/EnergyPlus repo.
Sorry this is coming from my phone. But yes the problem turned out to be that the develop branch on Nrel was out of date and there was a pr coming from an external fork where they made changes on their develop branch. I confirmed it was picking up that other pr as the baseline, so there is definitely some code path that allows it. It hasn't ever happened because we've never had this combination of our develop being out of date so far and the external pr situation. And it might not happen again for a long time. But it was definitely a crazy thing to debug! 🙃🙃
On Sat, May 12, 2018 at 11:06 Jason Turner [email protected] wrote:
It should always use develop if it's not currently testing the develop branch. The checks previous to the ones that you reference start by loading the baseline from https://github.com/NREL/EnergyPlus/blob/develop/.decent_ci.yaml#L6 so I don't personally see a pass through the code where it would not get the develop branch. Unless there's something going on with PR's coming from outside of the NREL/EnergyPlus repo.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/lefticus/decent_ci/issues/17#issuecomment-388565152, or mute the thread https://github.com/notifications/unsubscribe-auth/AAz3oBrfWkHGncyNO_F3DqD-kbV9HrJWks5txwihgaJpZM4T3BRB .
-- Edwin Lee, PhD Engineer, Commercial Buildings Research Staff National Renewable Energy Laboratory