snabblab
Release cycle to help conserve Hydra resources
For each change on the master branch, all Hydra jobsets that have that branch as their snabblab input will be reevaluated. Could we reduce the total number of evaluations caused by grouping changes to snabblab/snabblab-nixos in monthly releases in sync with snabbco/snabb?
Having dedicated branches and making jobsets refer to the most appropriate one is definitely something we should do.
The resistance for me is point-and-click to update all the jobsets. I wish we had the "declarative" ones that are defined by code in Git. Have never succeeded in setting that up though so maybe better to just roll up sleeves and do the clicking.
Or we could say that master is only for changes that should be pushed to all jobsets and do new work on another branch like next? (Could also make next the default branch on Github for PRs.)
Relatedly: Evaluating the jobsets is not such a big deal. That's just Nix waking up and checking which builds are up-to-date. The expensive part is building which only happens when a change actually impacts the way a test is run. So for example if you push an update to the R report that will not rerun any Snabb tests, but if you touch any dependencies of Snabb then the whole world gets rebuilt.
However I think branches are still the solution because it can be hard to accurately predict how many builds a given update will need to kick off.
See also previous discussion: https://github.com/snabblab/snabblab-nixos/pull/95#issuecomment-294731098