RFE: Evaluate NixOS tests on each PR

[danbst]

This weekend was tough, we've broken nixos tests eval several times (thanks @delroth for debugging those).

https://github.com/NixOS/nixpkgs/pull/55061#issuecomment-460047536 https://github.com/NixOS/nixpkgs/pull/55106 https://github.com/NixOS/nixpkgs/pull/55157

Something like

$ nix-instantiate ./. -A nixosTests >/dev/null

would have caught these issues. However the problem with above command is that:

• some tests use unfree software (vm-test-run-bcachefs). I recall that hydra doesn't build unfree packages, does it ever build tests, which use unfree packages? Should ofborg behave the same?
• some tests use insecure software:

error: Package ‘python2.7-Django-1.8.19’ in /home/danbst/dev/nixpkgs/pkgs/development/python-modules/django/1_8.nix:25 is marked as insecure, refusing to evaluate.

• tests form a tree, so some recursive walk is required. I came up with

let
# all-packages.nix

  nixosTestsRecursive =
    let f = n: v: builtins.trace n lib.flatten (
          if builtins.typeOf v == "set" && (v.type or "" != "derivation")
            then lib.attrValues (lib.mapAttrs f v)
            else [ v ]
        );
  in lib.foldl' (acc: x: acc // { ${x.name} = x; }) {} (f "" nixosTests);

• instantiate is slow as heck. Maybe https://github.com/NixOS/nix/issues/2652 can help?

[davidak]

Is this a duplicate of https://github.com/NixOS/ofborg/issues/368?

[FRidh]

This is about the evaluation of NixOS tests, not actually building (that is, running) them. Building them is nice, but heavy. Evaluating them is necessary.

[roberth]

Instantiating all NixOS tests for each push isn't feasible, but we could do something a bit more clever, such as

• adding a meta.tests option to NixOS (similar to meta.maintainers or meta.doc)
• evaluating this option in an empty configuration. This will give us a mapping from file names to tests. Estimate: 10 seconds
• evaluate the tests that belong to the modified files.

We can put a limit on the number of changed module files and the number of tests.