Tests for flake modules

[terlar]

Has there been any work/discussion regarding tests for flake modules? It would be nice to have a community defined flow for how to create tests for flake modules. Similar like we have tests for nixos modules and home-manager modules.

I think a testing framework or similar for flake-parts could potentially increase the best practices.

Things that could be nice to test:

• Validate that the generated flake is correct/evaluates.
• Test generated flake attributes to see if your output is what you expected.
• Inspect/evaluate attributes to see that what they produced were the correct thing.

What do you think?

[srid]

I recently did some work on haskell-flake to go from bash to flake checks for its testing story, which led me to wonder about a more general framework for all flake-parts modules.

@roberth Should #79 be closed as duplicate of this? Incidentally, I am unable to comment on #79 because Shane Sveller has me blocked on GitHub.

[terlar]

For another module project I went with eval tests, not sure if that is best practice though, I wonder if something similar could be used here?:

{
  pkgs,
  lib,
  ...
}: let
  evalSettings = modules: let
    eval = lib.evalModules {
      specialArgs = {inherit pkgs;};
      modules = [./.] ++ (lib.toList modules);
    };
  in
    eval.config.settings;

  defaults.rules = [
    {
      changes = {
        compare_to = "refs/heads/main";
        paths = [];
      };
      "if" = "$CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH";
    }
    {
      changes = {
        paths = [];
      };
      "if" = "$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH";
    }
  ];
in
  lib.pipe {
    testJobDefaults = {
      expr = evalSettings {
        jobs.job-a = {
          command = "job-a-cmd";
        };
      };
      expected = {
        job-a = {
          script = "nix run .#job-a-cmd ";
          inherit (defaults) rules;
        };
      };
    };

    testJobCommandArgs = {
      expr = evalSettings {
        jobs.myjob = {
          command = "deploy";
          commandArgs = ["with" "args"];
          settings.environment = "test";
        };
      };
      expected = {
        myjob = {
          script = "nix run .#deploy 'with' 'args'";
          environment = "test";
          inherit (defaults) rules;
        };
      };
    };

    testJobCustomSettings = {
      expr = evalSettings {
        jobs.myjob = {
          command = "deploy";
          settings.environment = "test";
        };
      };
      expected = {
        myjob = {
          script = "nix run .#deploy ";
          environment = "test";
          inherit (defaults) rules;
        };
      };
    };
  } [
    lib.runTests
    (lib.generators.toPretty {})
    (res: lib.assertMsg (res == "[ ]") res)
  ]

Then in flake-parts integaration I did something like this:

  config.flake.tests.ci =
    withSystem "x86_64-linux" ({pkgs, ...}:
      pkgs.callPackage ./test.nix {});

Then I executed via nix eval .#tests.ci.

These were purely data tests, so I thought eval was fine.

[srid]

These were purely data tests, so I thought eval was fine.

Yea.

You can go one small step further by supporting nix flake check (for example) by wrapping the result in a pkgs.writeTextFile. One of haskell-flake tests is an eval test (here), but I also create a flake check wrapping the evaluated thing (here) so that the CI runs it in an uniform manner.