NixOS configuration - testing, debugging, and organization

[olafklingt]

The aim of this PR is to contribute a guide to the topic of NixOS configuration testing debugging and organization. It revises https://nix.dev/tutorials/integration-testing-using-virtual-machines and utilizes additional information from nix wiki and manuals. It might happen that the topic of configuration file organization is split into a separate guide. This initial commit has the function to show that I am working on the topic.

[github-actions[bot]]

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

[olafklingt]

I have read the CLA Document and I hereby sign the CLA

[olafklingt]

recheck

[roberth]

I hope to merge the refactoring and documentation bits of https://github.com/NixOS/nixpkgs/pull/176557 this month. It'd be nice to have links going from the tutorial to the reference manual.

@fricklerhandwerk could you use the "Start a review" button next time? I got a ton of emails for each comment. Not terrible, but also not great. But most of all, keep up the good work :heart:

[domenkozar]

If you rebase on top of master, preview url should be generated as part of the CI.

[olafklingt]

I will take care of this PR tonight. (Rebase and outstanding comments)

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-35/21701/1

[roberth]

Please read up on the changes in writing-nixos-tests.section.md

• Arguments/attributes are now options
• An implicit consequence of those changes is that pkgs.nixosTest isn't quite future proof. Its argument should be a module now, but migrating it is virtually impossible, because the types can not be distinguished: the current argument is a function, and a module can also be a function... We should either add a new function or use the "NixOS lib" method (as linked).

Note that nixos-lib is lib.nixos in the flake. We're not documenting flake stuff yet though (which I disagree with, because now we don't have a single reference, but a lot of cargo culting that will be hard to correct). Figured I'd share the knowledge nonetheless.

[roberth]

Also significant is that you'll be able to link to the options reference.

[roberth]

Also significant is that you'll be able to link to the options reference.

https://github.com/NixOS/nixpkgs/pull/193498 changes the prefix #opt- to #test-opt-, fyi.

[olafklingt]

@roberth your changes should make it into 22.11, right? Than I would say that I target 22.11 too. I don't care about flakes until they are officially introduced.

[roberth]

Correct. I didn't think about which branch you'd target, but indeed these docs will be a 22.11 feature.

I don't care about flakes until they are officially introduced.

That's ok.

[olafklingt]

@roberth why is pkgs.nixosTest not future proof? to me it rather seem to be not backward compatible, the option name needs to be defined.

[roberth]

@roberth why is pkgs.nixosTest not future proof? to me it rather seem to be not backward compatible, the option name needs to be defined.

There's more to it than the name option, which we could give a default value when invoked through nixosTest and be done with it. The problem is with the arguments passed to the test. Currently it behaves like callPackage, but it should let the module system pass module arguments instead.

It's not impossible to keep using and supporting it, but the arguments it passes to the test don't align with the architecture of the test framework anymore. Perhaps future proof isn't the right term. My goal is to simplify interfaces, and we'll need to replace nixosTest in order to simplify its interface and for it not to be "weird". The surprise we can't fix without breaking change would be something like: when I pass my test module to nixosTest, why can't I access the config module argument?

[roberth]

I think we should add a new function with the intent to replace nixosTest, giving the same convenience, but it's non-trivial because I also want to make sure that the handling of pkgs improves. That's why I haven't got around to it yet.

[olafklingt]

@roberth

passing arguments to the test

you mean something like this https://github.com/NixOS/nixpkgs/blob/16c5f5534dff5fe09cf53a1f08d6d94ef5eadb19/nixos/tests/sddm.nix#L25 ?

[roberth]

passing arguments to the test

you mean something like this https://github.com/NixOS/nixpkgs/blob/16c5f5534dff5fe09cf53a1f08d6d94ef5eadb19/nixos/tests/sddm.nix#L25 ?

That's a different function. I was talking about the x in nixosTest x. It used to be a function from pkgs attributes to test attributes, whereas now it would be more natural for x to take module arguments instead ({ config, lib, ... }: , etc).

[olafklingt]

From the perspective of the module convention I understand your concern. But I have don't have a usecase in mind where this would matter. I leave the use of nixosTest as it is for now and change it when the new function arrives.

[olafklingt]

I will close this PR because the three tutorials are in a different degree of completion.

Currently Reviewed: expand on nixos configuration testing #305 Currently Reviewed: nixos configurations on vm #334 A draft: NixOS configuration file organization #335

[fricklerhandwerk]

Thanks a lot for your patient, diligent work. I know first hand how tedious this is, and I'm looking forward to the first part (#334) getting merged. The topics you are working on are highly valuable for developers based on NixOS.

[olafklingt]

Thanks @fricklerhandwerk for your comment. I have the problem that PR #305 is locked for me. If you or @domenkozar can unlock it I would like to merge your suggestions. currently it says: This conversation has been locked and limited to collaborators.