[Enhancement] Use Nix for automated tests

[lucasew]

Improvement Suggestion

#368 suggested about adding CI tests but didn't actually suggest a mechanism to do this, like a feasible way to implement.

Well, I did some experiments with a few technologies using the NixOS testing framework for integration tests.

It's literally defining declaratively a network of machines and then puppeting them using Python. If this interface is not good enough you can use this interface to launch a script in the context of the VM or launch a systemd unit.

You can do basically anything with basically anything that's available on Nixpkgs. The exceptions would basically be something around manual clicking (stuff must be fully automated), Internet access (not available inside the sandbox) and GPU passthru (not supported inside the sandbox) but I don't think this is the case.

First roadblock would be running KVM based virtual machines on GitHub Actions but it seems that [1] has it covered.

All testing could be done in a code referenced in a flake.nix and the GitHub Actions code would just setup Nix on the environment and build that flake. If the build succeeds then it passed, if not, all logs will be available. If stuff can be done in parallel derivation-wise it will be done.

Nix is very weird when you just find out about it but the ecosystem around it is not any random package manager that just download trusted, or not, binaries somewhere. It's basically a source based package manager, like Gentoo, that builds stuff in a isolated sandbox but has a optimization that downloads trusted builds of derivations from a binary cache (initially, cache.nixos.org).

There are plenty of examples in [2].

BTW while nixing, eventually you guys will learn how bad is to download stuff while building xD.

[1] https://determinate.systems/posts/kvm-on-github-actions [2] https://github.com/NixOS/nixpkgs/tree/master/nixos/tests

Potential Benefits

• No YAML hell to setup tests.
• Lots of ready made primitives.
• Tests using literally VMs can be done in parallel as if it is any other build.
• VMs are not based on ready made images but assembled as build passes, like the VM run itself.
• Defining build passes and tests using an actual programming language.

[dae-prow[bot]]

Thanks for opening this issue!

[sumire88]

Hey @jschwinger233, any thoughts on this one?