Run the flake regressions test suite

Open edolstra opened this issue 1 year ago • 3 comments

Motivation

This adds a GitHub action to run a subset of the flake regressions test suite, which is a set of 259 flakes with their expected evaluation results (which is a JSON serialization of the flake outputs, extracted using flake-schemas).

Since the full test suite takes a few hours to run, this only runs the first 25 flakes for now. We may want to have a manually triggered action to run the full test suite.

Context

Priorities and Process

Add :+1: to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

Apr 24 '24 16:04 edolstra

Team discussion:

Idea approved.
Move everything to the NixOS org.
Make sure it's possible to add non-FlakeHub flakes.

Jun 19 '24 19:06 edolstra

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-06-19-nix-team-meeting-minutes-154/47265/1

Jun 19 '24 20:06 nixos-discourse

We've moved the flake-regressions and flake-regressions-data repos to the NixOS org.

Jun 21 '24 13:06 edolstra