NixOS
[RFC 0185] Redistribute redistributable software
I think a drawback for the plan as-is would be that some of the stuff is actually pretty large so space usage at Hydra/cache goes up.
A missing related piece of work here is to look at the reverse dependencies of unfree packages and to check if they are a good idea to build. For normal packages we have some «it's large but downloading is longer than the build» packages marked as hydraPlatforms=[]. For stuff like TPTP («minimally» unfree license, a build compiling a few executables, some gigabytes of passive data) the position on whether storage:build-time is worth is currently «who cares it's technically unfree».
Good points! I just added them to the RFC. Also I'll incidentally say that TPTP seems to already be marked as hydraPlatforms = [], but it's still worth reviewing the changes.
I think that given the skew of unfree stuff towards large binaries, significantly increasing the evaluation time and somewhat increasing the storage growth are separate points to mention. (Probably we need to ask for feedback from the infrastructure team on all that at some point)
significantly increasing the evaluation time
@7c6f434c Why would binary-based packages significantly increase the evaluation time? Nixpkgs requires packages to pass strict evaluation, which means that downloading would never occur during evaluation.
I haven't experiment it, but I guess that packages that requires a long evaluation time typically falls into the categories below:
- have a large number of requisites (direct and indirect dependencies)
- have custom overriding applied to dependent packge sets
- read a lock file to produce a set of vendored packages
- are made up from large auto-generated Nix expressions (usually by those *2nix commands line tool)
none of the above are specific to unfree or binary-based packages.
Update: Some binary-based packages might be built with legacy versions of libraries, which would require custom overriding if such version is uncommon in Nixpkgs. Still, such situation also occurs to large packages like TensorFlow, and small projects with few dependencies wouldn't take too long to evaluate.
Evaluation time will increase because life is hard. Basically, even though glibc does not have non-free dependencies, its evaluations «as if it is allowed to have some non-free deps» and «as if it must be transitively free» have the same result, but are not the exact same evaluation. The same for every package in the closure of the large graphical ISOs (which are also evaluated in the non-free-redistributable allowed jobset)
I think I already listed the two points you're mentioning? The evaluation time issue was listed in the settings from the start; and I just added the built size issue as an unresolved question, as it's currently unclear whether it's negligible or not
Also you seem to be hinting at a doubling of the eval time but I don't think that'd be the case. Hydra would eval a pkgset that'd consist of essentially unfreeAllowedNixpkgs // filterAttrs drvsThatNeedToBeFoss noUnfreeNixpkgs. So we'd be adding eval only by the partial evaluation time of the closure of required-foss derivations; which should be very far from the full eval time, considering relatively few derivations would require no-unfree
We could do the Hydra's eval simply as now but with unfree allowed – and do this check separately in a channel-blocking job (executed on a builder instead of centralized with the eval). We have similar checks already in the tarball job (pkgs/top-level/make-tarball.nix), even though we've reduced them recently.
CI might check this as well, but such regressions seem quite unlikely to me.
Oh right, evaluating all the ISOs is not negligible, but indeed can be pushed to a build
Thank you so much for working on this. Since MongoDB is the biggest offender which causes many people serious trouble day-to-day to build, perhaps we could also consider a phased rollout plan where that is the first thing to be included :smile:
And MongoDB indeed has license which avoids most general concerns, in the sense that the source is available, and both arbitrary patches (as they are derivative, they are presumed same-license-as-MongoDB in Nixpkgs anyway) and running inside a network-isolated sandbox are permitted without restriction.
This is not true for all unfree-redistributable things…
The way I understand (and mean) the current RFC text, all currently unfree redistributable packages would stay out of hydra until marked buildableOnHydra. We could then start with just marking SSPL as buildableOnHydra, but it will be a license/package-specific discussion.
Are there any remaining concerns on the current RFC, that I could address? :)
Thank you for the work on this! I agree with the implementation. Redistributable in general in the license doesn't necessarily mean redistributable on Hydra, having a secondary flag that indicates whether we are comfortable doing so seems prudent.
I would second a phased rollout plan where MongoDB (and maybe CUDA) come first since they have elicited the most complaints.
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/lastmodified-attribute-mismatch-in-input-lastmodified-0/58395/1
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/rfcsc-meeting-2025-01-06/58396/1
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/rfcsc-meeting-2025-01-20/59113/1
Considering there's no self-nomination, I'd like to nominate:
- @NixOS/steering because this is mostly a political choice
- @NixOS/infra-build because you are one of the main stakeholders
Hopefully enough people will accept to make a shepherd team!
From Steering Committee: We are generally in favor of this. We have unanimous approval of the policy aspect. There are a few things for which we are exercising some caution (perhaps an abundance of caution);
- The Foundation board has legal responsibility. The imminent appointment of new board members, particularly the Director of Legal Affairs, should be involved.
- Impacts to infrastructure. Thanks infra!
RFCSC:
Please note that we need specific people to act as shepherds and move this RFC though the process. So please nominate specific individuals (or yourself!) to act as shepherds. Thanks.
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/rfcsc-meeting-2025-02-03/59763/1
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/rfcsc-meeting-2025-02-17/60444/1
I'd like to nominate, hoping they will agree: (I only picked 2 people from each team to avoid a mass-ping, using a totally fair die biased by "who already commented here?")
- From the Nix Steering Committee, for the political decision:
- @tomberek
- @roberth
- From the infra team, for the impact on hydra:
- @Mic92
- @vcunat
- From the foundation board, for the legal impact:
- @zimbatm
- @domenkozar
Independent from the selection of people, which might be still fine. Only Ron is still active in the foundation board: https://nixos.org/community/teams/foundation-board/
New people have already applied but we are still waiting for the SC to confirm.
From the foundation board, for the legal impact:
* @zimbatm * @domenkozar
These aren't active board members. Can nominate when the new board is announced.
EDIT: +1 on me helping out with shepherding
Honestly, it looks like there's consensus for this idea, I don't see anybody opposed to it. Mic92 also already confirmed that it's fine with infra. The only thing would be a legal double-check, but you could even include that into the RFC as a condition.
Considering that, I don't think there's even a need to go through the RFC process, but might as well do it since we already started.
So I'm also nominating myself as shepherd, with the intention to just approve it without further discussion.
Legal: my understanding of the "redistributable" attribute is that it can be used (only) if this is legally OK, by definition. So I'd rather say that perhaps docs about that could be improved and all users of the attribute could be rechecked.
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/rfcsc-meeting-2025-03-03/61097/1
RFCSC:
We have 2 shepherds confirmed with @roberth and @infinisil . We need a last shepherd to move on. Please don't hesitate nominating yourself or others here.
