rfcs
rfcs copied to clipboard
NixOS
[RFC 0180] Remove broken packages or unmaintained leaf packages
There are some packages with no maintainer yet don't exactly "become outdated" (an example is the intel-microcode package). How are we going to handle those?
There are some packages with no maintainer yet don't exactly "become outdated" (an example is the intel-microcode package). How are we going to handle those?
I am sure that someone wants to become a maintainer of a package like this, once we actually become serious about removing it, because their hardware won't function without it. However we leave the decision what to do in this case up to the person merging the pull request i.e. waiting for a reasonable amount of time depending on the importance package.
There are some packages with no maintainer yet don't exactly "become outdated" (an example is the intel-microcode package). How are we going to handle those?
There are some of those packages that are very critical. Ideally they should be kept by a team.
There are some packages with no maintainer yet don't exactly "become outdated" (an example is the intel-microcode package). How are we going to handle those?
There are some of those packages that are very critical. Ideally they should be kept by a team.
I think having this RFC in place, will make actually possible to find instances like this and hopefully improve the maintenance of these packages. I could see the security team for example maintaining microcode package.
I could see the security team for example maintaining microcode package.
Or the nixos-hardware team (when they become a NixOS team :D )
I hope the problem infrastructure or something like it can be implemented first, so we can add warnings to end users like "foo is deprecated and is subject to removal because ...", and guide them to give feedback. If they give reasonable reasons, we will cancel or postpone the removal.
We can divide the deprecation of a package into three stages:
- Throws a warning to end users but still builds on hydra
- Make a mark similar to insecure. If the user needs to use it, it needs to be manually allowed and it will not be built on hydra.
- Completely remove related expressions.
Each of these stages can last three to six months, giving everyone plenty of time to react and plenty of reason to remove it.
Additionally going back to maintainerless packages, I think itd be best to do a broad sweep and find which packages are still orphans and find them a team, especially for critical packages like the microcode package. Once this is done anything left behind would be non essential stuff, which can follow the removal procedure
Each of these stages can last three to six months, giving everyone plenty of time to react
Wouldn't anything shorter than six months per stage not be enough time for users of stable channels to notice before the removal process may move on to the next stage?
About orphaning, we also need to deal with the silent retirement. There are a truckload of maintainers that did not maintain their codes from a long span of time.
I suggested to do a soft deletion of silent retired maintainers at https://github.com/NixOS/nixpkgs/pull/310759.
Regardless the above:
We have the infamous Zero Hydra Failures event. We can do something similar around the months 3 and 9 in order to mark and sweep unmaintained packages.
I hope the problem infrastructure or something like it can be implemented first, so we can add warnings to end users like "foo is deprecated and is subject to removal because ...", and guide them to give feedback. If they give reasonable reasons, we will cancel or postpone the removal.
We can divide the deprecation of a package into three stages:
1. Throws a warning to end users but still builds on hydra 2. Make a mark similar to insecure. If the user needs to use it, it needs to be manually allowed and it will not be built on hydra. 3. Completely remove related expressions.Each of these stages can last three to six months, giving everyone plenty of time to react and plenty of reason to remove it.
I don't see really enough benefit in step 2, it just creates more work for us. Removing packages should not be more work than adding packages.
It’s really easy to revert a removal in Git. If someone steps up to maintain a package that was removed for being unmaintained, that’s no obstacle. I think having a warning cycle is good because it means people can get involved before the point where they get annoyed at us breaking their machines, but a warning for one release and then removing it seems perfectly adequate to me. People who aren’t keeping their systems up to date and looking at warnings probably aren’t going to become active maintainers, anyway.
By the way, I understand there are detailed statistics available for popularity of package downloads and I really wish they would be publicly posted, even in heavily bucketed form. We have basically no way of telling what unmaintained (or maintained!) packages are actually used by literally anyone and it means we spend untold cycles fixing packages for the benefit of probably no one at all. It would help avoid removals too: “wow, this package is ubiquitous but nobody is taking care of it, I should step up!”.
@fricklerhandwerk Is it possible to get statistics based on pname from our cache's CDN provider?
I have seen it written elsewhere that there are very granular statistics being tracked on the Fastly end, so I think it would just be a matter of setting a pipeline to clean them up and publish them. Something really simple and completely non‐identifying like total downloads per pname over the last n month(s), bucketed by order of magnitude, and everything <10 downloads omitted, would be a game changer.
Of course people could start gaming this with a DDoS attack or whatever but really, it’d be hard to do worse than the complete blind flying we operate on now.
@Aleksanaa this is possible and we should definitely do that (see latest notes about the cache), but currently only @edef1c is active in that area and I would advise against prioritising working on insights before we get the caching costs down.
I don't see really enough benefit in step 2, it just creates more work for us. Removing packages should not be more work than adding packages.
It does not follow. The scenarios are not 100% symmetric. Before a package exists in the repo, we are 100% sure there is no one using it. Before it being abandoned, we are not so sure no one is using it.
Example:
- https://github.com/NixOS/nixpkgs/pull/326930
- https://github.com/NixOS/nixpkgs/pull/326930#discussion_r1677126273
It’s really easy to revert a removal in Git.
This is not a serious problem. It is just a matter of coding again.
I don’t really follow. If a release cycle passes where everyone who uses the package gets a red warning about how it’s about to be removed and does nothing, we know that anyone using it isn’t reading our notices to them. If they only realize when their configuration completely breaks and still absolutely need the package and aren’t willing to pick it up, they can get it from an old pinned version of Nixpkgs. After all, it will be completely unmaintained and not checked for breakage by CI at that point anyway, so updates aren’t going to do much for it.
I am merely saying that this is mostly independent from underlying source code control. Any package can be (re)inserted at any time, given the current contributing guidelines. And this is mostly independent whether Git is easy or hard to use.
I don't see really enough benefit in step 2, it just creates more work for us.
Is there a programmatic way to check for warnings? if not then the benefit is for users who, for example, monitor the exit code of nixos-upgrade but not the logs.
There’s grep, I guess. We already do deprecation warnings for module stuff, so if you can’t get diagnostics that’s a separate problem that will already bite you.
Is there a programmatic way to check for warnings?
With NIX_ABORT_ON_WARN all warnings are treated as errors. This however does not work in pure flake evaluation.
I built RFC #127 precisely with the purpose to provide a mechanism for orderly removing packages from Nixpkgs, while leaving the exact rules on what to remove open as future work. I appreciate you continuing these efforts, and would be disappointed if they didn't end up making use of our RFC 127 infrastructure (which is implemented but not merged yet, FWIW).
(which is implemented but not merged yet, FWIW)
Anything blocking the merge?
Anything blocking the merge?
It mostly lacks somebody motivated to bring this over the finish line, IIRC the code is complete but somebody should write a couple of tests for it
Some old packages that no longer update should be considered. Maybe we should add a tag to point out it shouldn't be deleted.
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/rfcsc-meeting-2024-08-05/50170/1
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/rfcsc-meeting-2024-08-19/50831/1
RFCSC:
This RFC has not acquired enough shepherds. This typically shows lack of interest from the community. In order to progress a full shepherd team is required. Consider trying to raise interest by posting in Discourse, talking in Matrix or reaching out to people that you know.
If not enough shepherds can be found in the next month we will close this RFC until we can find enough interested participants. The PR can be reopened at any time if more shepherd nominations are made.
See more info on the Nix RFC process here
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/rfcsc-meeting-2024-09-02/51514/1