[RFC 0137] Nix language versioning

[fricklerhandwerk]

Introduce a convention to determine which version of the Nix language grammar to use for parsing and evaluating Nix expressions. Add parameters to the Nix language evaluator, controlling the behavior of deprecation warnings and errors.

Design goals: avoid breaking existing code, prevent inadvertently breaking reproducibility, minimise maintenance burden for implementors and users.

Regardless, changes to the language, especially breaking changes, should remain a rare exception.

Rendered

Reviewers: Please add inline comments adding your arguments as suggestions, ideally one per comment. Resolving these conversations will keep the amount of foreground information tightly limited to what's still relevant.

Readers: No need to inspect closed comments, the considerations and conclusions there can be assumed to be part of the RFC text.

Please, do not add regular comments concerning contents! Otherwise we will lose track quickly. Reserve regular comments for procedural issues, such as determining shepherds.

This work is sponsored by Antithesis ✨

[piegamesde]

I'd like to have a "Prior art" section on this RFC. I've been told this is not the first time we are discussing this. Also having a look at Rust's "edition" feature for comparison would be helpful.

[7c6f434c]

Can files with different versions be imported during the same evaluation?

[tazjin]

Can I take a step back and ask what this is about?

Which breaking changes are people actually wanting to make to the language (regularly) that would justify having a highly volatile language version scheme?

[thufschmitt]

I think such an annotation isn't really useful without at least an idea of the semantics it should have. And by semantics I mean both the technical ones (what do these versions mean for Nix?) and the social ones (what should be the policy for changing the version number?).

My take on these semantics would be something fairly similar to (what my vague memories and active imagination reconstitute of) the Rust editions system, namely:

1. These numbers should be totally independent of the Nix version number (probably match builtins.languageVersion, although I didn't know it existed until now);
2. Changing them should always be exceptional, maybe batching the incompatible changes to prevent changing the language version too often;
3. Nix should stay backwards compatible, so if a Nix version is able to evaluate the version N of the language, it should also be able to evaluate all the versions m for any m<N. This might mean adding conditionals in the parser, or even keeping different parsers in Nix – which makes it a strong incentive for being strict about 2;
4. Moreover, it should be possible to freely mix files written in different versions of the syntax in the same expressions.

This makes it for a very restrictive semantics, balancing the two ideas that:

• Nix should stay backwards-compatible and an expression that is valid Nix today will still be valid Nix in ten years;
• We also want the language to evolve, if only to remove the cruft and all the in-retrospect-not-really-happy bits that it has accumulated.

[tazjin]

Nix should stay backwards compatible, so if a Nix version is able to evaluate the version N of the language, it should also be able to evaluate all the versions m for any m<N.

This also raises the question if it would apply retroactively, of course. For example, the flakes branch has introduced breaking changes in fetchers that made old Nix code using builtins.fetchGit not evaluate in newer versions. Should that retroactively be declared a lang-version bump, and should it be rewritten in a compatible way? Or is the breakage a bug that should be fixed? Or are interfaces of builtins not part of the language at all?

[fricklerhandwerk]

Thanks everyone for the feedback on this draft! Once I find time, I will sort through the comments and synthesise the suggestions to reflect the state of the discussion in the document.

[Ericson2314]

So one thing we could use this for is switching the type of string context drvPath uses. (That is, from NixStringContextElem::DrvDeep to NixStringContextElem::Built to use the implementation's terminology). This would mean we wouldn't need builtins.unsafeDiscardOutputDependency and instead would (more safely) opt in to depending on the build closure of a derivation.

I should create an issue for this problem and then update this comment to refer to it.

[sternenseemann]

So one thing we could use this for is switching the type of string context drvPath uses.

I like this change, but I don't see why we couldn't do this via langVersion.

[ghost]

So one thing we could use this for is switching the type of string context drvPath uses.

I like this change, but I don't see why we couldn't do this via langVersion.

Or just derivation { __drvPathAlgorithm = "deep"; }.

[sternenseemann]

For additional context, I have reconstructed a changelog of Nix (language) versions, at least insofar as they are reflected in builtins.langVersion. There have been other, sometimes breaking changes to the language that have not received a bump in language version (e.g. the recent fetchGit changes).

[fricklerhandwerk]

Once again, many thanks to everyone for your input. We have picked up work on this draft with @infinisil and @yorickvP.

@yorickvP had a closely related RFC draft in the drawer, and as a first step, we merged his work and all of the comments and suggestions made here into a scratchpad in order to take them into account. I marked all comments as resolved even if it's not reflected in the PR, but nothing will be lost.

Feel free to take a look even if there's nothing interesting to see yet, but please be kind and don't mess with the document. We will meet regularly over the next few weeks to sort out all the arguments, and will eventually update this PR when it's ready for taking comments and suggestions again.

[Ericson2314]

Here are examples of things that would be good to do if we had versioning. (I intend to link such things here as I find them.):

• https://github.com/NixOS/nix/pull/2911 / https://github.com/NixOS/nix/issues/3759
• https://github.com/NixOS/nix/pull/3480

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-45/26397/1

[mightyiam]

Just jumping in to say that Rust's editions system seem to be going great for it.

[piegamesde]

@mightyiam Rust has a totally different approach towards project structuring. Since in Nix you don't have the concept of a "project folder" but instead just a bunch of files randomly importing each other, we are forced to make this scoped to individual Nix files instead.

It is worth a discussion whether we want to move towards a design that is more "project-based" and less "file-based" as it would give us things like language versioning almost for free, but I think that's not the plan discussed in this RFC. (There has been some discussion about making this "per flake" though, which goes into exactly that direction.)

[ghost]

Just jumping in to say that Rust's editions system seem to be going great for it.

I think that depends on whether it is "great" to have only one implementation of a language.

[fricklerhandwerk]

Thanks everyone for your patience. Returning from the Nix lairs, we're proud to present a new version of the proposal in which we tried to answer all questions and incorporate all suggestions. Based on your input it seemed most sensible to change course about how we encode version information. Please give us the benefit of the doubt and work through all relevant arguments and alternatives before commenting.

Please add your comments inline, even if they are about the premise or general direction, to keep the top-level discussion focused.

I will add new considerations and alternatives as they arise. Ideally you'd express them as code suggestions, which would save me a lot of time.

[fricklerhandwerk]

The proposal is now mature enough to call for shepherds! Of course, reviews and comments are still appreciated.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfc-137-nix-language-versioning-call-for-shepherds/28918/1

[alyssais]

I'd like to nominate @sternenseemann as a shepherd. He has made useful contributions so far, and is working on a Nix implementation.

[sternenseemann]

I'd like to nominate @sternenseemann as a shepherd. He has made useful contributions so far, and is working on a Nix implementation.

I accept. I wonder about two things mainly which rather surround the proposal:

• Are we operationally able to do this? Given that breaking changes have in the past slipped through the cracks, why shouldn't they again? Having a versioned language, but retaining unversioned breaking changes seems like a very bad time indeed.
• Is it worth it? A lot of the prospective changes to the language listed are cleanup jobs with aims of making the language make more sense in the eyes of the newcomer. I personally believe that a) some idiosyncrasies do make sense, even if not immediately, and b) the weirdness that remains can be dealt with differently: We just need to acknowledge it and document the best and less-than-great practices. The countless enthusiastic treewide refactors of nixpkgs prove how receptive our community is to best practices, so I don't think that'd be a bad solution at all if it means avoiding a breaking change.

[gabriel-doriath-dohler]

I would love to help by taking part in the shepherd team.

[piegamesde]

I am a bit dissatisfied with how difficult is currently is to keep track of the discussion. Some things that might help:

• Have a Matrix room
• Try out developing this RFC in a dedicated repository (I'm generally not a fan of making that the default for our process, but in this case it looks like it might be worth a shot)
• Have a synchronous meeting

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-06-22-nix-team-meeting-minutes-65/29643/1

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-06-26-nix-team-meeting-minutes-66/29650/1

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-50/29793/1

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/pre-rfc-visually-differentiate-the-symbol-in-nixpkgs/29929/1

[ghost]

Can we please do something about @nixos-discourse spamming github issues like this? It makes github notifications much less useful.

A lot of these "$MY_TEAM update" threads on discourse are long lists of "we talked about $GITHUB_ISSUE". In at least some cases this borders on an attempt to advertise/recruit-for $MY_TEAM by getting @nixos-discourse to do the spamming on the team-lead's behalf.

[infinisil]

Try out developing this RFC in a dedicated repository (I'm generally not a fan of making that the default for our process, but in this case it looks like it might be worth a shot)

Yes please, #138 is the RFC trying to push this approach! Unfortunately there's no way to predict how much discussion an RFC will need in advance, and now that the discussion already started in a PR, it's kind of too late to switch to a repository without a lot of work to migrate all the discussion points. If you're interested in this, please discuss this in #138 though, shepherds still needed btw :)

Can we please do something about @nixos-discourse spamming github issues like this? It makes github notifications much less useful.

I think the bot is useful in a lot of cases, but I agree it can be a bit spammy, I'd argue it should collapse links into single comments. This also shouldn't be discussed here though, I'd encourage you to open a Discourse thread for this

[fricklerhandwerk]

@amjoseph-nixpkgs I agree, it's spammy. Most of that is on me, but I don't intend to advertise or flood your inbox, just make things more discoverable for people who aren't already involved.

We will move this RFC to a separate repo, following suggestions in #138. Feel free to unsubscribe until further notice. There won't be any content-related activity until end of August.