NixOS
Lazy trees v2
Motivation
This adds a setting lazy-trees that causes flake inputs to be "mounted" as virtual filesystems on top of /nix/store as random "virtual" store paths. Only when the store path is actually used as a dependency of a store derivation do we materialize ("devirtualize") the input by copying it to its content-addressed location in the store.
String contexts determine when devirtualization happens. One wrinkle is that there are cases where we had store paths without proper contexts, in particular when the user does toString <path> (where <path> is a source tree in the Nix store) and passes the result to a derivation. This usage was always broken, since it can result in derivations that lack correct references. But to ensure that we don't change evaluation results, we introduce a new type of context that results in devirtualization but not in store references. We also now print a warning about this.
Supersedes/incorporates #12432 and #12915. It replaces #6530. The big difference with #6530 is that the latter treated each source tree as its own root filesystem, which caused a lot of backward compatibility issues that required ugly hacks. With the new approach, lazy source trees appear under /nix/store, just using a virtual store path that doesn't exist in the real filesystem.
TODO: run the test suite with lazy-trees = true.
Context
Add :+1: to pull requests you find important.
The Nix maintainer team uses a GitHub project board to schedule and track reviews.
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/determinate-nix-3-5-introducing-lazy-trees/64350/7
Running the test suite uncovered a couple of failures: https://github.com/NixOS/nix/actions/runs/15130602489/job/42530714771?pr=13235
@roberth I understand your concerns but I think it would be hard to make improvments if changes are not allowed incrementally. I think having large pull requests lingering also makes impacts negatively other pull requests where we are too afraid to apply large scale refactorings. Would you be fine if the settings would be replaced with a compile-time flag?
How about guard behind an experimental flag and let more people use it first?
How about guard behind an experimental flag and let more people use it first?
It's already a flag not enabled by default, so that should be covered. Commonly we use experimental flags for new features that might change over time. This flag should not introduce any new features so, but potentially bugs, so I don't think it needs to be experimental in this case. The documentation of the flag should potentially mention the current randomness however.
@Mic92 Incremental improvements are allowed, but this is not an improvement.
If any contributor, including Nix team members, wants to make breaking changes to the language, they should discuss that with the team first, and it is unlikely to result in an actual change to the language, as Nix promises reproducible behavior on old expressions.
Large PRs are ok if they're refactors, especially if they're previously agreed upon. However, this PR is not a refactor. It changes the semantics of the Nix language significantly, and for worse.
compile-time flag
What would be the purpose, except enabling a bug? Any kind of flag, compile time or otherwise, has a real cost in terms of maintenance. Also we shouldn't be seducing users into bad workarounds. We've seen how that ends - or, I guess, doesn't end.
I appreciate that you try to find a middle ground @Mic92, but this is not a negotiation. We don't balance out insufficient work with personal authority or upvotes. Good work gets merged.
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/2025-05-21-nix-team-meeting-minutes-228/65204/1
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/changelog-lazy-trees-and-security-improvements/66033/8
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/determinate-nix-3-8-0-changelog-a-faster-nix-flake-check-improved-flake-locks-and-lazy-trees-rolled-out-to-20-of-users/66499/2
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/determinate-nix-3-8-0-changelog-a-faster-nix-flake-check-improved-flake-locks-and-lazy-trees-rolled-out-to-20-of-users/66499/5
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/dropping-upstream-nix-from-determinate-nix-installer/69181/13
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/dropping-upstream-nix-from-determinate-nix-installer/69181/16
Using randomness or even fingerprints for placeholders makes the language non-deterministic and impure.
Some things we can do, in approximate order of increasing cost/benefit:
* copy to the store on demand (and concurrently) * have lazy path values * have rope-structured strings with lazily computed fixed-length substrings to represent lazy hashing * extend the language with weaker interfaces that perform better, e.g. as described in [#10689 (comment)](https://github.com/NixOS/nix/issues/10689#issuecomment-2835688644)With those solutions in mind, I don't think we have to ruin the language semantics with any kind of "entropy", including fingerprints. Path equality and ordering (as observable to users) must remain identical, and we have plenty of options for making it lazier without sacrificing determinism or compatibility.
I assume this is why lazy tree won't get merged anytime in the near future since the approach needs to be revisited, correct?
I assume this is why lazy tree won't get merged anytime in the near future since the approach needs to be revisited, correct?
Correct. The most promising approach in my and @roberth's opinion on the matter is:
have rope-structured strings with lazily computed fixed-length substrings to represent lazy hashing
This needs some work on the internal representation of strings, but is certainly doable and intersects somewhat with the pascal strings rework that has been in the works for Lix and for lesser degree in CppNix. I hope to get around to tackling that at some point.
