Motivation

Subflakes are flakes in the same tree, accessed in flake inputs via relative paths (e.g. inputs.foo.url = "path:./subdir"). Previously these didn't work very well because they would be separately copied to the store, which is inefficient and makes references to parent directories tricky or impossible. Furthermore, they had their own NAR hash in the lock file, which is superfluous since the parent is already locked.

Now subflakes are accessed via the accessor of the calling flake. This avoids the unnecessary copy and makes it possible for subflakes to depend on flakes in a parent directory (so long as they're in the same tree).

Lock file nodes for relative flake inputs now have a new parent field:

{
  "locked": {
    "path": "./subdir",
    "type": "path"
  },
  "original": {
    "path": "./subdir",
    "type": "path"
  },
  "parent": [
    "foo",
    "bar"
  ]
}

which denotes that ./subdir is to be interpreted relative to the directory of the bar input of the foo input of the root flake.

Extracted from the lazy-trees branch.

Depends on #10088.

Context

Priorities and Process

Add :+1: to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

:tada: All dependencies have been resolved !

Discussed during the Nix maintainers meeting on 2024-03-04.

• Idea approved
• Document that directly accessing .. in subflakes is likely to break in the future
• Use path literals as the url for subflakes

• Great overall
• Right now, the PR allows import ../. in a subflake. Do we want that?
  • Directly accessing .. is convenient, but breaks the encapsulation property of flakes
    • Relates to https://github.com/NixOS/nix/issues/4414
  • We can however access other parts of the tree with relative inputs (that can go up until the root)
    • inputs.something_else = ../foo
  • From an implementation perspective, hard to forbid without lazy trees
  • Decision:
    • Don't formally forbid direct access to .. since it's hard without lazy trees
    • Once lazy trees lands, make that conditional on a flag
    • Potentially make the default restrictive. Would be a breaking change, but
      • can be mitigated by only making that change effective after a flake update
      • small breakage as getting the old behavior back would only be a one-line change
• Syntax for relative flakes?
  • Currently reuses url = "path:./foo" that already kinda works on master
  • A more natural syntax would be url = ./foo
    • This is actually what most people seem to try first when they want subflakes

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-03-04-nix-team-meeting-minute-130/40830/1

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-11-27-nix-team-meeting-minutes-198/56691/1

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-12-18-nix-team-meeting-minutes-204/57602/1

Does this also mean that subflakes will auto-update? In other words, nix flake update subflake ought to be unnecessary with this right?

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-12-23-nix-team-meeting-minutes-205/57783/1

(using: nix-2.26.0pre20241223_9223d64) An update to the input of a subflake does not get automatically picked up by the root when running a build.

1. Root and child are sync'd.
2. Parent uses path:./subdir to point to child and re-export its outputs.
3. Child has an input to a third, remote-flake.
4. Remote flake has a new commit pushed.
5. Child is updated with nix flake update in the subdir.
6. Run a build in the parent.

This will run the root with the "old" child and the mismatch/update is not detected. At this point the flake:root/subdir/src and flake:subdir/src are locked to different revisions.

7. Run nix flake update in the root.
8. Lock in the root now points to the same sources as the child.

Not sure if this is intended,,, but was a bit surprising. I'd guess that the root flake would detect the mismatch between its lock and that of the child - similar to how a significant change to an input is detected and locks updated when running things like nix build.

Those are really good observations. "Subflake" suggests some sort of directional relationship other than just an input. We should avoid that term in the release notes, and/or explain what's not implemented. The alternative term, relative path input would be ok to use for this feature.

Maybe subflakes should be the name for the pattern or feature where all input declarations are moved to the root of the flake? Or is it merely a subdirectory relationship?

Multi flake repositories might also be a useful name for the use case this pr enables.

I still need to more thoroughly test it, but I'm afraid the unsync'd lockfiles is a problem. When evaluating the root flake with an outdated/updated/modified lockfile, it will still use the flake.nix from the current checkout via path:./subdir, but not the subdir/flake.lock, instead using the inputs from ./flake.lock. It would also be weird to have an old subdir/flake.nix be used - user keeps editing and doesn't see the changes.

Not sure if this is intended. Idea brainstorming:

• nix flake update --recursive
• detect de-sync and provide warning
• do not store the subflake's lock information, just a pointer to the other flake.lock
• document "multi-flake" vs "subflake".

note: I do love the new capability to avoid the "infinite updating lockfile"! Excited to keep playing with it.

At this point the flake:root/subdir/src and flake:subdir/src are locked to different revisions.

I think I'm reading pseudo-syntax that's a little ambiguous, but I would think that the locking of the local sources is fixed by this PR.

The locking of transitive dependencies between flakes in the same repo I think should be solved with #7730; otherwise we'll need a lot of new logic to deal with its symptoms.

@tomberek

An update to the input of a subflake does not get automatically picked up by the root when running a build.

That's right, and probably not ideal, but it's consistent with how lock files currently work in general: lock files are independent, and the only time that we use the lock file of an input is if the referring flake doesn't have a lock for that input yet. So a change to the lock file of an input doesn't propagate to the referring flake.

This should indeed be fixed by having sparse lock files (#7730).

@roberth

"Subflake" suggests some sort of directional relationship other than just an input. We should avoid that term in the release notes, and/or explain what's not implemented.

Yeah we don't use the term "subflake" anywhere (except previously in the PR description, fixed).

Added a release note. IMHO this is ready to be merged.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-01-15-nix-team-meeting-minutes-208/58993/1

Maybe I misunderstood something or made a mistake somewhere, but path:.. doesn't work for me on 043df13f724cc084a805372b0455cc6d8684cd5b. I expected it to resolve to the flake in the directory above the current flake's. I get "error: path /nix/store/flake.nix doesn't exist".

I also tried path:., but it issues a warning about circular imports (which makes sense, I think).

path:..

Is your flake in a git repo?

If not, that would explain the symptoms.

IMO a flake should be self-contained unless a flake attribute specifies otherwise. Flakes that aren't in a repo are assumed to be self-contained, or in a more practical sense copied at the dirOf flake.nix and not any higher directories. By providing a flake.nix attribute that makes this parent relationship explicit, we could infer which directory is the root of the flake source and work from there.

Alternatively, we'll also want attributes that specify how a flake should be fetch, e.g. enable submodules or lfs. Those could also be assigned the responsibility to determine the flake source root, such that path:.. can work without Git to provide that implicitly.

@roberth, the main flake and the "subflake" both were in the same Git repo.

I had the following layout:

/.git
/flake.nix
/raspberry-pi-5/flake.nix
/thinkpad/flake.nix
...
/shared/...

I have specified nix.package to refer to the commit which closes this issue and I have rebuilt.

I wanted to access /shared from the Raspberry Pi's flake. As it stands, I did not understand how to do this, so now I'm rewriting stuff to have one big /flake.nix.

Well, I figured how to achieve this also using git+file, but it required to constantly nix flake update, which I found inconvenient. Also there's a later commit in Nix which says that relative git+file may be removed in the future.

To be clear, maybe I just did something wrong.

We have a few test cases here that suggest it should work, but maybe yours is more complicated than what we have? From what you describe I would think something representative is in there, except perhaps a dependency chain like the following. A chain like sub1 -> sub2 is tested, but maybe root -> sub1 -> sub2 is a problem? Maybe root -> sub1 copies sub1, cutting it off. Did you try cd-ing into rasperry-pi-5 and build it from there, or did you forward some attr to the root flake?

I'm pretty sure I was in the subdirectory, the top level flake didn't have any nixosConfigurations.

I'll try to see what's going on tomorrow, thanks for the link to the tests.

@roberth, alright, I was doing something wrong, it works. Sorry for bothering.

However I think I found out why I was confused, I think it's an issue with nixos-rebuild repl. I made a small example. Please look at it. If it's a bug indeed, I can create an issue in the tracker (though I'm not sure which project nixos-rebuild belongs to).

/test: Test

/subdir/flake.nix:

{
  inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";

  outputs = inputs: {
    nixosConfigurations.test = inputs.nixpkgs.lib.nixosSystem {
      modules = [
        {
          nixpkgs.system = "aarch64-linux";
          fileSystems."/".device = "/dev/null";
          boot.loader.systemd-boot.enable = true;
          environment.etc.my-test.text = builtins.readFile ../test;
        }
      ];
    };
  };
}

All of the following is executed in subdir.

Test 1.

$ nix repl
nix-repl> :lf .
nix-repl> nixosConfigurations.test.config.environment.etc.my-test.text
"Test\n"

This works as I expect it to work.

Test 2.

$ nixos-rebuild .#test build
$ cat result/etc/my-test
Test

This works as I expect it to work.

Test 3.

$ nixos-rebuild --flake .#test repl
nix-repl> nix-repl> config.environment.etc.my-test.text
error:
[Part of the log omitted...]
       error: opening file '/nix/store/test': No such file or directory

I expected Test and not an error.

Can you confirm if this is a bug, at least conceptually?

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/relative-path-support-for-nix-flakes/18795/7

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-2-26-released/59211/1