Allow flakes to refer to other flakes by relative path

[edolstra]

E.g.

{
  inputs.foo = { url = ./../foo; };
}

This is primarily useful when you have multiple flakes in the same repository.

However this means that we can't have a lock file entry for the foo input.

[roberth]

A lock file shouldn't be necessary as long as the input is going to be committed to the same repository.

It only becomes a problem when the reference points outside the repository. Doing is useful for testing a change in a dependency, but a warning seems appropriate.

[zimbatm]

It would be good to disallow .. in the relative paths to encourage composing things reliably.

[kloenk]

Can we only do one lockfile for this, like cargo workspaces does it?

[roberth]

@zimbatm Disallowing .. in relative paths forces the flakes to form a tree rather than a DAG. Suppose you have a repo with flakes with dependencies A -> P -> X and A -> Q -> X, you'd have to duplicate flake X for flakes P and Q. .. is also useful for ad-hoc testing. For example, you may want to use this feature temporarily to iterate on a dependency that's outside the repo.

@Kloenk I think you get half of this behavior for free if you don't invoke nix flake commands on the dependency flakes. The other half is that if you do invoke it, it can use the parent's lockfile. In cargo this seems to be implicit. Another example is Maven, where you reference the parent pom explicitly. If we require an explicit reference to the parent flake, we preserve the ability to opt out of the single lockfile, which can be important for monorepos. I also like that it reinforces that the flake isn't quite independent. For example, you can't just copy its files and expect nix flake commands to work the same outside the context of the parent. This is true whether the parent reference is explicit or not, but if it's explicit, you'll get an error message to "manage" that expectation; a useful message that's easy to resolve.

[stephank]

So, I’m considering if this is something I want/need. In my case, I have a repository of nixosConfiguration flakes in subdirectories (each a project with one or more machines), and a shared lib. There’re currently hacks in place to make the lib stuff work, because it technically lives outside any of the flake contexts.

• Workspaces sound like a different piece of functionality (pinning all flake dependencies as a whole), and in my case, I want separate lockfiles (per project).

• At the same time, workspaces could be a more sensible way to declare a toplevel than ‘discover git/vcs root’, if we want logic to prevent/warn escaping a certain context.

• My current setup would require .. to work, because lib is outside the flake context.

[MagicRB]

This can sort of be simulated with https://github.com/edolstra/flake-compat, you must make your build impure, but even though it sounds bad, it isn't that bad. If you know your build is pure, you're fine, if you aren't sure, you can temporarily disable the sub-flakes and test each one separately build nix build-ing it. To get this to work with nixos-rebuild you have to edit the script and add "--impure" to extraBuildFlags. Or add an option into the switch

oh and all the sub-flakes have their own lock-files, therefore its necessary to update them recursively, I'll write a script for that.

[MagicRB]

@stephank

* My current setup would require `..` to work, because `lib` is outside the flake context.

perhaps something like this could work, though I have no clue how to actually implement it

{
  inputs = {
    your-lib = {
      path = ./your-lib; # different from url to distinguish it
    };
    
    machine-1 = {
      path = ./machine-1;
      pass-on = your-lib; # I don't really know what `your-lib` should be, it could be a string, but that could cause confusion as registry references look the same. Also, I already saw something like this somewhere, can't remember what was the option, if you know then please tell me.
    };
  };
}

{
  inputs = {
    your-lib = {
      follows = "your-lib"; # this is the option used for referencing flake registries, I'm sure this time
    };
  };
}

[jaen]

I just want to add another two cents to the bandwagon – I have two use cases that keep me wanting to reach for flakes with a relative path (and possibly some kind of arguments, but that's a separate discussion).

One is the already mentioned idea to factor out common code into a lib flake shared across some nixosConfiguration flakes and possibly version things like nixpkgs on a per-configuration basis. Another is using nix to build a multi-language repo I have for work, where each sub-project could be a flake I then bring together.

I'm not sure if it's the right instinct to try reaching out for things like that, as I've been using NixOS mostly for Terraform/Ansible angle of reproducible configuration and I'm not sure how that fits in the package building conventions Nix has – maybe there are other options that are better suited to those use-cases I'm not aware of.

[MagicRB]

I'd add that sub-flakes are a useful tool to "partition" your repo, for example my dotfiles (EDIT: I moved those packages to my systems repo) themselves provide custom packages I created and I propagate those up from "half-flakes", flakes whose inputs are declared at the toplevel flake, but look like separate flakes and mostly behave the same way. Just call the outputs lambda with the unified inputs and self

[jaen]

Another semi-related thing I've bumped against just now is that after a few tries to use things like submodules or impure imports via flake-compat (dunno if that's kosher, like I'm said, I'm new at this) I bumped into enough issues that I decided to just split it out as separate flakes.

Unfortunately this now incurs the change->commit->push->update-flake penalty to make a simple change which is kind of annoying, is there plans for some way to develop flakes alongside a la yarn link (basically, substitutes a package with a local copy, so the changes you make are visible to the package that depends on it) for flakes. If not, would it make sense to open an issue for it?

[MagicRB]

impure imports via flake-compat (dunno if that's kosher, like I'm said, I'm new at this)

definitely not

Unfortunately this now incurs the change->commit->push->update-flake penalty

look at my dotfiles, I had the same issue and I solved it by those half-flakes

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/possibility-for-subflakes/11589/5

[jorsn]

Currently, relative paths work already with the path fetcher. However, this comment looks like the restrictions are just not implemented. flake-compat doesn't allow relative paths, because builtins.path needs an absolute path string or a path (e.g. ./. + "/lkjsfl") as attribute path in the argument set.

Edit: What is the problem with a lock file?

[BBBSnowball]

basically, substitutes a package with a local copy, so the changes you make are visible to the package that depends on it

nix flake update --override-input ... might do what you want. If you use it with path:/something, it will record the hash of that path so you have to run the command again after changing any file in /something. This should remove the "commit->push" part but you still have to update the flake.

This was still too much overhead in my edit-test-loop. Therefore, I have merged most of my repos and use a similar approach as MagicRB.

I hope relative paths with ".." will remain possible for path: and I will switch to plain .. if this becomes possible. I use (misuse?) them for these purposes:

1. I have a repo with modules and configurations (nixosConfigurations) for several hosts. Later, I wanted to change the nixpkgs input for one of them. I did that with a sub-flake (in the same repo) that uses the main flake via path:../... The sub-flake has its own lockfile.
2. A friend wanted to use a package from my flake. He is using nixos-unstable while I'm forcing a specific version of nixos-20.09 in the flake. Therefore, any package from my flake will download lot's of dependencies for him. I have created a sub-flake that sets inputs.nixpkgs.url = "nixpkgs" and doesn't have a lock file. That way, he can add ?dir=nolock to the flake url and all is well.

[nrdxp]

Currently this already works with inputs.subflake.url = "path:./adirectory", but if a lock file is not already generated, it can fail if your not in the directory with the flake.nix. Once the lock file exists, you can reference the flake from anywhere and it works as expected.

Perhaps a more elegant solution to this, which would also solve the above mentioned issue, would be to pass a special self reference to the flake inputs, in a similar manner as is passed to outputs. Then you could reference a subflake as inputs.subflake.url = "self:adirectory".

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/possibility-for-subflakes/11589/1

[DieracDelta]

This issue with path is that you cannot mark submodules = true;. This complicates things if you want to build from a local checkout of a GitHub repo and all its submodules.

[zhaofengli]

The current path:../sub-dir-of-same-repo solution doesn't actually work as it comes with a giant catch: Everything, including .gitignore'd files, in that directory is copied. This is unacceptable when you are developing and your flake directory contains loads of build artifacts. In the current implementation you will need to cargo clean or manually move the offending directories out from the input flake in order to not copy lots (in my case, gigabytes) of data into the Nix store.

[roberth]

@zhaofengli does git+file:../sub-dir-of-same-repo work?

[zhaofengli]

@roberth

Thanks for the suggestion. I tried the following and got:

• git+file:../sub-dir -> fetching Git repository 'file:../sub-dir'ssh: Could not resolve hostname file: Name or service not known
• git+file://../sub-dir -> error: file:// URL 'git+file://../sub-dir' has unexpected authority '..'
• git+file:/..?dir=sub-dir -> error: file:// URL 'git+file://..?dir=sub-dir' has unexpected authority '..'
• git+file:///home/zhaofeng/repo?dir=sub-dir -> Works, but it's not really scalable when you have multiple flakes and collaborate with other people

[edolstra]

Closing, you can have inputs like inputs.foo.url = "path:../bar" now.

[winston0410]

@edolstra can we? I have an input inputs.hostname.url = "path:../hostname.nix", and I got this:

error: relative path '../hostname.nix' points outside of its parent's store path '/nix/store/w5ymrwy0igygh8swkm449kb0f7kx1k3v-source'

[roberth]

Relative paths currently rely on a source with the correct hash to be already present in the store. This is fragile but won't be noticed by the flake author, because creating the lockfile adds locally. See #5437

[therealpxc]

This is an annoyance for my config, as I use subflakes to provide flake interfaces for non-flake inputs to my top-level flake.nix, to make things a little cleaner there and to pave the way for sending the flake interfaces for those inputs upstream.

This issue is definitely noticeable for flake authors if you don't go into the subdirectories for your subflakes and run nix flake lock every time you change those things (which you probably don't want to bother to do during development).

Since adding an in-repo flake input by relative path sometimes doesn't work (depending on whether that subflake is already in the Nix store), I think this issue should be reopened.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/locally-excluding-nix-flakes-when-using-nix-independenly-of-upstream/16480/19

[schuelermine]

However this means that we can't have a lock file entry for the foo input.

This seems untrue if the directory is a git repository.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/relative-path-support-for-nix-flakes/18795/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/relative-path-support-for-nix-flakes/18795/2

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/workflows-with-flakes/19223/2

[galenhuntington]

The above discussion indicates to me that this issue has not been solved and should be reopened.