NixOS
feat: support managing submodules via inputs.self
Provides special handling for inputs.self. If submodule is set to true, will re-call getFlake as if the original reference was set. The intended behavior is for CLI management of the parameter overrides any direct flake.nix settings.
Thanks to: @x10an14, @aaronchall, @mrene
Motivation
- https://github.com/NixOS/nix/pull/4922
- https://github.com/NixOS/nix/pull/5284
- https://github.com/NixOS/nix/pull/4423
- https://github.com/NixOS/nix/pull/5434
- https://gist.github.com/mstone/4218e6fa9ef98e5153f543fb8e7b4743
Context
Provides a mechanism for inputs.self.submodules to control the availability of submodules in a local flake.
Checklist for maintainers
- [x] test with remote resources (@kranzes)
- [ ] provide check/error for that this logic won't work with "github:"
Maintainers: tick if completed or explain if not relevant
- [ ] agreed on idea
- [ ] agreed on implementation strategy
- [x] tests, as appropriate
- functional tests -
tests/**.sh - unit tests -
src/*/tests - integration tests -
tests/nixos/*
- functional tests -
- [x] documentation in the manual
- [ ] code and comments are self-explanatory
- [x] commit message explains why the change was made
- [ ] new feature or incompatible change: updated release notes
Hm, it seems pretty ugly to add a git-specific submodules field to inputs.self. IMHO the better solution would be to enable submodules by default (after we've implemented efficient submodules support on the lazy-trees branch).
Hm, it seems pretty ugly to add a git-specific
submodulesfield toinputs.self. IMHO the better solution would be to enable submodules by default (after we've implemented efficient submodules support on the lazy-trees branch).
Yeah it isn't fancy. We just implemented it overnight in a call yesterday. We're not even sure if this is even a good idea to have. The thing is that the git-specific submodules inputs field already exists, it's just take effect in the CLI. That's what this PR does, it makes it function as you expect. This isn't an addition of a new feature, it is a fix of an already existing functionality.
Triaged in the Nix team meeting:
- @edolstra: prefer to enable submodules by default
- right now they are extremly inefficient, but may want to make it work with lazy trees
- @thufschmitt: wouldn't hurt to have a feature to forcibly enable them
- @edolstra: this would change the flake language, requires us to continue supporting this eventually
- @tomberek: often in practice people use submodule for private dependencies
- @fricklerhandwerk: how high on our global priority list is this?
- to discuss
@edolstra: prefer to enable submodules by default
I believe that used to be the case but it was reverted because it was fetching when people didn't want it.
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/2023-02-24-nix-team-meeting-minutes-35/25757/1
@edolstra: prefer to enable submodules by default
I believe that used to be the case but it was reverted because it was fetching when people didn't want it.
To be clear it was because of their inefficiency https://github.com/NixOS/nix/issues/5280. But that sounds like a perfect application of lazy trees as mentioned.
To chime in, I've seen C++ projects before where certain submodules are optional except for special cases like testing. While enabling by default would be helpful, I can also imagine a feature like .?submodules=extern/X,extern/Y being useful to keep clones efficient. This wouldn't fit neatly with always including submodules. I don't know how this fits in with flakes though as generally you'd want the flake to cover every case (incl. testing) so it may be very niche, or be on a per output level.
Either way, this feature as implemented in this PR or submodules by default would be incredibly useful for me.
I think we should consider closing this, because the non-libgit fetching has too many problems and a more elegant replacement is in progress in #6530. We'd better make some progress on that, because we should neither develop a dead end, nor let fetcher development be blocked for a year.
Discussed in the Nix team meeting:
We recommend instead to build a Nix language function that re-fetches an input by adding ?submodules=true. This is dirty but avoids changing Nix itself.
We plan on delivering a principled solution eventually anyway, and the soft implementation is a good-enough stopgap.
Complete discussion
- @Ericson2314: it would make sense if we implemented
libgitfirst - @roberth: it's a good mechanism, but we don't have a need for it yet
- @edolstra: the real problem is that it changes the interface by adding
inputs.self, and we'd have to continue supporting that in the future- especially if it hardcodes
submodulesdefault tofalse - @thufschmitt: don't think the default is a problem
- @fricklerhandwerk: is anything speaking against treating that addition as "extra-experimental" in isolation, even if we currently have no means of expressing it in the code?
- @thufschmitt: that's the problem, there is no way of enforcing it whatsoever
- @edolstra: in principle since flakes are experimental, we can change it arbitrarily, but ideally we shouldn't have to change the interface if it can be avoided
- especially if it hardcodes
- @thufschmitt: since
submodulesisfalseby default, one would have to specify the parameter to build the top-level flakes every time- would it make sense set it
truefor the top-level flake by default? - @edolstra: no, that would be very expensive
- would it make sense set it
- @edolstra: it's convenient to the user, but architecturally questionable to do the double-fetching
- @roberth: with lazy trees this will become super cheap
- @thufschmitt: couldn't we instead build a Nix language function that re-fetches an input by adding
?submodules=true?- this is dirty but avoids changing Nix itself
- we will deliver a principled solution eventually, and this is good enough as a stopgap
- decision: postponed
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/2023-03-20-nix-team-meeting-minutes-42/26615/1
What's the status here? I'd really like not to have to specify ?submodules=1 on the command-line every time.
fetchTree will fetch submodules lazily in the upcoming release, 2.20.
We'll have very little reason to disable submodules by default...
EDIT: added to team discussion queue for Friday.
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/get-nix-flake-to-include-git-submodule/30324/13
Is this available on nix version 2.20 onwards? I would very much like an example of how I can build a derivation using a local submodule
Is this available on nix version 2.20 onwards?
It is not.
We'll have very little reason to disable submodules by default...
We're still exploring the solution space for efficiently fetching from a forge such as GitHub, which can return a tarball as a significantly quicker operation, so it's hard to tell whether changing the default setting would make sense.
I do think we'll need a configurable inputs.self regardless.
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/get-nix-flake-to-include-git-submodule/30324/14
Lack of support for submodules for current(self) repo makes packaging most of our software not viable using Nix Flakes.
I'm really hoping we can get support of submodules for self in Nix.
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/handling-git-submodules-in-flakes-from-nix-2-18-to-2-22-nar-hash-mismatch-issues/45118/1
