Nix: Package script collections with self references

[clkamp]

Is there already a way to use resholve to package a collection of scripts that have (non-cyclic) references between each other in nix? Thanks in advance for your help and the nice tool. :)

[abathur]

It should, though I can't promise some holes won't turn up as we put it through the paces.

If the scripts don't need incompatible solutions, you can just specify all of them:

https://github.com/abathur/resholve/blob/master/ci.nix#L65

At the tail end of the nix demo you'll find the output of the libressl.sh script mentioned above, and see that it references submodule/helper.sh as well. You can run the demo locally with nix-build ci.nix if you want to inspect the resulting files, but you can also see a log of the demo at:

https://github.com/abathur/resholve/blob/master/demos.md

[clkamp]

Thanks for your quick reply. I tried it with a minimal example (and some variations of it) but I don't manage to replicate the example.

This is my try:

───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: a
───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ bin/b
───────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: b
───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ echo "Hello World"
───────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: default.nix
───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ { nixpkgs ? import <nixpkgs> {} }:
   2   │ nixpkgs.callPackage ./test.nix {}
───────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: test.nix
───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ { resholvePackage
   2   │ , bashInteractive
   3   │ }:
   4   │ 
   5   │ resholvePackage rec {
   6   │   pname = "test";
   7   │   version = "unreleased";
   8   │ 
   9   │   src = ./.;
  10   │ 
  11   │   solutions = {
  12   │     profile = {
  13   │       scripts = [ "bin/b" "bin/a" ];
  14   │ 
  15   │       interpreter = "${bashInteractive}/bin/bash";
  16   │ 
  17   │       inputs = [ ];
  18   │ 
  19   │     };
  20   │   };
  21   │ 
  22   │   installPhase = ''
  23   │     mkdir -p $out/bin
  24   │     install a $out/bin/a
  25   │     install b $out/bin/b
  26   │   '';
  27   │ }
───────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

But the result is:

───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: result/bin/a
───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ #!/nix/store/mshinwyjb1imfpdjmrpbbh3r7zz5sn6x-bash-interactive-4.4-p23/bin/bash
   2   │ bin/b
   3   │ 
   4   │ ### resholve directives (auto-generated) ## format_version: 2
   5   │ # resholve: keep bin/b
   6   │ 
───────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: result/bin/b
───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ #!/nix/store/mshinwyjb1imfpdjmrpbbh3r7zz5sn6x-bash-interactive-4.4-p23/bin/bash
   2   │ echo "Hello World"
───────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

And I don't really see the difference to the example you gave...

[abathur]

@clkamp Ah. I do see the same locally.

I guess you're at the intersection of a few things that aren't quite sorted out, yet:

• The main difference between this and the example is the use of source, which I have wired up to be able to resolve from inputs, or from scripts [].
• Beyond source, I haven't intentionally built in any support for resolving relative paths (in part because I'm not sure what the correct/desired behavior here is). But Python's lookup function does resolve relative paths contained by the PWD regardless of what I set the PATH to, and I likewise haven't quite figured out what to do with it. https://github.com/abathur/resholve/blob/591ae30b839d0bae6f02ec7abd852004c6acbbb8/resholve#L1829-L1834 May need to seek out input on this.
  • I think your instance makes a good case for figuring out how to prefix $out there
  • but I'm also a little cautious about accidentally resolving in unexpected files (this implicitly/magically treats all of $out as if it were included in inputs, and I've been trying to avoid magic...)

If you are ~blocked on getting something working, the quick way to proceed is probably:

1. Explicitly add $out to your inputs, i.e. inputs = [ (builtins.placeholder "out") ];
2. Change file "a" to invoke b instead of bin/b.

[clkamp]

Thanks a lot. Your tips solves it for me. Removing the bin/ prefix is even more natural for me and adding the input does not do harm in my case. Should the issue stay open because of the more fundamental considerations?

[abathur]

@clkamp Glad to hear it. It's probably best to leave it open for now.

[abathur]

Came to the late realization tonight that the the 0.6.0 release broke this placeholder workaround. I assume from your description that this won't affect you, but wanted to ~document it publicly :)

[abathur]

The resholve 0.8.0 update (it's merged to nixpkgs, but looks like it'll take a bit to hit channels) includes some incremental improvements on this. There's probably still some work to do here (hopefully driven by the ergonomics of working on a few real packages) so I'll hold off on closing it.

I don't particularly expect you to read the below, but to leave breadcrumbs--the main improvements should be:

1. inputs can now include strings which are interpreted as $out-relative paths (more or less like with scripts). This is intended to replace the need for the placeholder workaround. If these are shell scripts, they should still be in scripts so that they also get resholved.
2. Lore can also be $out-relative.
3. The introduction of lore in 0.6.0 ~broke the inputs/placeholder workaround because it required lore for all of the inputs--and that part is still true for relative path strings per 1 above. The Nix API will now build the target package without resholve, collect lore on it, and then resholve the package in a separate build so that the lore's available.

   • Unless the lore ruled out the possibility that it's an execer, you'll still need to override lore for it. This will be all shell scripts for now, since the lore binlore generates is not tailored for shell yet (https://github.com/abathur/binlore/issues/8). This would look roughly like:

     execer = [
       "cannot:libexec/something"
     ];
     

     Don't just do this automatically, though; look at the script it's complaining about and see if execs its args (and if it does, if any of the call-sites use an external dependency as an arg). If you find one, please report it so that we can start thinking through what to do about it. It would look a little like:

4. Lore is now generated for lib and libexec (before it was just bin), and generated recursively (I've seen at least one package that namespaces its lib/libexec directories).

These should help for golden-path cases, but I still need to do some more package conversions with it to see how well it works in practice. I've already run into at least one case where most (but not all) of the uses are prefixed with a variable expansion (${PACKAGE_LIBEXEC_DIR}/something) that was only minimally improved by this :)