Shellbang support with flakes (blocked on nix run/shell/develop semantics)

[tomberek]

Enables shebang usage of nix shell. All arguments with #! nix get added to the nix invocation. This implementation does NOT set any additional arguments other than placing the script path itself as the first argument such that the interpreter can utilize it.

Example below:

    #!/usr/bin/env nix
    #! nix shell --quiet
    #! nix nixpkgs#bash
    #! nix nixpkgs#shellcheck
    #! nix nixpkgs#hello
    #! nix --ignore-environment --command bash
    # shellcheck shell=bash
    set -eu
    shellcheck "$0" || exit 1
    function main {
        hello
        echo 0:"$0" 1:"$1" 2:"$2"
    }
    "$@"

Overall it should make sense to any users of nix-shell.

• [ ] add documentation
• [ ] more tests
• [ ] decide on defaults
  • [ ] automatically inject "shell" argument? (edit: nix run would work if it added PATHs)
  • [ ] ~re-use the "-i interpreter" concept from nix-shell?~
  • [ ] lock file options?
  • [ ] whitelist/blacklist certain options
• [ ] remove duplication of heuristic logic/code with nix-build
• [ ] support a bootstrap mechanism detecting if nix is installed and fetching otherwise (eg nix —store or nix-portable)?
• [ ] merge

[roberth]

I really like this. I didn't think we could use nix because portable /usr/bin/env doesn't support extra arguments, but you've proven me wrong :tada:.

My only objection as far as the design is concerned, is that this isn't about shells at all, but on closer reading, this isn't specific to the dubious nix shell command at all. In fact, this seems to solve #801.

[roberth]

To support nix develop, you could override the various basePath/baseDir variables to point to the script's location. That way its functioning does not depend on the caller's working directory, which could be anywhere.

[tomberek]

Another issue that has come up as I'm writing the docs (stealing most of it from nix-shell), is that we don't run the setup or shell hooks to modify PYTHONPATH or PERL5LIB and so forth. Not sure if we want to adopt exactly the legacy nix-shell behavior which led us down an convoluted path. Is there an easier way?

[roberth]

we don't run the setup or shell hooks

This is why we need to rename the current nix shell to something like nix env run. If you want to run stdenv, currently your only nix-command option is nix develop.

I would like Nix to defer to packages, allowing it to take "installing to environment variables" seriously. Maybe it's overkill because for regular commands PATH+XDG_DATA_DIRS will generally do, but perhaps this idea can be used for more complicated stdenv-like environment composition. Assuming Nix can somehow learn how to compose variables. (Not everything is :-separated)

Previously my (and apparently Eelco's) assumption was that we don't need the stdenv/setup-based ad-hoc shells like you are asking for. What's your use case? For packaging we've assumed writing an expression is best.

Anyway that's all fairly orthogonal to the shebang implementation, as long as we don't

automatically inject "shell" argument

I hope I was able to provide some context.

[roosemberth]

   #!/usr/bin/env nix
   #! nix shell --quiet
   #! nix nixpkgs#bash
   #! nix nixpkgs#shellcheck

I find utterly confusing references to derivations are written as #! nix f#pkg instead of #! nix shell f#pkg or simply #! f#pkg.

I think we should also support a varadic derivation references until the end of the line (e.g. #! nix shell f#pkg1 f#pkg2) like nix shell already does.

[roberth]

The #! nix has the same role as #!nix-shell which seemed to have served a technical purpose for potentially mixing shebang aware interpreters. I wonder to what extend this is required. Perhaps it could be optional? For example, we could strip either #! or #!nix, whichever is a longer prefix. That way, we only need to confuse the reader when it's necessary for technical reasons.

Note that #! nix with the space would then be parsed as a #! prefix and a nix argument instead of #! nix as a prefix. I think this is an improvement, because #!nix reads a bit more like a single keyword.

Example:

#!/usr/bin/env nix
#! shell --quiet
#! nixpkgs#bash
#! nixpkgs#shellcheck

And in case we're invoking an interpreter that can and must ignore #!nix lines, but not #! lines.

#!/usr/bin/env nix
#!nix shell --quiet
#!nix nixpkgs#bash
#!nix nixpkgs#shellcheck
#! coolLang foo bar baz use imagination

[roosemberth]

I agree that #!nix would be much more idiomatic. I think making both #! and #! nix valid will just create FUD for no reason.

Do you have examples of shebang-aware interpreters that'd play nice with nix shebangs as described in this document? I've never seen them in the wild (doesn't mean they don't exist), but the idea of supporting bot cases does seem a bit feature-creep to me.

On 2 September 2021 12:45:22 CEST, Robert Hensing @.***> wrote:

The #! nix has the same role as #!nix-shell which seemed to have served a technical purpose for potentially mixing shebang aware interpreters. I wonder to what extend this is required. Perhaps it could be optional? For example, we could strip either #! or #!nix, whichever is a longer prefix. That way, we only need to confuse the reader when it's necessary for technical reasons.

Note that #! nix with the space would then be parsed as a #! prefix and a nix argument instead of #! nix as a prefix. I think this is an improvement, because #!nix reads a bit more like a single keyword.

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/NixOS/nix/pull/5189#issuecomment-911524500 -- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[tomberek]

The proposed approach is variadic and can be put all on one line. Sometimes these lines end up very long and having them split up can be helpful.

As to the prefix, this was meant to copy the way nix-shell does it( most of the code was simply copied or adopted over).

There are several shebang-aware interpreters, eg perl. I’m sure there are more.

The biggest issue is being unable to load the stdenv in the “right way”. I suspect that is actually what people will want. The examples of legacy nix-shell show bringing in python and Perl libraries and that will only work if we run shellHooks or at least do more than just modify PATH.

[ymeister]

Forgive me if I am wrong, but is it really necessary considering env -S option:

-S, --split-string=S
              process and split S into separate arguments; used to pass multiple arguments on shebang lines

That way your example becomes:

#!/usr/bin/env -S nix shell --quiet nixpkgs#bash nixpkgs#shellcheck nixpkgs#hello --ignore-environment --command bash
# shellcheck shell=bash
set -eu
shellcheck "$0" || exit 1
function main {
    hello
    echo 0:"$0" 1:"$1" 2:"$2"
}
"$@"

Aside from multi-line support, I don't see any problem here.

[roberth]

@ymeister that's a good suggestion, but it is not universally supported on the other unixes. It also doesn't allow spaces in arguments for use with --expr.

[tomberek]

Forgive me if I am wrong, but is it really necessary considering env -S option

The original nix-shell shebang mechanism did not rely on that feature - i presume for various compatibility reasons. If one wishes to use it for their own scripts, they are welcome to do so. It should work with this PR (if not, I'd consider that a bug).

My greater concern is that examples such as (https://github.com/NixOS/nix/blob/7ab3bc32b139e251ff8d9abf25d6cadf3c0c6945/doc/manual/src/command-ref/nix-shell.md#use-as-a--interpreter) won't work with nix shell as it is currently implemented. Thus this is involved with the whole nix-shell/nix shell/nix develop/nix run/nix env do-a-thing design and debate. My summary of the situation is that there are a few use cases:

1. Please put some packages A, B, C into my PATH. - usually nix-shell -p somepkg now nix shell
2. I want to develop on package A. - usually nix-shell -A somepkg now nix develop
3. Please make language-specific libraries available: eg: PYTHONPATH, PERL5LIB... - usually nix-shell -p or nix-shell shell.nix
4. Just run a program, don't change any env vars - nix run

To make (1)(3) possible nix-shell would create a derivation behind the scenes on your behalf, composing it together and additionally invoking stdenv environment setup and shell hooks. This is why nix shell nixpkgs#python3 nixpkgs#python3Packages.requests does not work as expected. The shebang example from this PR hits this issue. The only way to express "please run the shellHooks" is to create your own flake devShell, but ad-hoc CLI mechanisms don't seem to allow it.

Another approach

• Can we add "multiple installables" support to nix develop? Then nix develop is always the one that brings in stdenv/shellHooks.

This would create three distinct tools:

• nix run - only runs an executable, no changes to env vars, multiple installables makes no sense
• nix shell - only adds things to PATH, multiple installables means to add them all to the PATH
• nix develop - add things to PATH and run shell hooks. multiple installables means to do a "buildEnv" composition.

[ymeister]

@roberth that's a good notice. Albeit the wide adoption of env -S among other unixes, it doesn't seem to be POSIX-compliant, so that's a bummer. On another note, I haven't experienced any problem using it with --expr:

#! /usr/bin/env -S nix shell --impure --expr "with import <nixpkgs> {}; python.withPackages (pkgs: with pkgs; [ prettytable ])" -c python

import prettytable

# Print a simple table.
t = prettytable.PrettyTable(["N", "N^2"])
for n in range(1, 10): t.add_row([n, n * n])
print t

(with import <nixpkgs> {} requires nixpkgs flake path in $NIX_PATH. That can be achieved either by pinning it system-wide (NixOS), or by exporting NIX_PATH="nixpkgs=$(nix flake metadata nixpkgs --json | jq -r .path)")

Above code also showcases an example of a workaround of the issue discussed by @tomberek. Although I agree that it is just a workaround and is nowhere near a proper solution like one proposed by @tomberek.

[roberth]

I may have been wrong about -S with quotes. I only read the man page, not the info page (:confused:) which goes into more detail, confirming your observations for at least GNU env and FreeBSD. Instead of <nixpkgs> you should be able to reference a local flake when the base path is passed correctly. Also note that nix shell --expr is very different from nix-shell --expr (without and with stdenv respectively; also use case (1) vs (2))

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-shell-with-shebang/17088/2

[Artturin]

Please make sure these 2 keep working, Thanks.

#!/usr/bin/env nix-shell
//! ```cargo
//! [dependencies]
//! time = "0.1.25"
//! ```
/*
#!nix-shell -i rust-script -p rustc -p rust-script -p cargo
*/
fn main() {
    for argument in std::env::args().skip(1) {
        println!("{}", argument);
    };
    println!("{}", std::env::var("HOME").expect(""));
    println!("{}", time::now().rfc822z());
}
// vim: ft=rust

#!/usr/bin/env nix-shell
#![allow()] /*
#!nix-shell -i bash -p rustc
rsfile="$(readlink -f $0)"
binfile="/tmp/$(basename "$rsfile").bin"
rustc "$rsfile" -o "$binfile" --edition=2021 && exec "$binfile" $@ || exit $?
*/
fn main() {
    for argument in std::env::args().skip(1) {
        println!("{}", argument);
    };
    println!("{}", std::env::var("HOME").expect(""));
}
// vim: ft=rust

[chvp]

For anyone stumbling onto this in the future, you can avoid having to set NIX_PATH by using the following (as roberth hinted):

#!/usr/bin/env -S nix shell --impure --expr "(import (builtins.getFlake \"nixpkgs\") {}).python3.withPackages (ps: [ ps.requests ])" --command python

[tomberek]

This has been in draft for too long. I'm going to accept for now that multiple nix develop installables is an independent problem and proceed. @Artturin your examples look like this:

#!/usr/bin/env nix
#![allow()] /*
#!nix shell nixpkgs#rustc --command bash
rsfile="$(readlink -f $0)"
binfile="/tmp/$(basename "$rsfile").bin"
rustc "$rsfile" -o "$binfile" --edition=2021 && exec "$binfile" $@ || exit $?
*/
fn main() {
    for argument in std::env::args().skip(1) {
        println!("{}", argument);
    };
    println!("{}", std::env::var("HOME").expect(""));
}
// vim: ft=rust

#!/usr/bin/env nix
//! ```cargo
//! [dependencies]
//! time = "0.1.25"
//! ```
/*
#!nix shell nixpkgs#rustc nixpkgs#rust-script nixpkgs#cargo --command rust-script
*/
fn main() {
    for argument in std::env::args().skip(1) {
        println!("{}", argument);
    };
    println!("{}", std::env::var("HOME").expect(""));
    println!("{}", time::now().rfc822z());
}
// vim: ft=rust

[grahamc]

I wonder if we should review this a bit more carefully, since it may not work for common use cases like shells with python packages.

Like folks above, I do wonder if we want to add more information to each line, in case we wanted to support different behavior later, like this:

    #! nix shell --quiet
    #! nix shell nixpkgs#bash
    #! nix shell nixpkgs#shellcheck
    #! nix shell nixpkgs#hello

I also wonder: does this support something like this:

.
|_ myscript.sh
\_ flake.nix

where myscript.sh refers to the dev shell of flake.nix?

One weird thing about nix-shell shebangs is that additional arguments can be added in the middle / at the end of a script, which I found surprising in a number of places. I wonder if we could restrict it to find shebang data at the top of a file and then stop reading?

Has anyone tested to see if this works with #!/usr/bin/env nix ... on a mac?

[tomberek]

Are script arguments supported?

Yes

[fricklerhandwerk]

Triaged in Nix team meeting on 2022-11-25:

• @thufschmitt: have to discuss this, the design space is pretty broad here

-> to discuss

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2022-11-25-nix-team-meeting-minutes-11/23601/1

[fricklerhandwerk]

Discussed in Nix team meeting 2022-12-23:

• @roberth: @tomberek is not here, so let's keep it short
• @edolstra: nix shell is not sufficient for things like perl with perl packages
• @roberth: nix develop is supported
• @edolstra: need a nix-shell -p equivalent
  • @roberth: this is mostly an orthogonal problem, not sure if I said that in the meeting
• @roberth: I don't see a nix develop or nix run test case (anymore?? not sure)

[tomberek]

Discussed in Nix team meeting 2022-12-23:

• @roberth: @tomberek is not here, so let's keep it short

Apologies.

• @edolstra: nix shell is not sufficient for things like perl with perl packages

• @roberth: nix develop is supported

• @edolstra: need a nix-shell -p equivalent

  • @roberth: this is mostly an orthogonal problem, not sure if I said that in the meeting

The current design is doing the bare minimum and merely interprets the command line in a specific way. It therefore supports any nix command. This might be too powerful, but to help explain:

#!/usr/bin/env nix
#! nix eval
#! nix nixpkgs#hello --json --option dummy
# the dummy option parameter is to swallow the file arg that is passed at the end

or

#!/usr/bin/env nix
#! nix store add-file
when run, this script will add itself into the store

But I don't think these are the priority.

• @roberth: I don't see a nix develop or nix run test case (anymore?? not sure)

The issue about develop supporting perlPackages is that one cannot compose the environment you want without expressing a function call in Nix (buildEnv, mkDerivation, withPackages, etc). We can interpret multiple installables for the develop command to have a specific composition, or use --apply, or something else. This lack of support is not shebang specific, but is about develop, and any fix for it is easily adopted by the shebang support.

nix run support is largely subsumed by nix shell --command bash, but this brings up a good question. One might want to support this:

#!/usr/bin/env nix
#! nix run
#! nix nixpkgs#hello --option dummy

But the file parameter is a bit annoying, should we have a form of the shebang that does NOT append the own-file-name to the call?

#!/usr/bin/env nix
#! nix run nixpkgs#hello        # comments are ignored and will swallow the own-file-name?

I'm open to ideas.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2022-12-23-nix-team-meeting-minutes-19/24400/1

[fricklerhandwerk]

Reviewed in Nix team meeting 2023-05-01:

• status unclear
• @roberth: stuck on syntax?
• @edolstra: not a replacement for nix-shell -p yet
• @roberth: that's ok, separate scope
• @roberth: can technically be done with nix develop --expr '...'
• @tomberek: what about the syntax? Currently scans all comment-like lines
• @tomberek will rebase, @roberth still assigned for review

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-05-01-nix-team-meeting-minutes-51/27803/1

[roberth]

Oops, I didn't reset the ~upstream~ pushRemote. Made a separate PR with my additions. I'll revert this in a sec. EDIT: github doesn't seem to tell me the old hash, or at least I'm not sure. I'll leave it at these rebased commits.

[roberth]

I've changed the quoting and interpretation of relative paths in #8327. wdyt?

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/flox-nix-community-update-2023-04/28233/1

[tomberek]

I've changed the quoting and interpretation of relative paths in #8327. wdyt?

Looks reasonable, but more contentious. I'm okay with either approach.