Setting PATH in NIX_SSHOPTS no longer works!

[Fuuzetsu]

Describe the bug

Due to https://github.com/NixOS/nix/issues/1078 , we have been using nix-copy-closure (or nix copy, whichever) with

NIX_SSHOPTS='PATH=/nix/var/nix/profiles/default/bin:"$PATH"'

This worked! However upon updating nix to 2.15, this stopped working for me. I made a shim in /tmp/bin to show the ssh command:

[shana@aya:~/programming/engine]$ cat /tmp/bin/ssh 
#!/usr/bin/env bash
arr=( "$@" )
echo ssh "$@" > /tmp/ran
for i in "${arr[@]}"; do
    echo "$i" >> /tmp/ran
done

/run/current-system/sw/bin/ssh "$@"

and ran with known-working nix 2.14:

[shana@aya:~/programming/engine]$ PATH=/tmp/bin:"$PATH" NIX_SSHOPTS='PATH=/nix/var/nix/profiles/default/bin:"$PATH"' $(nix-build --no-out-link ~/programming/nixpkgs -A nixVersions.nix_2_14)/bin/nix copy --from ssh://kobuta.production.tsuru.capital /nix/store/ch8jsr2qdymmz0xqbmy02fspb8inxanx-liveviewer

[shana@aya:~/programming/engine]$ cat /tmp/ran
ssh kobuta.production.tsuru.capital -x PATH=/nix/var/nix/profiles/default/bin:"$PATH" nix-store --serve --write
kobuta.production.tsuru.capital
-x
PATH=/nix/var/nix/profiles/default/bin:"$PATH"
nix-store --serve --write

This works. Now I replace nix with 2.15 instead and it no longer works.

[shana@aya:~/programming/engine]$ PATH=/tmp/bin:"$PATH" NIX_SSHOPTS='PATH=/nix/var/nix/profiles/default/bin:"$PATH"' $(nix-build --no-out-link ~/programming/nixpkgs -A nixVersions.nix_2_15)/bin/nix copy --from ssh://kobuta.production.tsuru.capital /nix/store/ch8jsr2qdymmz0xqbmy02fspb8inxanx-liveviewer
bash: -oPermitLocalCommand=yes: command not found
error: failed to start SSH connection to 'kobuta.production.tsuru.capital'

[shana@aya:~/programming/engine]$ cat /tmp/ran
ssh kobuta.production.tsuru.capital -x PATH=/nix/var/nix/profiles/default/bin:"$PATH" -oPermitLocalCommand=yes -oLocalCommand=echo started nix-store --serve --write
kobuta.production.tsuru.capital
-x
PATH=/nix/var/nix/profiles/default/bin:"$PATH"
-oPermitLocalCommand=yes
-oLocalCommand=echo started
nix-store --serve --write

Steps To Reproduce

Just try using PATH with NIX_SSHOPTS on nix 2.15

Expected behavior

I'd expect it to continue to work...

nix-env --version output

2.15.0

Additional context

Quick search in history points to #8018, notably https://github.com/NixOS/nix/pull/8018/commits/5291a82cd9b9d8d7cd6b8338a5224c94c6f23eb7

I'm guessing this doesn't work as it now thinks that the -o stuff is a command, not nix-store command. Maybe NIX_SSHOPTS should be appended, not pre-pended?

Priorities

Add :+1: to issues you find important.

[thufschmitt]

I'm guessing this doesn't work as it now thinks that the -o stuff is a command, not nix-store command. Maybe NIX_SSHOPTS should be appended, not pre-pended?

That sounds reasonable. Care to open a PR for that? (Using it that way is a bit of a hack though, and I wouldn't make any guaranty about it not breaking again in the future).

Note that for your original issue (which is utterly annoying, I'm honestly quite angry at #1078 still being such a thing) you can also add ?remote-program=/run/current-system/sw/bin/nix-store (or nix-daemon for ssh-ng) to the store URL

[Fuuzetsu]

@thufschmitt I tried remote-program and it works with ssh:// though not ssh-ng:// (see at the bottom, :shrug:) thank you for pointing me to that. This is much better than the PATH hack.

I think this should be very much louder stated... somewhere. At least I can't find the reference to it in #1078! Sadly it only works for nix copy and not nix-copy-closure that I can tell but I think most people on nix 2.15 can enable nix-command even if only for just this case. I guess there is some use case where this isn't possible but it's not mine.

That sounds reasonable. Care to open a PR for that?

I can try but it's going be be inefficient. If someone can do it in 5 minutes, that's probably better. If there are no takers, I can spend however long trying to set it up and verify it works.

[shana@aya:~/programming/engine]$ ./bin/viewer # ssh-ng version
warning: Git tree '/home/shana/programming/engine' is dirty
error: no operation specified
Try '/nix/var/nix/profiles/default/bin/nix-store --help' for more information.
error: cannot open connection to remote store 'ssh-ng://kobuta.production.tsuru.capital': error: unexpected end-of-file

[shana@aya:~/programming/engine]$ ./bin/viewer # ssh version
warning: Git tree '/home/shana/programming/engine' is dirty
<snip, works here>

[thufschmitt]

I can try but it's going be be inefficient. If someone can do it in 5 minutes, that's probably better. If there are no takers, I can spend however long trying to set it up and verify it works.

Fair enough, I've opened #8303 for that.

I think this should be very much louder stated... somewhere. At least I can't find the reference to it in https://github.com/NixOS/nix/issues/1078!

Indeed. @balsoft opened https://github.com/NixOS/nix/pull/6628 for that, but the reviews got it lost in over-engineering so it never landed. We should resurrect it (probably in a simpler form).

it only works for nix copy and not nix-copy-closure

Mh, it should. nix-copy-closure just does some nasty string concatenation to build an ssh:// url out of the given host, so we can pass arbitrary arguments to it. At least I just tried nix-copy-closure --to localhost\?remote-program=/bin/sh /run/current-system and it does try to run sh as the remote program.

[georgefst]

Note that for your original issue (which is utterly annoying, I'm honestly quite angry at #1078 still being such a thing) you can also add &remote-program=/run/current-system/sw/bin/nix-store (or nix-daemon for ssh-ng) to the store URL

Thanks for this! This is the nicest workaround I've found for #1078.

By the way, & appears to be a typo, as I've found that only ? works, as you used in your later comment. It might be helpful to correct this for the sake of future readers.

[thufschmitt]

& appears to be a typo, as I've found that only ? works, as you used in your later comment.

It's not entirely a typo. The store urls are urls(ish) as their name indicate, and the parameters are passed as url parameters (?param1=value2&param2=value2&param3=value3). So it depends whether you already have parameters. But yeah, it's confusing indeed, I'll fix that in the comment.

[FlafyDev]

Does anybody know how to set remote-program decoratively with nix.buildMachines(or any other way)?

[malteneuss]

just wanted to mention the needed \? escaping for remote-program:

nix store info --store ssh://<user>@<ip>\?remote-program=/nix/var/nix/profiles/default/bin/nix-store
# older Nix package manager versions:
nix store ping --store ssh://<user>@<ip>\?remote-program=/nix/var/nix/profiles/default/bin/nix-store