util-linux inclusion in pkgs.proxmox-ve paths somehow breaks tailscale ssh

[Patricol]

I wish I were nix-savvy enough to know exactly why this is happening. Tailscale's SSH feature intercepts SSH connections on port 22 (when those connections arrive via Tailscale's network) and sends them to its own binary. I've had trouble getting it to work on my proxmox-nixos hypervisors.

After bisecting my entire nix configuration, I narrowed the setting that breaks Tailscale SSH first to services.proxmox-ve.enable = true;; then to the environment.systemPackages = [ cfg.package ]; line in this repo's modules/proxmox-ve/default.nix; then to this line.

When that one line is commented out, Tailscale SSH always works. When it is not commented out, Tailscale SSH always fails.

Please let me know if you have any intuition as to why.

I see that util-linux was added in support of the toggle-able linstor functionality. If we can't figure out why exactly this is breaking; maybe we can move util-linux into the conditionally-included list? (like this)

[Zocker1999NET]

To me it seams that Tailscale’s SSH functionality depends on certain binaries being available in the system environment. I conclude this because the package you mentioned subsequently adds util-linux to the path of the system environment. So I assume (but have no time to test it for you) that Tailscale SSH also break with just setting environment.systemPackages = [ pkgs.util-linux ]; in your case.

Because you did not mention how Tailscale SSH breaks (e.g. error logs), it could theoretically also be an issue with the SSH session handling (e.g. opening your default shell, loading .profile ...), which is expected to rely on the system environment. But any internal behavior of Tailscale installed from nixpkgs should IMO not depend on any packages existing (or not existing) in the system environment, so you might give it a shot by reporting this upstream to nixpkgs (esp. if my example above without Proxmox also fails).

[Patricol]

At the time I assumed you were completely correct, but now upon revisiting this I find that environment.systemPackages = [pkgs.util-linux] does not break tailscale ssh.

I did not provide error logs because the ones I can get do not seem particularly useful. ssh -vvv output is completely identical in the fixed (removing util-linux from the proxmox-ve file) and unfixed runs, up until one succeeds and the other outputs Authentication failure. Of course, I should still have specified that error message.

server-side journalctl is also identical until pam_unix(login:session): session opened for user user(uid=1000) by (uid=0) and

pam_warn(remote:account): function=[pam_sm_acct_mgmt] flags=0 service=[remote] terminal=[/dev/pts/0] user=[user] ruser=[<unknown>] rhost=[100.104.6.24]
login[2866148]: Authentication failure

which I guess is my next lead to follow; though I may just stick with my fork for now.

Edit: /etc/pam.d/* contents are identical in both cases too.

[joshryandavis]

Just had this too. Thanks @Patricol for identifying.

Does seem to be an issue with the packaging of Tailscale rather than proxmox-ve.

Explicitly installing util-linux fixes it.

[Patricol]

Installing util-linux does not fix it for me.

I managed to make a kludgey overlay that I can use instead of a fork.

nixpkgs.overlays = [
  (self: super: {
    proxmox-ve = super.proxmox-ve.override (previous: {
      util-linux = previous.wget;
    });
  })
];

pkgs.buildEnv doesn't remove null etc. when processing paths; so I duplicated a harmless package that's already present.

You could likely do something nicer with overrideAttrs; but I think that'd require (de/re)serializing json strings etc. - enough code that I prefer the hack.