netbird icon indicating copy to clipboard operation
netbird copied to clipboard

SSH broken on Debian 13 due to new login package

Open afonsofrancof opened this issue 1 month ago • 11 comments

Environment

  • NetBird Version: 0.60.3
  • OS: Debian 13 (Trixie)
  • Architecture: amd64
  • Installation Method: apt repository

Description

NetBird SSH connections fail immediately on Debian 13 (Trixie) systems. After successful authentication and PTY allocation, the session closes immediately without spawning an interactive shell. Non-interactive commands work correctly.

Root Cause

After looking through logs and debugging this issue for hours I finally found the cause.

NetBird's SSH server uses login with some flags for interactive sessions which works fine with shadow-login (Debian 12) but fails with util-linux login (Debian 13).

Debian 13 switched from shadow's to util-linux implementation of the login binary (check the right side of the page on those two links. It also shows it on the man page of the packages). The util-linux version behaves differently when invoked by NetBird and exits instead of spawning an interactive shell.

Behavior

  • Connection authenticates successfully
  • PTY allocation is accepted
  • Shell request is accepted
  • Session immediately closes
  • Non-interactive commands work: ssh user@peer 'echo test' succeeds

Receiving peer logs (without prefix because of horizontal space):

... client/ssh/server/server.go:607: SSH connection from NetBird peer 100.64.0.5:62942 allowed

... client/ssh/server/session_handlers.go:35: SSH session started

... client/ssh/server/command_execution_unix.go:147: starting interactive shell: /usr/bin/login

Connecting peer output

❯ ssh [email protected]
Shared connection to togepi.netbird.selfhosted closed.

Workaround

The only way I could get NetBird's ssh to work again was to do the following:

1 - Edit /etc/apt/sources.list and change the repositories back to bookworm 2 - sudo apt update 3 - sudo apt install login=1:4.13+dfsg1-1+deb12u1 4 - Use NetBird's SSH (here I didn't use netbird.selfhosted suffix but it's the same)

Image

This is obviously not desired and I only used it for testing.

Change

Something needs to change here

https://github.com/netbirdio/netbird/blob/aca0398105fd0662c09533e1368a8682310efd94/client/ssh/server/userswitching_unix.go#L77-L100

I will try to experiment with some of the login command flags and will report back in case I find something.

Hope this can be fixed. I was going crazy trying to get this to work 😅

afonsofrancof avatar Nov 28 '25 00:11 afonsofrancof

Tailscale uses basically the same code as netbird for the login so I decided to look in their issues and found some things.

This tailscale issue might be relevant although it mentions musl vs glibc for shadow-utils, not shadow-utils vs util-linux.

There is also this tailscale issue I already tried their solution, but it didn't work for me.

I also tried updating to the latest version of login from debian unstable, because of this bug report and of it's supposed fix, but it did nothing.

The only thing that worked for me was the downgrade I mentioned in the main post

afonsofrancof avatar Nov 28 '25 02:11 afonsofrancof

Found a fix!! Using runuser instead of login works correctly for me!

runuserPath, err := exec.LookPath("runuser")
if err == nil {   
	return runuserPath, []string{"-l", username}, nil
}

I will try to find a way to check for debian 13 from golang and add that as a PR

afonsofrancof avatar Nov 28 '25 11:11 afonsofrancof

Opened PR #4873 to fix this :)

afonsofrancof avatar Nov 28 '25 11:11 afonsofrancof

@afonsofrancof Exactly the same issue running Ubuntu Server 25.10 - this is running util-linux 2.41 which I believe is the same as Debian 13.

I think the PR will need to check more for whether util-linux is being used rather than which OS is installed. I know you can run login -V which will return login from util-linux 2.41 so we can parse that to see if it contains the relevant string?

alexmoras avatar Nov 29 '25 14:11 alexmoras

@alexmoras Hey! Yes, I agree with you and even said so in my PR. The login binary is the problem, so we should fix it on all util-linux distros. I will change that now, just wanted a second opinion :)

Btw, Have you tried the PR? Does it work well for you?

afonsofrancof avatar Nov 29 '25 14:11 afonsofrancof

Ah noted! Apologies, didn't think of looking at the PR. Standby, will test the PR now.

alexmoras avatar Nov 29 '25 14:11 alexmoras

No need to apologize :)

afonsofrancof avatar Nov 29 '25 14:11 afonsofrancof

@afonsofrancof Can confirm that the PR has appeared to have fixed the issue. Will comment separately on the PR now. Nice work!

alexmoras avatar Nov 29 '25 15:11 alexmoras

As noted in https://github.com/netbirdio/netbird/issues/4845#issuecomment-3603319266 this issue is also affecting modern Arch installs. From a very quick bit of research, it would appear that most non-LTS distros will be using util-linux now and therefore impact will be widespread.

alexmoras avatar Dec 02 '25 18:12 alexmoras

Hey folks, can you test this build https://github.com/netbirdio/netbird/pull/4900 (binaries here on the bottom) please?

lixmal avatar Dec 03 '25 08:12 lixmal

@lixmal This fixes the problem for me completely. It shows up correctly in w/who and the loginuid is set correctly. I will close my PR. Thanks!

afonsofrancof avatar Dec 03 '25 13:12 afonsofrancof