netbird icon indicating copy to clipboard operation
netbird copied to clipboard

Require SSH Policy, but not allow PubKey?

Open baughmann opened this issue 1 month ago • 7 comments

Describe the problem

Since Netbird v0.60.0, Netbird requires an SSH policy. This is a great step, but there seems like PasswordAuthentication is still required. As standard practice, I disable PasswordAuthentication on all servers and require PubKey only.

Not only has the SSH UI within the web client never worked with pubkey-required auth, but it now appears that servers running the new client no longer work. See below.

This is not a huge deal. My personal SOP is to not enable SSH for the peer in Netbird anyway, but rather to restrict SSH to the wt0 interface. So far this has worked just fine as long as I bypass Netbird.

# (on my local machine)
➜  ~ ssh my-server.netbird.cloud
SSH authentication required.
Please visit: https://login.netbird.io/activate?login_hint=me%40email.com&user_code=SOME-CODE
Or visit: https://login.netbird.io/activate?login_hint=me%40email.com and enter code: SOME-CODE
Waiting for authentication...
# I go to the browser and authenticate
➜  ~ 
# Client disconnects immediately with no errors logged to the client

Here are the corresponding lines from journalctl:

Nov 23 10:42:23 my-server sshd-session[3850414]: Accepted publickey for my_server_username from my_local_lan_ip_not_netbird_ip port 38086 ssh2: RSA SHA256:vUN0/gptFbbaxzVWIkmhPntRHUcZ4KZGf0AKVtkVFFs
Nov 23 10:42:23 my-server sshd-session[3850414]: pam_unix(sshd:session): session opened for user my_server_username(uid=1000) by my_server_username(uid=0)

To Reproduce

Steps to reproduce the behavior:

  1. Forbid any SSH authentication method except PubKey, restart SSH on the server
  2. On your client machine, ssh [email protected]
  3. Click the link to auth in the browser
  4. See immediate disconnection

Expected behavior

I should be dropped into an SSH session.

Are you using NetBird Cloud?

Yes

NetBird version

Client: 0.60.2 Server: 0.60.2

Is any other VPN software installed?

If yes, which one?

N/A

To help us resolve the problem, please attach the following anonymized status output

netbird status -dA

Create and upload a debug bundle, and share the returned file key:

netbird debug for 1m -AS -U

Uploaded files are automatically deleted after 30 days.

Alternatively, create the file only and attach it here manually:

netbird debug for 1m -AS

Not sure that's necessary.

Screenshots

N/A

Additional context

Add any other context about the problem here.

Have you tried these troubleshooting steps?

  • [ ] Reviewed client troubleshooting (if applicable)
  • [X] Checked for newer NetBird versions
  • [X] Searched for similar issues on GitHub (including closed ones) #1558
  • [X] Restarted the NetBird client
  • [X] Disabled other VPN software
  • [X] Checked firewall settings

baughmann avatar Nov 23 '25 15:11 baughmann

Yes, same for me!

dionesku avatar Nov 24 '25 14:11 dionesku

Can you please upload a debug bundle from the peer running the ssh server?

Also

 NB_LOG_LEVEL=debug ssh -vv  my-server.netbird.cloud

on the client

lixmal avatar Nov 24 '25 16:11 lixmal

I'm seeing exactly the same issues on my end, running the same version on both the SSH server and client. Running netbird ssh nas exits immediately with no message. The traditional ssh nas.mesh.X.X command immediately returns Connection to nas.mesh.X.X closed.

To assist in debugging, I've run the above steps on my server / client. The server debug-log key is 6a027a4f5b46388625fbc889ae50c64a00b6ea8c44729154280fc0f067782827/9cac4bc7-900c-42d5-843f-ed8efc94b6f2.

I've run the above debug command on the client which returns the following:

Debug output
alex@laptop:~$  NB_LOG_LEVEL=debug ssh -vv nas.mesh.X.X
debug1: OpenSSH_10.0p2, OpenSSL 3.5.4 30 Sep 2025
debug1: Reading configuration data /home/alex/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf
debug1: Reading configuration data /etc/ssh/ssh_config.d/30-libvirt-ssh-proxy.conf
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug2: checking match for 'final all' host nas.mesh.X.X originally nas.mesh.X.X
debug2: match not found
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Reading configuration data /etc/ssh/ssh_config.d/99-netbird.conf
debug1: /etc/ssh/ssh_config.d/99-netbird.conf line 8: Applying options for nas.mesh.X.X
debug2: checking match for 'exec "/usr/bin/netbird ssh detect %h %p"' host nas.mesh.X.X originally nas.mesh.X.X
debug1: Executing command: '/usr/bin/netbird ssh detect nas.mesh.X.X 22'
2025-11-25T21:51:58Z DEBG client/ssh/detection/detection.go:82: SSH server banner: SSH-2.0-NetBird-SSH-Server-0.60.2
debug2: match not found
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /home/alex/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf
debug1: Reading configuration data /etc/ssh/ssh_config.d/30-libvirt-ssh-proxy.conf
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug2: checking match for 'final all' host nas.mesh.X.X originally nas.mesh.X.X
debug2: match found
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Reading configuration data /etc/ssh/ssh_config.d/99-netbird.conf
debug1: /etc/ssh/ssh_config.d/99-netbird.conf line 8: Applying options for nas.mesh.X.X
debug2: checking match for 'exec "/usr/bin/netbird ssh detect %h %p"' host nas.mesh.X.X originally nas.mesh.X.X
debug1: Executing command: '/usr/bin/netbird ssh detect nas.mesh.X.X 22'
2025-11-25T21:51:58Z DEBG client/ssh/detection/detection.go:82: SSH server banner: SSH-2.0-NetBird-SSH-Server-0.60.2
debug2: match not found
debug2: resolving "nas.mesh.X.X" port 22
debug1: Connecting to nas.mesh.X.X [100.X.X.238] port 22.
debug1: Connection established.
debug1: identity file /home/alex/.ssh/id_rsa type 0
debug1: identity file /home/alex/.ssh/id_rsa-cert type -1
debug1: identity file /home/alex/.ssh/id_ecdsa type -1
debug1: identity file /home/alex/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/alex/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/alex/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/alex/.ssh/id_ed25519 type -1
debug1: identity file /home/alex/.ssh/id_ed25519-cert type -1
debug1: identity file /home/alex/.ssh/id_ed25519_sk type -1
debug1: identity file /home/alex/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/alex/.ssh/id_xmss type -1
debug1: identity file /home/alex/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_10.0
debug1: Remote protocol version 2.0, remote software version NetBird-SSH-Server-0.60.2
debug1: compat_banner: no match: NetBird-SSH-Server-0.60.2
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to nas.mesh.X.X:22 as 'alex'
debug1: load_hostkeys: fopen /home/alex/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: mlkem768x25519-sha256,curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,ext-info-c,[email protected]
debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: [email protected],[email protected],aes256-ctr,[email protected],aes128-ctr
debug2: ciphers stoc: [email protected],[email protected],aes256-ctr,[email protected],aes128-ctr
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512
debug2: compression ctos: none,[email protected]
debug2: compression stoc: none,[email protected]
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,[email protected]
debug2: host key algorithms: ssh-ed25519
debug2: ciphers ctos: [email protected],[email protected],[email protected],aes128-ctr,aes192-ctr,aes256-ctr
debug2: ciphers stoc: [email protected],[email protected],[email protected],aes128-ctr,aes192-ctr,aes256-ctr
debug2: MACs ctos: [email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96
debug2: MACs stoc: [email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:gZ8M6WOlDfNTagQm2CTmWvYB7sDLplIoxBlofLY5mL4
debug1: load_hostkeys: fopen /home/alex/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'nas.mesh.X.X' is known and matches the ED25519 host key.
debug1: Found key in /home/alex/.ssh/known_hosts:232
debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: ssh_packet_read_poll2: resetting read seqnr 3
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 4294967296 blocks
debug2: KEX algorithms: mlkem768x25519-sha256,curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,ext-info-c,[email protected]
debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: [email protected],[email protected],aes256-ctr,[email protected],aes128-ctr
debug2: ciphers stoc: [email protected],[email protected],aes256-ctr,[email protected],aes128-ctr
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512
debug2: compression ctos: none,[email protected]
debug2: compression stoc: none,[email protected]
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_ext_info_client_parse: server-sig-algs=<ssh-ed25519,[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-sha2-512,ssh-rsa,ssh-dss>
debug1: kex_ext_info_check_ver: [email protected]=<0>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
Authenticated to nas.mesh.X.X ([100.X.X.238]:22) using "none".
debug1: pkcs11_del_provider: called, provider_id = (null)
debug1: channel 0: new session [client-session] (inactive timeout: 0)
debug2: channel 0: send open
debug1: Entering interactive session.
debug1: pledge: filesystem
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug1: channel 0: setting env COLORTERM = "truecolor"
debug2: channel 0: request env confirm 0
debug2: channel 0: request shell confirm 1
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 2097152 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
debug2: channel 0: rcvd close
debug2: channel 0: output open -> drain
debug2: chan_shutdown_read: channel 0: (i0 o1 sock -1 wfd 4 efd 6 [write])
debug2: channel 0: input open -> closed
debug2: channel 0: obuf empty
debug2: chan_shutdown_write: channel 0: (i3 o1 sock -1 wfd 5 efd 6 [write])
debug2: channel 0: output drain -> closed
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send_close2
debug2: channel 0: send close for remote id 0
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
Connection to nas.mesh.X.X closed.
Transferred: sent 2088, received 1324 bytes, in 0.0 seconds
Bytes per second: sent 262289.7, received 166317.8
debug1: Exit status -1

I've confirmed that SSH is enabled in both the web interface and on the server, with the relevant flags.

alexmoras avatar Nov 25 '25 22:11 alexmoras

Yes, same issue.

Also hitting this on v0.60.4. Environment:

  • Client: Arch Linux with NetBird 0.60.4
  • Server: AWS EC2 (Debian 13) in private subnet with NetBird 0.60.4
  • Setup with all SSH flags enabled

Symptoms:

  • JWT auth succeeds, PTY allocated, shell request accepted
  • Session immediately exits with remote command exited without exit status or exit signal
  • Success rate: ~5% (works maybe 1 in 20 attempts). Browser SSH client also sometimes works

This makes NetBird SSH essentially unusable in production for me for now.

ullanar avatar Dec 02 '25 18:12 ullanar

@ullanar I wonder if this might be related to issue #4869 (PR #4873) since there appears to be ongoing issues with more modern versions of Linux that rely on util-linux rather than shadow-login.

Do you still get the issues when password authentication is enabled?

alexmoras avatar Dec 02 '25 18:12 alexmoras

@alexmoras Good call - tested with PasswordAuthentication yes and the issue persists with the same success rate.

ullanar avatar Dec 02 '25 18:12 ullanar

@lixmal Please find two connection attempts, one after the other with no changes. The first (actually 20 failed) but the last one connected successully and didn't immediately disconnect. I can reproduce this by repeatedly attempting to connect in quick succession.

❯  NB_LOG_LEVEL=debug ssh -vv  [email protected]

debug1: OpenSSH_10.0p2, OpenSSL 3.5.4 30 Sep 2025
debug1: Reading configuration data /home/michael/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf
debug1: Reading configuration data /etc/ssh/ssh_config.d/30-libvirt-ssh-proxy.conf
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug2: checking match for 'final all' host 100.78.84.244 originally 100.78.84.244
debug2: match not found
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Reading configuration data /etc/ssh/ssh_config.d/99-custom.conf
debug1: /etc/ssh/ssh_config.d/99-custom.conf line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config.d/99-netbird.conf
debug1: /etc/ssh/ssh_config.d/99-netbird.conf line 8: Applying options for 100.78.84.244
debug2: checking match for 'exec "/usr/bin/netbird ssh detect %h %p"' host 100.78.84.244 originally 100.78.84.244
debug1: Executing command: '/usr/bin/netbird ssh detect 100.78.84.244 22'
2025-12-08T22:47:52Z DEBG client/ssh/detection/detection.go:82: SSH server banner: SSH-2.0-NetBird-SSH-Server-0.60.7
debug2: match not found
debug1: configuration requests final Match pass
debug2: resolve_canonicalize: hostname 100.78.84.244 is address
debug1: re-parsing configuration
debug1: Reading configuration data /home/michael/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf
debug1: Reading configuration data /etc/ssh/ssh_config.d/30-libvirt-ssh-proxy.conf
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug2: checking match for 'final all' host 100.78.84.244 originally 100.78.84.244
debug2: match found
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Reading configuration data /etc/ssh/ssh_config.d/99-custom.conf
debug1: /etc/ssh/ssh_config.d/99-custom.conf line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config.d/99-netbird.conf
debug1: /etc/ssh/ssh_config.d/99-netbird.conf line 8: Applying options for 100.78.84.244
debug2: checking match for 'exec "/usr/bin/netbird ssh detect %h %p"' host 100.78.84.244 originally 100.78.84.244
debug1: Executing command: '/usr/bin/netbird ssh detect 100.78.84.244 22'
2025-12-08T22:47:52Z DEBG client/ssh/detection/detection.go:82: SSH server banner: SSH-2.0-NetBird-SSH-Server-0.60.7
debug2: match not found
debug1: Connecting to 100.78.84.244 [100.78.84.244] port 22.
debug1: Connection established.
debug1: identity file /home/michael/.ssh/id_rsa type 0
debug1: identity file /home/michael/.ssh/id_rsa-cert type -1
debug1: identity file /home/michael/.ssh/id_ecdsa type -1
debug1: identity file /home/michael/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/michael/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/michael/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/michael/.ssh/id_ed25519 type -1
debug1: identity file /home/michael/.ssh/id_ed25519-cert type -1
debug1: identity file /home/michael/.ssh/id_ed25519_sk type -1
debug1: identity file /home/michael/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/michael/.ssh/id_xmss type -1
debug1: identity file /home/michael/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_10.0
debug1: Remote protocol version 2.0, remote software version NetBird-SSH-Server-0.60.7
debug1: compat_banner: no match: NetBird-SSH-Server-0.60.7
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 100.78.84.244:22 as 'root'
debug1: load_hostkeys: fopen /home/michael/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: mlkem768x25519-sha256,curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,ext-info-c,[email protected]
debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: [email protected],[email protected],aes256-ctr,[email protected],aes128-ctr
debug2: ciphers stoc: [email protected],[email protected],aes256-ctr,[email protected],aes128-ctr
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512
debug2: compression ctos: none,[email protected]
debug2: compression stoc: none,[email protected]
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: mlkem768x25519-sha256,curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,[email protected]
debug2: host key algorithms: ssh-ed25519
debug2: ciphers ctos: [email protected],[email protected],[email protected],aes128-ctr,aes192-ctr,aes256-ctr
debug2: ciphers stoc: [email protected],[email protected],[email protected],aes128-ctr,aes192-ctr,aes256-ctr
debug2: MACs ctos: [email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96
debug2: MACs stoc: [email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: mlkem768x25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: mlkem768x25519-sha256 need=32 dh_need=32
debug1: kex: mlkem768x25519-sha256 need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:SOfjLpHt8CJWZ8HIH/ZOSapTVXAckI7wa7gWnlERuok
debug1: load_hostkeys: fopen /home/michael/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '100.78.84.244' is known and matches the ED25519 host key.
debug1: Found key in /home/michael/.ssh/known_hosts:47
debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: ssh_packet_read_poll2: resetting read seqnr 3
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 4294967296 blocks
debug2: KEX algorithms: mlkem768x25519-sha256,curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,ext-info-c,[email protected]
debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: [email protected],[email protected],aes256-ctr,[email protected],aes128-ctr
debug2: ciphers stoc: [email protected],[email protected],aes256-ctr,[email protected],aes128-ctr
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512
debug2: compression ctos: none,[email protected]
debug2: compression stoc: none,[email protected]
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_ext_info_client_parse: server-sig-algs=<ssh-ed25519,[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-sha2-512,ssh-rsa,ssh-dss>
debug1: kex_ext_info_check_ver: [email protected]=<0>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
Authenticated to 100.78.84.244 ([100.78.84.244]:22) using "none".
debug1: pkcs11_del_provider: called, provider_id = (null)
debug1: channel 0: new session [client-session] (inactive timeout: 0)
debug2: channel 0: send open
debug1: Entering interactive session.
debug1: pledge: filesystem
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug1: channel 0: setting env COLORTERM = "truecolor"
debug2: channel 0: request env confirm 0
debug2: channel 0: request shell confirm 1
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 2097152 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
debug2: channel 0: rcvd close
debug2: channel 0: output open -> drain
debug2: chan_shutdown_read: channel 0: (i0 o1 sock -1 wfd 4 efd 6 [write])
debug2: channel 0: input open -> closed
debug2: channel 0: obuf empty
debug2: chan_shutdown_write: channel 0: (i3 o1 sock -1 wfd 5 efd 6 [write])
debug2: channel 0: output drain -> closed
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send_close2
debug2: channel 0: send close for remote id 0
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
Connection to 100.78.84.244 closed.
Transferred: sent 3272, received 2444 bytes, in 0.0 seconds
Bytes per second: sent 413876.2, received 309142.3
debug1: Exit status -1

~ 
❯  NB_LOG_LEVEL=debug ssh -vv  [email protected]

debug1: OpenSSH_10.0p2, OpenSSL 3.5.4 30 Sep 2025
debug1: Reading configuration data /home/michael/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf
debug1: Reading configuration data /etc/ssh/ssh_config.d/30-libvirt-ssh-proxy.conf
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug2: checking match for 'final all' host 100.78.84.244 originally 100.78.84.244
debug2: match not found
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Reading configuration data /etc/ssh/ssh_config.d/99-custom.conf
debug1: /etc/ssh/ssh_config.d/99-custom.conf line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config.d/99-netbird.conf
debug1: /etc/ssh/ssh_config.d/99-netbird.conf line 8: Applying options for 100.78.84.244
debug2: checking match for 'exec "/usr/bin/netbird ssh detect %h %p"' host 100.78.84.244 originally 100.78.84.244
debug1: Executing command: '/usr/bin/netbird ssh detect 100.78.84.244 22'
2025-12-08T22:47:52Z DEBG client/ssh/detection/detection.go:82: SSH server banner: SSH-2.0-NetBird-SSH-Server-0.60.7
debug2: match not found
debug1: configuration requests final Match pass
debug2: resolve_canonicalize: hostname 100.78.84.244 is address
debug1: re-parsing configuration
debug1: Reading configuration data /home/michael/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf
debug1: Reading configuration data /etc/ssh/ssh_config.d/30-libvirt-ssh-proxy.conf
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug2: checking match for 'final all' host 100.78.84.244 originally 100.78.84.244
debug2: match found
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Reading configuration data /etc/ssh/ssh_config.d/99-custom.conf
debug1: /etc/ssh/ssh_config.d/99-custom.conf line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config.d/99-netbird.conf
debug1: /etc/ssh/ssh_config.d/99-netbird.conf line 8: Applying options for 100.78.84.244
debug2: checking match for 'exec "/usr/bin/netbird ssh detect %h %p"' host 100.78.84.244 originally 100.78.84.244
debug1: Executing command: '/usr/bin/netbird ssh detect 100.78.84.244 22'
2025-12-08T22:47:52Z DEBG client/ssh/detection/detection.go:82: SSH server banner: SSH-2.0-NetBird-SSH-Server-0.60.7
debug2: match not found
debug1: Connecting to 100.78.84.244 [100.78.84.244] port 22.
debug1: Connection established.
debug1: identity file /home/michael/.ssh/id_rsa type 0
debug1: identity file /home/michael/.ssh/id_rsa-cert type -1
debug1: identity file /home/michael/.ssh/id_ecdsa type -1
debug1: identity file /home/michael/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/michael/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/michael/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/michael/.ssh/id_ed25519 type -1
debug1: identity file /home/michael/.ssh/id_ed25519-cert type -1
debug1: identity file /home/michael/.ssh/id_ed25519_sk type -1
debug1: identity file /home/michael/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/michael/.ssh/id_xmss type -1
debug1: identity file /home/michael/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_10.0
debug1: Remote protocol version 2.0, remote software version NetBird-SSH-Server-0.60.7
debug1: compat_banner: no match: NetBird-SSH-Server-0.60.7
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 100.78.84.244:22 as 'root'
debug1: load_hostkeys: fopen /home/michael/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: mlkem768x25519-sha256,curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,ext-info-c,[email protected]
debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: [email protected],[email protected],aes256-ctr,[email protected],aes128-ctr
debug2: ciphers stoc: [email protected],[email protected],aes256-ctr,[email protected],aes128-ctr
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512
debug2: compression ctos: none,[email protected]
debug2: compression stoc: none,[email protected]
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: mlkem768x25519-sha256,curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,[email protected]
debug2: host key algorithms: ssh-ed25519
debug2: ciphers ctos: [email protected],[email protected],[email protected],aes128-ctr,aes192-ctr,aes256-ctr
debug2: ciphers stoc: [email protected],[email protected],[email protected],aes128-ctr,aes192-ctr,aes256-ctr
debug2: MACs ctos: [email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96
debug2: MACs stoc: [email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: mlkem768x25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: mlkem768x25519-sha256 need=32 dh_need=32
debug1: kex: mlkem768x25519-sha256 need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:SOfjLpHt8CJWZ8HIH/ZOSapTVXAckI7wa7gWnlERuok
debug1: load_hostkeys: fopen /home/michael/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '100.78.84.244' is known and matches the ED25519 host key.
debug1: Found key in /home/michael/.ssh/known_hosts:47
debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: ssh_packet_read_poll2: resetting read seqnr 3
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 4294967296 blocks
debug2: KEX algorithms: mlkem768x25519-sha256,curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,ext-info-c,[email protected]
debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: [email protected],[email protected],aes256-ctr,[email protected],aes128-ctr
debug2: ciphers stoc: [email protected],[email protected],aes256-ctr,[email protected],aes128-ctr
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512
debug2: compression ctos: none,[email protected]
debug2: compression stoc: none,[email protected]
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_ext_info_client_parse: server-sig-algs=<ssh-ed25519,[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-sha2-512,ssh-rsa,ssh-dss>
debug1: kex_ext_info_check_ver: [email protected]=<0>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
Authenticated to 100.78.84.244 ([100.78.84.244]:22) using "none".
debug1: pkcs11_del_provider: called, provider_id = (null)
debug1: channel 0: new session [client-session] (inactive timeout: 0)
debug2: channel 0: send open
debug1: Entering interactive session.
debug1: pledge: filesystem
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug1: channel 0: setting env COLORTERM = "truecolor"
debug2: channel 0: request env confirm 0
debug2: channel 0: request shell confirm 1
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 2097152 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0

Wrote a little loop to test it

for i in {1..100}; do
    ssh [email protected] && break
    echo "Attempt $i failed"
    sleep 0.2
done

... Attempt 70 failed Connection to 100.78.84.244 closed. Attempt 71 failed Connection to 100.78.84.244 closed. Attempt 72 failed Connection to 100.78.84.244 closed. Attempt 73 failed root@pve:~#

Once connected, cecking the client.log

2025-12-08T23:01:43Z INFO client/ssh/server/server.go:607: SSH connection from NetBird peer 100.78.8.249:50302 allowed
2025-12-08T23:01:43Z INFO client/ssh/server/server.go:607: SSH connection from NetBird peer 100.78.8.249:50306 allowed
2025-12-08T23:01:43Z INFO client/ssh/server/server.go:607: SSH connection from NetBird peer 100.78.8.249:50320 allowed
2025-12-08T23:01:43Z INFO [session: [email protected]:50320-9468067b] client/ssh/server/session_handlers.go:35: SSH session started
2025-12-08T23:01:43Z INFO [session: [email protected]:50320-9468067b] client/ssh/server/command_execution_unix.go:147: starting interactive shell: /usr/bin/login
2025-12-08T23:01:43Z INFO [session: [email protected]:50320-9468067b] client/ssh/server/session_handlers.go:45: SSH session closed after 56ms
2025-12-08T23:01:43Z INFO client/ssh/server/server.go:607: SSH connection from NetBird peer 100.78.8.249:50328 allowed
2025-12-08T23:01:43Z INFO client/ssh/server/server.go:607: SSH connection from NetBird peer 100.78.8.249:50334 allowed
2025-12-08T23:01:43Z INFO client/ssh/server/server.go:607: SSH connection from NetBird peer 100.78.8.249:50346 allowed
2025-12-08T23:01:43Z INFO client/internal/peer/guard/sr_watcher.go:105: reconnected to Signal or Relay server
2025-12-08T23:01:43Z INFO shared/signal/client/grpc.go:159: connected to the Signal Service stream
2025-12-08T23:01:43Z INFO [session: [email protected]:50346-89b93a7d] client/ssh/server/session_handlers.go:35: SSH session started
2025-12-08T23:01:43Z INFO [session: [email protected]:50346-89b93a7d] client/ssh/server/command_execution_unix.go:147: starting interactive shell: /usr/bin/login
2025-12-08T23:01:43Z INFO [session: [email protected]:50346-89b93a7d] client/ssh/server/session_handlers.go:45: SSH session closed after 56ms
2025-12-08T23:01:44Z INFO client/ssh/server/server.go:607: SSH connection from NetBird peer 100.78.8.249:50350 allowed
2025-12-08T23:01:44Z INFO client/ssh/server/server.go:607: SSH connection from NetBird peer 100.78.8.249:50366 allowed
2025-12-08T23:01:44Z INFO client/ssh/server/server.go:607: SSH connection from NetBird peer 100.78.8.249:50376 allowed
2025-12-08T23:01:44Z INFO [session: [email protected]:50376-40d39722] client/ssh/server/session_handlers.go:35: SSH session started
2025-12-08T23:01:44Z INFO [session: [email protected]:50376-40d39722] client/ssh/server/command_execution_unix.go:147: starting interactive shell: /usr/bin/login

turnah avatar Dec 08 '25 22:12 turnah

Following up on this, with v0.60.8, I no longer have this issue (password authentication remains disabled). I think the issue I was experiencing was actually due to #4900. I'm interested to see if anyone else is still facing these problems?

alexmoras avatar Dec 18 '25 20:12 alexmoras