tabby Starting with 1.0.216, ssh authentication via ssh keys is broken and oddly incorrect.

Describe the problem: All tests are with current version 1.0.219.

After the upgrade (or a fresh install from 1.0.216, 217,218, and 219 it fails to send my private keys. This happens on start up (defaults to ssh into one of my hosts) or when I select a ssh profile to connect to. If I revert to 1.0.215 (pre russh2?), ssh works again.

If I create a new Profile with just the password or remove the ssh keys and set a password, login seems to proceeds normally (see next line), but as if I gave it a valid ssh key (initial failed profile has the key). These keys were generated with PuttyGEN, specifically RSA 1024 bit.

If I create a new Profile with an incorrect password or remove the ssh keys and set a password, login proceeds normally, but as if I gave it a valid ssh key (initial failed profile has the key).

Accepted publickey for user from 192.168.32.X port 64427 ssh2: RSA SHA256:cWqiyQvnXbqqh6+vq1SwPN8JV5zUVNe2PY+IYZ2la8k

If I create a new profile to the same host without a password in the same session, it will connect as if the ssh key was attached.

Accepted publickey for user from 192.168.32.X port 64427 ssh2: RSA SHA256:cWqiyQvnXbqqh6+vq1SwPN8JV5zUVNe2PY+IYZ2la8k

If I replace the hostname with an IP address it still connects as if the ssh key is attached (from initial failed profile).

If I create a new profile to a different host with no password, password is asked and I can login with password.

sshd[1591965]: Accepted password for user from 192.168.32.XXX port 51625 ssh2

It looks like ssh keys are

failing to work for profiles with them configured.
attached to a host's key and (likely incorrectly) shared between profiles for successful login if not directly attached.

To Reproduce: Start a ssh connection via Profile & Connections.

The following error is displayed if the ssh key is attached to the profile:

 SSH  Connecting to home.lifergeek.com
 SSH   !  Could not load private key file:///home/linuxuser/.ssh/id_rsa: Error: Not found
 SSH   !  Agent auth selected, but no running Agent process is found
 SSH   !  Agent auth selected, but no running Agent process is found
 SSH  Host key fingerprint:
 SSH   ecdsa-sha2-nistp256  SHA256:O6PefDPOp+PJWzWrrYGFDaOJmS51OUEQsQu/gKx2TaM 
 SSH  Trying private key: file://C:\Users\WinUser\Putty\user_key_priv.txt
 SSH   !  Failed to load private key file://C:\Users\WinUser\Putty\user_key_priv.txt: Error: SendError
 SSH  Trying private key: file://C:\Users\WinUser\.ssh\id_rsa
 SSH   !  Failed to load private key file://C:\Users\WinUser\.ssh\id_rsa: Error: SendError
 SSH  Trying private key: file://C:\Users\WinUser\Putty\user_key_priv.ppk
 SSH   !  Failed to load private key file://C:\Users\WinUser\Putty\user_key_priv.ppk: Error: SendError

A password is asked, also fails. It attempts twice.

Jan 14 09:42:34 host sshd[1846841]: Connection closed by authenticating user user 192.168.32.X port 52436 [preauth]
Jan 14 09:42:35 host sshd[1846852]: Connection closed by authenticating user user 192.168.32.X port 52444 [preauth]

Ran something from sshd in debug mode:

$ sudo sshd -d -p 2222
debug1: sshd version OpenSSH_9.6, OpenSSL 3.0.13 30 Jan 2024
debug1: private host key #0: ssh-rsa SHA256:EM7F/hR1hau9tnoZ98bwLPaUgfMD2XYp1tuXVeRlyXs
debug1: private host key #1: ssh-dss SHA256:mmkm21Ult4B3Ob1XV3tNkY7Lk2CfzgH9ZxxuMBRlUIE
debug1: private host key #2: ecdsa-sha2-nistp256 SHA256:O6PefDPOp+PJWzWrrYGFDaOJmS51OUEQsQu/gKx2TaM
debug1: private host key #3: ssh-ed25519 SHA256:VXGWd32RBiPgdvBKZ/f/9+AEG2XyVZbqxS09cRi4w2A
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 2222 on 0.0.0.0.
Server listening on 0.0.0.0 port 2222.
debug1: Bind to port 2222 on ::.
Server listening on :: port 2222.

debug1: Server will not fork when running in debugging mode.
Connection from 192.168.32.XXX port 53352 on 192.168.32.XX port 2222 rdomain ""
debug1: Local version string SSH-2.0-OpenSSH_9.6p1 Ubuntu-3ubuntu13.5
debug1: Remote protocol version 2.0, remote software version russh_0.50.0-beta.7
debug1: compat_banner: no match: russh_0.50.0-beta.7
debug1: permanently_set_uid: 111/65534 [preauth]
debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: algorithm: curve25519-sha256 [preauth]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha1 compression: none [preauth]
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha1 compression: none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: SSH2_MSG_KEX_ECDH_INIT received [preauth]
debug1: rekey out after 4294967296 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: rekey in after 4294967296 blocks [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user user service ssh-connection method publickey [preauth]
debug1: attempt 0 failures 0 [preauth]
debug1: connection from 192.168.32.XXX matched 'Address 192.168.32.0/20' at line 97
debug1: PAM: initializing for "user"
debug1: PAM: setting PAM_RHOST to "192.168.32.XXX"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth_pubkey: publickey test pkalg rsa-sha2-256 pkblob RSA SHA256:6LIgSBtC3zornz7cUyznJkvJDws9quFhHyC8h1aR++M [preauth]
debug1: temporarily_use_uid: 1000/1000 (e=0/0)
debug1: trying public key file /home/user/.ssh/authorized_keys
debug1: fd 3 clearing O_NONBLOCK
debug1: /home/user/.ssh/authorized_keys:14: matching key found: RSA SHA256:6LIgSBtC3zohnz7cUyznJkvJDws9quFhHyC8h1aR++M
debug1: /home/user/.ssh/authorized_keys:14: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
Accepted key RSA SHA256:6LIgSBtC3zohnz7cUyznJkvJDws9quFhHyC8h1aR++M found at /home/user/.ssh/authorized_keys:14
debug1: restore_uid: 0/0
Postponed publickey for wmason from 192.168.32.XXX port 53352 ssh2 [preauth]
Connection closed by authenticating user wmason 192.168.32.XXX port 53352 [preauth]
debug1: do_cleanup [preauth]
debug1: monitor_read_log: child log fd closed
debug1: do_cleanup
debug1: PAM: cleanup
debug1: Killing privsep child 1849936
debug1: audit_event: unhandled event 12

Jan 14 '25 14:01 tdp4

UPDATE: I figured it out, more or less,

I discovered the source of the authenticated keys. It is reading the keys I have configured in $HOME/.ssh (or %USERPROFILE%/.ssh). I was not expecting this, but I can see why this might be considered correct. Not sure if this is yet a documented feature.

So...

If I rename the .ssh/id_rsa key, all the working logins now fail.
If I specify a ssh key from a different location, it fails with the above error.
If I only specify the same key in my .ssh folder and restore the filename back to id_rsa, it works.

Ok, so what is different. my id_rsa is 4096 bit. My older puttygen key I was using is 1024bit. Antiquated by today's standards. I'm guessing russh doesn't support 1024bit keys since they are depreciated and out of scope. It may be a way to force lazy admins to upgrade, I feel seen.

I generated new keys with PuTTYgen (4096 bit) and everything seems to be working.

Things I think still need to be done:

Pointing out in documentation that files in .ssh will be read and used if no ssh key is explicitly given, and will override password use.
A note that 1024bit keys are definitely not supported anymore, even if they worked in 1.0.215.

Jan 14 '25 20:01 tdp4

Have the same issue, tried to generate new key, put it into .ssh/id_rsa, but that didn't help

Jan 15 '25 11:01 Livonsky

Have the same issue, tried to generate new key, put it into .ssh/id_rsa, but that didn't help

What command line did you use (with options)? Mine was to make a rsa key with 4096 bits:

ssh-keygen -t rsa -b 4096

Also make sure permissions are 0700 on .ssh and 0600 on the id_rsa file. And that the new id_rsa.pub contents is appended to the .ssh/authorized_keys file on the remote end.

Sorry if this is new-usery, I don't know your skill level and don't wanna assume.

Jan 15 '25 22:01 tdp4

I'm having the same problem with my install. 1.0.215 works fine but after upgrading I can't connect using my previously working ED25519 keys nor my RSA keys (both 1024 and 4096 bytes in size).

Installed from WinGET and Scoop using UniGetUI. Tried versions 1.0.219 and 1.0.218 from WinGET and 1.0.219 from Scoop. Reinstalled 1.0.215 from WinGET using UniGetUI and it's working perfectly again.

I'm using Windows 11 with OpenSSH installed, trying to connect to various RHEL versions if it matters.

Jan 17 '25 12:01 ple34

I can confirm this bug also seeing many SSH Keys not working anymore with versions above and including 1.0.216 -> 1.0.220 some errors with : SSH ! Failed to load private key ...: Error: SendError

Will rollback to 1.0.215

Jan 20 '25 09:01 lolomin

@tdp4 1024-bit RSA keys are actually supported - at least the ones straight out of ssh-keygen -t rsa -b 1024. I suspect that something might be different in your old key's format - would be nice if you could generate another one exactly like it, test to ensure it's not working in Tabby and post it here

Jan 22 '25 21:01 Eugeny

@tdp4 1024-bit RSA keys are actually supported - at least the ones straight out of ssh-keygen -t rsa -b 1024. I suspect that something might be different in your old key's format - would be nice if you could generate another one exactly like it, test to ensure it's not working in Tabby and post it here

Tested on Tabby 1.0.221: Failed with "Failed to load private key file://C:\Users\User.ssh\id_rsa_1024: Error: SendError"

Key created with:

# Windows Powershell.  OpenSSH_9.5p1 (9.5.2.1) for Windows
PS C:\Users\User> ssh-keygen.exe -t rsa -b 1024 -f .ssh/id_rsa_1024

I have removed the key from my use to I'm happy to post it here:

Pub

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQDdrDp5XEIydm2PJFlPhmVbNgoUGwKOurMFxBtjwKbeyCTKmwvXrINejWAIwTiuZW6If4qLBbJ/F1nKvmegN6YelAGu7R3666cWl8sPAc4xgrTspyH0z65T90q8zc2Y2m1VNVy+S6Z1LF6KpcyOlVUA0aP+xy2P8Lts2L5pMoVbFQ== User@HOST

I have also created the same key in Ubuntu 22.04.5 (WSL) with OpenSSH_8.9p1 and that key also gave an error.

Pub

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQDVOsopHJIr95qr+8E0PewNlYg5jowqi3j4nqJFxA0IY6YHvRwbQRQE7Yf6bUXQiVJp1lcgJqYOX0qL9gCDR0xZiQkYdjPJVo6dxpErgOvvFliFrFO5+NbHmdRMngTdrTCrFnUtD6fn43HZXfcQR+T4RRIdsYgUg3UaeHrGWKlBsw== User@HOST

I also generated a key the same way under Ubuntu 24.04.1 with OpenSSH_9.6p1, same issue.

This error "SendError" seems to only happen when the pub key exists on the remote side. If I remove the pub key from authorised_keys, I see no error, just asked for a password.

My OpenSSH Server version is: 1:9.6p1-3ubuntu13.5 (Ubuntu24.04.1)

Jan 29 '25 01:01 tdp4

One of the cases where this happens when using Auth authentication with multiple RSA keys in ~/.ssh and when server will abort authentication after a couple (~3) attempts.

In the current release, Tabby will try each RSA key 3 times (with decreasing hash security - sha512 -> sha256 -> sha1), and each of these counts towards the limit by OpenSSH servers.

The nightly build already works around this by supporting the server-sig-algs OpenSSH extension to only try each key once.

If this is the case for you, you could either upgrade or work around it on stable by specifying the exact key that the connection needs.

Jan 29 '25 12:01 Eugeny

Just to note that I just tried the latest release of Tabby (installed to userspace using the installer from GitHub) and I still get the same problem

Output from Tabby:

 SSH  Connecting to {{ Connection name }}
 SSH  Host key fingerprint:
 SSH   ssh-rsa  SHA256:{{ redacted }}Nw2Q 
 SSH  Trying private key: file://C:\Users\{{ username }}\.ssh\{{ ED25519 key }}
 SSH   !  Failed to load private key file://C:\Users\{{ username }}\.ssh\{{ ED25519 key }}: Error: SendError

 SSH  Connecting to {{ Connection name }}
 SSH  Host key fingerprint:
 SSH   ssh-rsa  SHA256:{{ redacted }}Nw2Q 
 X  Authentication rejected

The config file is the same config used by my working 1.0.215 install and I tried removing and re-adding the key to be sure but the issue persists

Relevant profile:

version: 7
profiles:
  - options:
      host: {{ hostname }}
      algorithms:
        cipher:
          - aes128-ctr
          - aes128-gcm
          - [email protected]
          - aes192-ctr
          - aes256-ctr
          - aes256-gcm
          - [email protected]
        kex:
          - curve25519-sha256
          - [email protected]
          - diffie-hellman-group-exchange-sha256
          - diffie-hellman-group14-sha256
          - diffie-hellman-group15-sha512
          - diffie-hellman-group16-sha512
          - diffie-hellman-group17-sha512
          - diffie-hellman-group18-sha512
          - ecdh-sha2-nistp256
          - ecdh-sha2-nistp384
          - ecdh-sha2-nistp521
        serverHostKey:
          - ecdsa-sha2-nistp256
          - ecdsa-sha2-nistp384
          - ecdsa-sha2-nistp521
          - rsa-sha2-256
          - rsa-sha2-512
          - ssh-ed25519
          - ssh-rsa
      privateKeys:
        - file://C:\Users\{{ username }}\.ssh\{{ ED25519 key }}
      scripts:
        - expect: ''
          send: ssh-agent /usr/bin/zsh
      skipBanner: true
      keepaliveInterval: 295
      input: {}
    name: {{ Connection name }}
    id: ssh:custom:{{ Connection name }}:703f3e96-717e-4b9d-82c3-1689276657cc
    icon: fas fa-code
    type: ssh

Nov 24 '25 17:11 ple34