clients icon indicating copy to clipboard operation
clients copied to clipboard

With Bitwarden SSH Agent, SSH client queried key order appears to be inconsistent by key name or creation time.

Open Aterfax opened this issue 6 months ago • 6 comments

Steps To Reproduce

With the Bitwarden SSH Agent, when an SSH client queries the available keys, the key order appears to be inconsistent with creation date, or key naming.

A lack of consistent ordering (or the ability to make particular SSH keys available or unavailable) makes it difficult to use the correct/required SSH key with Git and/or avoid rejected authentication when you have a large number of keys.

Create 4 new SSH keys named with lexicographic ordered names - created in the following order:

  1. 000-SSHKey
  2. 001-SSHKey
  3. 002-SSHKey
  4. 003-SSHKey

Start querying the agent in sequence as you create the keys and close then reopen the Bitwarden desktop client.

Expected Result

Expectation: The keys in the agent follow FIFO, lexicographic or some discernible order when queried by an ssh client application e.g. ssh-add -L.

Realistically, this should probably be resolved by following the same ordering the desktop client gives when you list SSH keys (lexicographic?):

Image

Actual Result

When queried with ssh-add -L the observed behavior:

# Created in order:

username@machinename:~$ ssh-add -L
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICBXSQSoc1fUbtJtPnUuccvvv7zEk3XeC1xpI5yd6bN/ 000-SSHKey

username@machinename:~$ ssh-add -L
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICBXSQSoc1fUbtJtPnUuccvvv7zEk3XeC1xpI5yd6bN/ 000-SSHKey
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEb4WJHTCVVS+ZYwVCTYfqMdOHFyzpzsG/XkNswE+8HB 001-SSHKey

username@machinename:~$ ssh-add -L
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDUAjhLHQgJbSIC+GrKbgajVDquRdlQPcXfu4f5w5Atn 002-SSHKey
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICBXSQSoc1fUbtJtPnUuccvvv7zEk3XeC1xpI5yd6bN/ 000-SSHKey
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEb4WJHTCVVS+ZYwVCTYfqMdOHFyzpzsG/XkNswE+8HB 001-SSHKey

username@machinename:~$ ssh-add -L
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDUAjhLHQgJbSIC+GrKbgajVDquRdlQPcXfu4f5w5Atn 002-SSHKey
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN/S5wkMqlM7zpcv+qO7KYeuXVKmc52KdRTm8OIt5Geh 003-SSHKey
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEb4WJHTCVVS+ZYwVCTYfqMdOHFyzpzsG/XkNswE+8HB 001-SSHKey
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICBXSQSoc1fUbtJtPnUuccvvv7zEk3XeC1xpI5yd6bN/ 000-SSHKey

# Desktop client now closed and reopened:

username@machinename:~$ ssh-add -L
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDUAjhLHQgJbSIC+GrKbgajVDquRdlQPcXfu4f5w5Atn 002-SSHKey
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICBXSQSoc1fUbtJtPnUuccvvv7zEk3XeC1xpI5yd6bN/ 000-SSHKey
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN/S5wkMqlM7zpcv+qO7KYeuXVKmc52KdRTm8OIt5Geh 003-SSHKey
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEb4WJHTCVVS+ZYwVCTYfqMdOHFyzpzsG/XkNswE+8HB 001-SSHKey

Screenshots or Videos

N/A

Additional Context

N/A

Operating System

Windows

Operating System Version

Windows 11 Home 23H2

Installation method

Microsoft Store

Build Version

2025.5.0

Issue Tracking Info

  • [x] I understand that work is tracked outside of GitHub. A PR will be linked to this issue should one be opened to address it, but Bitwarden doesn't use fields like "assigned", "milestone", or "project" to track progress.

Aterfax avatar Jun 02 '25 21:06 Aterfax

Thank you for reporting this issue! We've added this to our internal tracking system. ID: PM-22315

bitwarden-bot avatar Jun 02 '25 21:06 bitwarden-bot

Steps To Reproduce

With the Bitwarden SSH Agent, when an SSH client queries the available keys, the key order appears to be inconsistent with creation date, or key naming.

A lack of consistent ordering (or the ability to make particular SSH keys available or unavailable) makes it difficult to use the correct/required SSH key with Git and/or avoid rejected authentication when you have a large number of keys.

[...]

You can easily make OpenSSH ssh prefer a certain key for a host (even though this is not really well documented; or if it is, I haven't found it, yet):

  • For every key pair, save the public key into a .pub file in ~/.ssh (or the corresponding location on another OS):
    > ls ~/.ssh
    000-SSHKey.pub
    001-SSHKey.pub
    002-SSHKey.pub
    003-SSHKey.pub
    
    Note that you need to restrict these to 600, i.e., only accessible for your user, otherwise ssh will complain, see below.
    • Sidenote, I use chezmoi and a template to get the public key from Bitwarden and write it to a file. But the keys do not change often so this might be overkill if you only need the public keys on a single machine. Because I used chezmoi as a dotfile manager anyway, this is a convenient way to get all the public keys into my home on all my machines:
      > cat .local/share/chezmoi/private_dot_ssh/private_000-SSHKey.pub.tmpl # filename changed to match original report
      {{ (bitwarden "item" "e2c46688-89ed-4baf-86ad-0917fd70f1ab").sshKey.publicKey }}
      
      (The 'private_' prefix restricts the file to 600 permissions as mentioned.)
  • Add a section to your ~/.ssh/config for each server you need to connect to. I am using sections for github.com and gitlab.com as an example, which you may already have to set the username for these servers:
    > cat ~/.ssh/config
    Host github.com
            User git
            IdentityFile ~/.ssh/000-SSHKey.pub
    
    Host gitlab.com
            User git
            IdentityFile ~/.ssh/001-SSHKey.pub
    
  • Because we provide the public key to the 'IdentityFile' option, which would normally expect a private key, the ssh process will complain if the permissions on the file are too open (even though it is actually a public key and there is no such thing as "too open" for public keys). That's why it needs to have 600 permissions.
  • Now connect to github.com and observe in the verbose output how the client offers the public key from the IdentityFile option as the first possible authentication, which the server accepts, and how the client then uses that identity from the agent:
    > ssh -v github.com
    [...]
    debug1: get_agent_identities: agent returned 5 keys # **Note** that's just how many key pairs I have in Bitwarden at the moment
    debug1: Will attempt key: /home/l-c-g/.ssh/000-SSHKey.pub ED25519 SHA256:[redacted] explicit agent
    debug1: Will attempt key: [redacted] ED25519 SHA256:[redacted] agent
    [... enumerating more keys from my agent ...]
    debug1: Offering public key: /home/l-c-g/.ssh/000-SSHKey.pub ED25519 SHA256:[redacted] explicit agent
    debug1: Server accepts key: /home/l-c-g/.ssh/000-SSHKey.pub ED25519 SHA256:[redacted] explicit agent explicit agent
    sign_and_send_pubkey: signing failed for ED25519 "/home/l-c-g/.ssh/000-SSHKey.pub" from agent: agent refused operation
    [... I did refuse access in Bitwarden at this point ...]
    
    And when connecting to gitlab.com:
    > ssh -v gitlab.com
    [...]
    debug1: get_agent_identities: agent returned 5 keys # **Note** that's just how many key pairs I have in Bitwarden at the moment
    debug1: Will attempt key: /home/l-c-g/.ssh/001-SSHKey.pub ED25519 SHA256:[redacted] explicit agent
    debug1: Will attempt key: [redacted] ED25519 SHA256:[redacted] agent
    [... enumerating more keys from my agent ...]
    debug1: Offering public key: /home/l-c-g/.ssh/001-SSHKey.pub ED25519 SHA256:[redacted] explicit agent
    debug1: Server accepts key: /home/l-c-g/.ssh/001-SSHKey.pub ED25519 SHA256:[redacted] explicit agent explicit agent
    sign_and_send_pubkey: signing failed for ED25519 "/home/l-c-g/.ssh/001-SSHKey.pub" from agent: agent refused operation
    [... I did refuse access in Bitwarden at this point ...]
    

I know this is a bug report and not a help forum but I decided to answer anyway. I think more people should know about this config option.

l-c-g avatar Jun 12 '25 11:06 l-c-g

Hi there,

Thanks for adding a comment. I had thought about doing what you've suggested with a slight amendment due to having multiple accounts and identities on a single host (GitHub). The same key order issue will arise again.

I believe the suggested bypass for this particular problem is to use hostname 'aliases' for the targeted endpoints e.g. editing your devices hosts list.

For a particular repository add a new hostname to aim at , e.g. repositoryname.github.com then use this in the SSH config.

The problem with this is that it's still super duper clunky!

It requires editing of a hosts file or DNS entities which you might not be able to do depending your environment.

You also have to maintain your own bunch of extra configuration for OpenSSH. (Arguably less annoying than DNS shenanigans).

In my mind, the best solution here is that we have the ability to toggle which keys are currently exposed by the Bitwarden agent. It seems like a missing necessary feature to me! Not only to make this issue go away but also because you might have many keys:

If you have a lot of keys, the fact the agent exposes all of them by default means you could run into authentication issues when you run out of attempts to connect if the valid key is the N+1 attempt where N is the allowed number of attempts. (I have seen this happen in the wild).

This might present an issue in corporate environments with shared SSH keys (although I'd be heavily inclined to tell them that they're using SSH keys wrong in this case).

It could potentially also rear its head if you're a freelancer of some kind interacting with multiple customer environments where you might be required to have different keys (although again, I'd be heavily inclined to tell them that they're using keys wrong).

As an aside:

It also looks like the Bitwarden SSH agent will refuse to allow you to remove keys via the ssh-add -D command so I expect trying to remove specific keys will also fail.

I gave that a very brief go and it actually looked like the Bitwarden SSH agent will break and stop working if you attempt to remove a specific key.

Aterfax avatar Jun 12 '25 13:06 Aterfax

OK, I see your use case, I am just not sure if there is a non-clunky way of exposing only a (changing) subset of keys via Bitwarden's SSH agent.

Based on my original suggestion, I think you can do what you want, without having to mess with the hosts file or DNS. If you look at the man page for ssh_config (5), the 'Host' in each block is followed by one or more patterns, separated by whitespace, which is matched against the host given to ssh. In particular, it doesn't have to be a DNS name. These patterns can be your aliases and you can then override the hostname in each block:

Host identity1-github github.com
    HostName github.com
    Username git
    IdentityFile ~/.ssh/id1_github.pub
Host identity2-github
    HostName github.com
    Username git
    IdentityFile ~/.ssh/id2_github.pub
Host identity3-github
    HostName github.com
    Username git
    IdentityFile ~/.ssh/id3_github.pub

With this configuration, you can simply do something like

> git clone identity1-github:bitwarden/clients.git

Note that I added github.com as a second pattern to the first block, just so git clone github.com:bitwarden/clients.git has a default block/identity and not some undefined behavior.

In fact, I use similar blocks to SSH into servers using a shorter name instead of the FQDN, ssh myserver is just so much more convenient than having to type out the FQDN and the options every time (again, the second pattern is there just so it also matches and sets all the options if I ever do type out the FQDN):

Host myserver myserver.mydomain.tld
    HostName myserver.mydomain.tld
    UserName ...
    IdentityFile ...
    LocalForward ...

l-c-g avatar Jun 12 '25 14:06 l-c-g

TIL that Git respects SSH configuration files in this fashion (i.e. respects the HostName, not the Host). I'd have expected Git to query identity2-github and crap out with a DNS error as that host does not exist before it attempts to use the correct key id2_github.pub.

Thanks for letting me know!

Aterfax avatar Jun 12 '25 21:06 Aterfax

Hi there,

This has been escalated for further investigation. If you have more information that can help us, please add it below.

Thanks!

rmcdowell-bitwarden avatar Jun 16 '25 07:06 rmcdowell-bitwarden

And how about just a per key toggle in the Bitwarden client for now?

minionflo avatar Jun 28 '25 23:06 minionflo