openssh-portable Proposal for hardening agent forwarding

Mailing list message: https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-March/039209.html

This is an attempt to add some features to improve the security of ssh-agent forwarding. I know that this feature is long-maligned, but it is still useful and it's a shame that it's so difficult to harden.

My use case in brief:

I have a number of ssh identities tied to different roles. For instance one identity might be needed to push changes to a particular git server, another to manage a remote server via ansible, etc.
Because I need to make many requests with these keys, typing a passphrase each time is not realistic. Therefore, these keys are parked with ssh-agent on my local machine.
In order to minimize risk to said machine, I want to do some development work elsewhere (VMs, remote machines, EC2 instances, etc)
I certainly don't want to put a passphrase-less ssh key for my git repos on these machines which I may not physically control. I just want to temporarily grant them permission to make git requests on my behalf while I'm logged in. Therefore agent-forwarding is the closest existing thing to what I need.

As anyone familiar with how agent-forwarding works will know, there is a serious weakness here: since the forwarded ssh-agent socket is treated the same as the local socket, it has access to all of the same functionality. In particular, if the remote machine were compromised (even just as my local user) then for as long as the session lasts the attacker could sign authentication requests with any of my identities, not just the one I used to connect to the compromised host. So I wanted to temporarily grant the remote host with the ability to make password-less connections to a git repo, but now they also can do everything my ansible key can!

These are some openssh changes I just coded up trying to address this problem for my own use. However, I think that my needs aren't particularly unique so I thought I'd clean up it up a bit and see some of it would be interesting to the upstream.

Prior Art

There is the ssh-agent-filter project from Timo Weingärtner (@tiwe-de) which is available in Debian: https://git.tiwe.de/ssh-agent-filter.git This requires wrapping the ssh program itself with something that can set $SSH_AUTH_SOCK
Carlo Contavalli (@ccontavalli) has the ssh-ident python script which can automate the running of segregated ssh-agent processes. Again, requires wrapping the ssh program.
Late in 2019, ForwardAgent was extended to allow specifying an alternate socket path (40be78f503277bd91c958fa25ea9ef918a2ffd3d) This opens up the possibility of running multiple segregated ssh-agent binaries. However, this is awkward at best -- doubly so if you want to continue to use something like OS/X's modified ssh-agent as your root identity store.
Damien Miller (@djmdjm) has a proposal where ssh-agent listens on two UNIX sockets for forwarding -- $SSH_AUTH_SOCK_LOCAL that knows aboout all identities while the original $SSH_AUTH_SOCK could be more limited and safer to forward. (see https://marc.info/?l=openssh-unix-dev&m=160181549101461) I see a couple disadvantages to this method:
- Requires ssh-agent changes so, again, it can't be immediately used with an existing vendor-supplied ssh-agent binary
- The classification into a simply binary of "local" vs "non-local" identities probably addresses many users' immediate needs... but it is unnecessarily limiting. For instance a user might have ssh access to two external organizations; so they might want a "local identity", "school identity", and "work identity"
- Lacks a way to apply identity filtering at an intermediate hop. i.e. if you ssh -A into a host and have multiple identities visible via that forwarded agent there isn't a way for that host to filter the identities before ssh -A to another hop.

This patch

I was originally looking into using the ssh-agent-filter tool mentioned above, but I wondered how difficult it would be to integrate that functionality more cleanly with the ssh binary itself. It turns out that it's not too bad:

The channels.c API already has channel_register_filter() which allows the programmer to plug into a channel's data stream and do (some) modifications. As I describe in the agentfilter.c comments, the API doesn't make this particularly easy for our use case but it can be done without too much pain.
The amount of code is also kept fairly small. authfd.c already has all of the code to parse messages from both directions of the agent's protocol. Not only does this mean a lot less code needs to be written, but it should be simple to keep it up-to-date with any ssh-agent changes.

Theory of operation

When ssh has an agent socket available it takes a few steps:

It sends a request to the agent (SSH2_AGENTC_REQUEST_IDENTITIES)
ssh-agent a reply with zero or more identities that it knows how to use (SSH2_AGENT_IDENTITIES_ANSWER)
The ssh client then can try those in turn by asking ssh-agent to sign an auth request using a particular identity's key (SSH2_AGENTC_SIGN_REQUEST)

This code (like ssh-agent-filter) inserts itself in the middle of this conversation and does two things:

When ssh-agent provides its SSH2_AGENT_IDENTITIES_ANSWER reply, we intercept it and optionally filter out some identities. In other words, if you run ssh-key -L on the remote host you may only see a subset of the identities you'd get if you ran that command locally.
When the remote side issues the SSH2_AGENTC_SIGN_REQUEST request we verify that the key used is one that we'd returned in the SSH2_AGENT_IDENTITIES_ANSWER response earlier. In other words, if they try to issue a signing request using a key we didn't tell them about, the request is blocked.

In addition, ssh-agent has other options for adding and removing keys, and for locking and unlocking the agent (ssh-add -x/ssh-add -X) I suspect that there is little use for allowing this functionality across the forwarded agent connection. Therefore, when agentfilter.c is enabled these types of requests are blocked by default. However, I did add configuration settings so they can be explicitly enabled if need be.

The likely-controversial bits

As described above, we decide on which keys that we'll allow SSH2_AGENTC_SIGN_REQUEST to use purely by watching which ones get returned by SSH2_AGENT_IDENTITIES_ANSWER (after we apply our filtering, of course) This is done on a per-channel basis. This means that we are assuming that ssh will always make the SSH2_AGENTC_SIGN_REQUEST on the same authfd connection that it used to make an earlier SSH2_AGENTC_REQUEST_IDENTITIES. If it were to use two seperate connections to the UNIX domain socket, it would fail. Also if it were to separately cache the ssh key ID and use SSH2_AGENTC_SIGN_REQUEST directly it would also fail. Neither of these seem to be a problem for the openssh CLI client, but it's possible that there exist other ssh-agent clients that would have an issue.
The identity data that ssh-agent holds (and is visible via ssh-add -L) consists of just two things: the key itself and the comment that the key was generated with. In order to decide which identities to let pass there is the new ForwardAgentFilterIdentitiesByComment directive. This may cause some offense since key comments are traditionally just that: purely human-readable comments.

Normally when we ask the user to specify a key we do so by filename, but the ssh-agent protocol doesn't give us that information. We also can't always look at the files ourselves to get their signatures -- for instance in the multi-hop agent forwarding cases discussed above the machine running ssh might not even have the keys local.

On the other hand, the "normal" way that people put comments in keys is likely to be amenable to their use as filters. For one thing, they're what you see with ssh-add -L when you look at what keys are available. When generating a key for use with a particular external organization it's likely that a comment was attached, so you can simply do things like.

$ ssh -A -o 'ForwardAgentFilterIdentitiesByComment *@github.com' build-host.example.com ssh pull

or:

$ ssh -A -o 'ForwardAgentFilterIdentitiesByComment !*ansible*' semi-trusted-vm.local

(also if the key doesn't have a useful comment that can always be fixed via ssh-keygen -C)

Note that @tiwe-de's ssh-agent-filter tool also supports specifying specific keys via their signature. I personally don't think that's likely to be useful to most people and it's a bit complicated to add. However it could be added later if really needed.

It would be nice if this had a feature to auto-add any identity that was used to authentication the connection itself. However that information isn't available post-authentication it seems, so it looked to me like that would probably require a lot of invasive changes.

Mar 12 '21 06:03 mitchblank

You might be interested in https://github.com/djmdjm/openssh-wip/pull/5 that does something similar, but doesn't require ssh's active inspection of agent requests. I wanted to avoid this because it would make ssh a trusted part of agent forwarding, and in the multi-hop case that's not really dependable.

Mar 12 '21 07:03 djmdjm

You might be interested in djmdjm/openssh-wip#5

OK, if there is a different approach in the works that is fine. Like I said, I originally wrote the agentfilter.c bits for my local use; I just cleaned it up a bit and added some manpage bits to make a PR out of it. If there are pieces that are interesting to you feel free to grab them, otherwise feel free to drop it entirely.

I'll try to walk through your WIP, although there is a lot there. Am I correct in understanding that it needs all of the following modified:

ssh-agent -- to track what session-id's have permission to receive which ids
local machine's ssh -- to communicate with ssh-agent what remote host is associated with what session-id
remote machine's ssh -- to communicate what session-id it is trying to communicate with

That obviously makes it a bit more complicated to use in the short-term.

Also, is there some way to put a destination-constraint on the other ssh-agent ops like add-id/remove-id/lock/unlock? Or do you think those are safe to always allow? At a minimum they make for an easy DoS by either doing REMOVE_ALL_IDENTITIES/LOCK or by just adding a lot of nonsense ids and tickling all of the O(N^2) algorithms. However I guess you could also just send millions of SIGN_REQUEST packets and get a reasonably effective DoS anyway.

it would make ssh a trusted part of agent forwarding

Well, somewhat. Since it's just blocking functionality that the unmodified ssh-agent connection would have (rather than adding any new capabilities) it didn't seem too offensive to me.

My first thought was going to make a minor protocol change where ssh would forward the agent socket connection bare except that it would first issue an extension request to tell the agent "This socket is actually forwarded from X" which would be a bit closer to what you are doing (but not require a modified ssh on the remote machine, which is a blocker for me) However I gave that up when I investigated channel_register_filter()'s capabilities and found a simple filter would be easier to implement. This also let me avoid any need for a modified ssh-agent, which I found attractive.

and in the multi-hop case that's not really dependable.

I guess I'll have to look at your proposal some more. I agree that by only doing protocol-level filtering we can't bind a key as HostA>HostB but only HostA>* (i.e. once we forward the filtered socket to HostA we no longer have any control over it) However am I understaning that for your multi-hop forwarding the local machine would have to have known_hosts for all of the multi-hop destinations? i.e. if I do ssh -A hostA ssh -A hostB then the local machine would need to be able to verify somehow that the second hop actually did contact "hostB"? Otherwise you're back to trusting the ssh binary on the intermediate host.

Mar 12 '21 08:03 mitchblank

Hallo Mitchell Blank Jr,

12.03.21 08:00 Mitchell Blank Jr:

Prior Art

There is the ssh-agent-filter project from Timo Weingärtner @.***) which is available in Debian: https://git.tiwe.de/ssh-agent-filter.git This requires wrapping the ssh program itself with something that can set $SSH_AUTH_SOCK

afssh is that wrapper and it is included in the ssh-agent-filter package.

My most-often-used invocation is: afssh -i id_example -- -t jumphost.example.org apt-dater

Late in 2019, ForwardAgent was extended to allow specifying an alternate socket path (40be78f503277bd91c958fa25ea9ef918a2ffd3d) This opens up the possibility of running multiple segregated ssh-agent binaries.

That looks cool. I guess I could change afssh to check for a recent-enough version of ssh and then supply -o ForwardAgent=$NEW_SOCKET -o IdentityAgent=$ORIGINAL_SOCKET to allow using all the keys for the initial connection.

However, this is awkward at best -- doubly so if you want to continue to use something like OS/X's modified ssh-agent as your root identity store.

It's the same with GNU/Linux's equivalent of OS/X: GNOME. There is a slow but constant flow of bug reports regarding ssh-add actually complaining about problems in gnome-keyring…

Grüße Timo

Mar 12 '21 08:03 tiwe-de