--mount ssh checks only first key in the agent

[LeKristapino]

trafficstars

I've come across an issue where the mount ssh option doesn't work if the first key in the agent does not grant access. The assumption was that it was going to go through all keys and try each of them, but apparently it doesn't happen. This can be replicated by creating a private github repo, clearing the ssh agent and adding first a key that doesn't grant access and then a key which does. If done the other way around, everything works fine.

[tonistiigi]

Are you seeing this with agent forwarding or when specifying 2 key paths? Do you see all keys when running ssh-add -L during build?

[LeKristapino]

@tonistiigi I see this with agent forwarding (using default). And yes, when debugging why my git clone command didn't work I saw the exact same keys inside the container using ssh-add -L as locally

[tonistiigi]

Does this work for you locally? I think it is just ssh-agent behavior and not specific to our forwarding.

To make sure that the correct key is used in this case you can set the IdentityFile option to the public key that you can copy into the build (or mount with secret but it doesn't need to be kept secure). If both clones are against the same host you also need to give them separate names.

~/.ssh/config

Host app1.github.com
  HostName github.com
  IdentityFile ~/.ssh/app1.pub

git clone [email protected]:repo

There might be better ways to do this, not 100% familiar with all the possible ssh-agent options.

[LeKristapino]

The reason, why I think that it might be something to do with forwarding, is that when doing things locally, everything works just fine. I don't have a ~/.ssh/config file so that shouldn't impact the different result. I was using Ubuntu 19.10 if it is of any help.

[tonistiigi]

I've included a repro that you can run that just runs ssh-agent with 2 included repo keys in ubuntu 19.10 container and seems to work exactly as you describe with only the key that is added first time to the agent working. Lmk if your setup is any different or if you see different results.

ssh-repro.tar.gz

[LeKristapino]

Seems that the ssh-agent setup is the same. I have since upgraded my PC to ubuntu 20.04 LTS so I'll try and see if I can reproduce this problem.

[tonistiigi]

You can change the ubuntu version to 20.04 in the Dockerfile and run run.sh and you'll see it makes no difference.

[LeKristapino]

Oh right.. Ok. If you need any more info, let me know.

EDIT: I re-read the issue comments, and realized that something seems off - when I use my ssh-agent locally (on Ubuntu 20.04 LTS). It scans through all the keys added, so that doesn't really match up with what you have replicated in Docker. This issue for me is only present when forwarding to docker. As I mentioned before, I don't have any extra config for ssh that could impact this.

I will try and find time to create a VM replicating the problem as it would better mirror the environment.

[rlipscombe]

I fixed this in my build by passing the correct private key as a secret, and then configuring git to use it.

docker build \
    --ssh default \
    --secret id=ssh,src=$(HOME)/.ssh/id_other \
    .

# tell git which identity we want to use
RUN git config --global core.sshcommand "ssh -i /run/secrets/ssh -o IdentitiesOnly=yes"

# mount the ssh-agent *and* the private key secret, then run 'npm install' (or whatever)
RUN --mount=type=ssh \
    --mount=type=secret,id=ssh \
    npm install

Instead of using git config, you could do @tonistiigi suggests in https://github.com/moby/buildkit/issues/1527#issuecomment-643027964 which sets it per-host (but I prefer the single RUN command).