ssh-keyscan options do not apply to plugin checkout

[zsims]

By default the buildkite agent will do an ssh-keyscan at checkout time.

However, plugins are checked out before the buildkite agent does a keyscan.

Example pipeline:

steps:
  - command: echo hello
    plugins:
      "ssh://github.com/buildkite-plugins/docker-buildkite-plugin": bash

Example output on 3.8:

upload step:

> Preparing working directory
cd /var/lib/buildkite-agent/builds/foo/bar
# Host "github.com" already in list of known hosts at "/var/lib/buildkite-agent/.ssh/known_hosts"

next step:

> Setting up plugins

Plugin "github.com/buildkite-plugins/docker-buildkite-plugin" will be checked out to "/var/lib/buildkite-agent/plugins/github-com-buildkite-plugins-docker-buildkite-plugin"
The authenticity of host 'github.com (52.74.223.119)' can't be established.
# snip
Are you sure you want to continue connecting (yes/no)? # Received cancellation signal

Note this will only happen on "fresh agents" as buildkite checkout will write to /var/lib/buildkite-agent/.ssh/known_hosts

[lox]

Good point! We'll fix.

[hades-6]

Was this fixed?

[ticky]

Having just had a look, I can’t actually figure out how this would have been an issue at the time; plugins have had an ssh-keyscan done on them since 2017 or earlier, and in every post-beta build of the 3.x agent line. I’ll loop @lox in and see if he knows any more about what’s happened here.

[hades-6]

Thanks @ticky; I figured out that there is an option to enable the ssh config in the plugin; which works for me.

plugins:
      - docker#v3.7.0:
          image: "ruby:2.5"
          mount-ssh-agent: true

[ticky]

Ah, you’re checking stuff out from inside docker? I think that’s a separate issue. In any case, I’m glad you figured it out! :)

[toote]

I was able to reproduce this issue quite reliably. The trick is to do so on a pristine agent that has never talked to github before. The setup of the environment, including plugin cloning, is done before the checkout phase, so the keyscan has not happened yet.

You can try executing the following pipeline:

steps:
  - label: test
    command: echo 'OK'
    plugins:
      - ssh+git://[email protected]/buildkite-plugins/detect-clowns-buildkite-plugin.git#master: {}

Start the agent:

docker run -ti --rm --name buildkite-agent-test-904 -e SSH_AUTH_SOCK="/var/ssh-agent" -v "${SSH_AUTH_SOCK}":/var/ssh-agent -e BUILDKITE_PLUGINS_ALWAYS_CLONE_FRESH=true -e BUILDKITE_AGENT_TOKEN="...REDACTED..." buildkite/agent:3.44.0

Result:

But if you, without stopping the container running the agent, do a simple: docker exec -ti buildkite-agent-test-904 ssh [email protected] and retry the step, it works no problem:

If you stop the container and start it again, you go back to having the issue until something connects to github successfully (like a step that doesn't have any plugins referenced with SSH).

PS: this is basically what was originally reported, but the example was using the docker plugin who took all the blame :)