Bazel CI is flaky due to cloning buildkite docker plugin

[meteorcloudy]

Bazel Auto Sheriff often reports flaky jobs failing like:

$ git clone -v -- https://github.com/buildkite-plugins/docker-buildkite-plugin .
Cloning into '.'...
fatal: unable to access 'https://github.com/buildkite-plugins/docker-buildkite-plugin/': Failed to connect to github.com port 443: Connection timed out

https://buildkite.com/bazel/bazel-auto-sheriff-face-with-cowboy-hat/builds/90

Is there a way we could cache this repo to avoid cloning from github every time?

This is also the cause for https://github.com/bazelbuild/continuous-integration/issues/963, because the flaky error will prevent updating last green commit and a rerun on the failed job won't trigger the updating step again.

/cc @philwo

[haxorz]

I'm running into this on BazelCI for an internal CL: https://buildkite.com/bazel/google-bazel-presubmit/builds/36883.

[meteorcloudy]

Yeah, our recent CI nightly failed only due to this error: https://buildkite.com/bazel/bazel-auto-sheriff-face-with-cowboy-hat/builds/216

/cc @philwo Do you have any idea on this?

[philwo]

Hmpf, I agree this is really annoying. Let's add retry support for cloning plugins to the Buildkite Agent. I'll file an issue.