Support retry w/ backoff for artifact downloads

[lox]

The agent should support retry with backoff for artifact downloads to avoid network or rate limiting issues.

[s01ipsist]

We're seeing this particular error regularly. The artifact is not found and the build fails, even though it is definitely there. Generally a manual build-retry will get it working but obviously this isn't ideal.

The steps are

steps:
  - label: ":hammer: Test"
    plugins:
      docker-compose:
        run: app
        config:
          - docker-compose.yml
          - docker-compose.test.yml
    command: .buildkite/scripts/tests.sh
    agents:
      queue: docker-heavy
    artifact_paths: "coverage/.resultset.json"

  - wait
  - label: ":codeclimate: Report coverage"
    command:
      - ".buildkite/report-coverage"
    env:
      CC_TEST_REPORTER_ID:
    agents:
      queue: docker-heavy

with this command in report-coverage buildkite-agent artifact download "coverage/.resultset.json" ./

all running in the elastic-ci stack v2.3.0, with agent version v3.0-beta.34

[lox]

Sorry you're having issues @s01ipsist. We'll see what we can do.

[williamblevins]

Not sure if this is explicitly related, but I have a similar issue that happens frequently.

2021-08-17 21:00:03 INFO   Found 1 artifacts. Starting to download to: <redacted>
--
# Received cancellation signal, interrupting
Terminated
🚨 Error: The command was interrupted by a signal
2021-08-17 21:20:14 DEBUG  Terminating bootstrap after cancellation with terminated

This seems like the same issue. Is there any way to change the https client settings that the buildkite-agent uses? Set retry count or include a retry strategy such as exponential backoff.

[keithduncan]

Hi @williamblevins I’m not sure what issue you’re describing here, would you be able to get in touch with Buildkite Support to provide more details and we can help you there?

[toote]

I was working on an artifact plugin request to implement this and found that this is already implemented https://github.com/buildkite/agent/blob/00f2048f77c0e2d6808d1f234b339150569b7279/agent/download.go#L58-L59

[moskyb]

thanks for pinging this @toote! will close ^_^