"FATAL Failed to see if meta-data exists" when buildkite starts from CLI

[create1st]

I start Buildkite from CLI, and I get 2021-03-19 17:12:47 WARN POST http://127.0.0.1:57330/jobs/75616fce-98b8-4a8a-8f8a-21d1d714cbce/data/exists: 404 (Attempt 1/10 Retrying in 5s) since the docker-compose plugin is trying to connect to the Buildkite agent, which may be difficult for the agent started on demand. I'm not sure if this is something to be concerned about, yet this pipeline is behaving in so an unstable fashion, so I expect it to cause some underlying issues.

Command to start the pipeline

bk run

pipeline.yaml

env:
  BUILDKITE_CLEAN_CHECKOUT: true
  DEV_PROFILE: 'dev'
  DEV_PROFILE_AGENT: 'dev'
steps:
  - name: 'Validate'
    plugins:
      docker-compose#v3.6.0:
        config: ${INFRA_PATH:-.}/docker-compose.yaml
        pull: validate
        run: validate
    agents:
      queue: ${DEV_PROFILE_AGENT}
    env:
      INFRA_PATH: ${INFRA_PATH:-.}

Console output:

~~~ Running plugin docker-compose command hook
$ /var/folders/5n/dp7wwjp54kzg6m_0vf4j_q080000gn/T/buildkite-plugins-950277107/github-com-buildkite-plugins-docker-compose-buildkite-plugin-v3-6-0/hooks/command
2021-03-19 17:12:47 WARN   POST http://127.0.0.1:57330/jobs/75616fce-98b8-4a8a-8f8a-21d1d714cbce/data/exists: 404 (Attempt 1/10 Retrying in 5s)
2021-03-19 17:12:47 FATAL  Failed to see if meta-data exists: POST http://127.0.0.1:57330/jobs/75616fce-98b8-4a8a-8f8a-21d1d714cbce/data/exists: 404

[eraserhd]

I'm also getting this when the build worked yesterday. Did you find a solution?

[create1st]

No response, no solution.

[toote]

Hi @create1st and @eraserhd! Sorry for the loong delay to get back to you.

From the looks of it, the metadata request happens when building the list of prebuilt image candidates. In version 3.6.0:

https://github.com/buildkite-plugins/docker-compose-buildkite-plugin/blob/b332edb099bcac0d372e14760525158b0d8fde5e/commands/run.sh#L42

With that in mind, the only effect a failed command like this has is that it would force a docker build of the service. It shouldn't cause any issues with the build itself.

Took me a while to figure out that the logs you are mentioning come from the bk CLI and not this plugin and returning a 404 for metadata that does not exist is the expected behaviour:

https://github.com/buildkite/cli/blob/985927f83317fac810f00ad1cf69f401e581435f/local/server.go#L528