Unable to re-use pulled image specified by docker-compose file

[KevinGrandon]

We have a base image which we use for several services in docker-compose.yml. This works great locally, but we haven't found a way that we can leverage this in Buildkite. We are able to pull the base image using pull, but it seems it can't find the named image that we pull down, and will build again unnecessarily.

Docker-compose file:

version: '3.1'
services:
  base_service:
    build: .
    image: myimage

  service1:
    image: myimage:latest
    depends_on:
      - base_service

  service2:
    image: myimage:latest
    depends_on:
      - base_service

Pipeline looks like:

steps:
  - name: ':docker: :package:'
    plugins:
      'docker-compose#v2.0.0':
        build: base_service
        image-repository: ...
  - wait
  - name: 'task 1'
    command: ./dosomething1
    plugins:
      'docker-compose#v2.0.0':
        run: service1
        pull:
          - base_service
  - name: 'task 2'
    command: ./dosomething2
    plugins:
      'docker-compose#v2.0.0':
        run: service2
        pull:
          - base_service

[KevinGrandon]

Ok, so I have kind of a crazy workaround that works both locally and in CI for now, but it would be great if this worked out of the box. Posting here in case it helps and if anyone else needs a workaround. Here's what we're doing:

New environment variable in the Buildkite UI:

IMAGE_NAME_PREFIX=myrepo.amazonaws.com/myimage:myimage-base_service-build-

Bash variable expansion in docker-compose that only works in CI:

image: ${IMAGE_NAME_PREFIX:-myimage:latest}${BUILDKITE_BUILD_NUMBER:-}

This is a really nasty hack though, and if we could get this working with simply image: myimage:latest that would be ideal.

[toolmantim]

Oh, interesting problem!

Hrmm, so given this:

  - name: ':docker: :package:'
    plugins:
      'docker-compose#v2.0.0':
        build: base_service
        image-repository: my.repo/image

This pushes to my.repo/image:docker-compose-plugin-${pipeline-slug}-${build-number}

And then the following run step:

  - name: 'task 1'
    command: ./dosomething1
    plugins:
      'docker-compose#v2.0.0':
        run: service1
        pull:
          - base_service

…creates a config override file that looks like this:

version: '3.1'
services:
  base_service:
    image: my.repo/image:docker-compose-plugin-${pipeline-slug}-${build-number}

And then runs:

docker-compose pull base_service

which works AOK. And then runs:

docker-compose run -f override.yml -f docker-compose.yml service1

but the definition of service1 in the docker-compose.yml file depends on myimage:latest, and not my.repo/image:docker-compose-plugin-${pipeline-slug}-${build-number}

So to make this work, we'd have to set the image for the service you're trying to run in the override file, in addition to the base_service that was pre-built, and only if the image name matches:

version: '3.1'
services:
  base_service:
    image: my.repo/image:docker-compose-plugin-${pipeline-slug}-${build-number}
  service1:
    image: my.repo/image:docker-compose-plugin-${pipeline-slug}-${build-number}

Sounds doable, I think?

[KevinGrandon]

Thanks for digging in so quickly! That definitely sounds like the right approach to me, and would help docker-compose behave "as-expected" I believe.

Would this also fix the pull: config so it would automatically know to only pull the base_image? (That's a much more minor concern, but might be nice to have. Happy to file a separate issue for that later.)

[toolmantim]

Great!

Oh hrmm, not sure I follow there? The base service, or myimage? Or do you want to file another issue with details?

[lox]

Hmmmm 🤔My initial instinct is to try and solve this with either multi-stage docker builds or the cache_from directive.

Is the base_service purely there as a hack to provide a common docker layer cache ancestry for service1 and service2 @KevinGrandon?

[toolmantim]

It does feel like a mis-use of the image and depends_on options… 🤔

[KevinGrandon]

Is the base_service purely there as a hack to provide a common docker layer cache ancestry for service1 and service2

Yes, that's essentially what we're doing. We're working in a monorepo and have so far found this to be the most performant way to build for development and also run tests in.

[toolmantim]

It might actually be too tricky to do what I suggested automatically? We don't really do config parsing at the moment.

But another way to solve this might be to allow you to configure the plugin to create the extra override?

Maybe like so:

steps:
  - name: ':docker: :package:'
    plugins:
      'docker-compose#v2.0.0':
        build: base_service
        image-repository: ...
  - wait
  - name: 'task 1'
    command: ./dosomething1
    plugins:
      'docker-compose#v2.0.0':
        run: service1
        prebuilt-image: base_service
  - name: 'task 2'
    command: ./dosomething2
    plugins:
      'docker-compose#v2.0.0':
        run: service2
        prebuilt-image: base_service

[lox]

What if we added cache_from support for run, so it was:

steps:
  - name: ':docker: :package:'
    plugins:
      'docker-compose#v2.0.0':
        build: base_service
        image-repository: ...
  - wait
  - name: 'task 1'
    command: ./dosomething1
    plugins:
      'docker-compose#v2.0.0':
        run: service1
        cache-from: base_service
  - name: 'task 2'
    command: ./dosomething2
    plugins:
      'docker-compose#v2.0.0':
        run: service2
        cache-from: base_service

[toolmantim]

Maybe you could try some things out in a fork of the plugin @KevinGrandon? Check our the readme for running the tests locally! And we’re here to help if you need a hand.

[toote]

I believe that the use of tags during builds and the override files created has taken care of this issue, haven't they?