Upgrade docker-compose (nominally to v2.5.1)

[jufemaiz]

Currently docker-compose is pinned to v1.29.2 (2021-04-13) which is now over 12 months old and is a major release behind.

https://github.com/buildkite/elastic-ci-stack-for-aws/blob/master/packer/linux/scripts/install-docker.sh#L6

Docker desktop etc have had considerable updates meaning most engineers experience in development varies considerably to the elastic ci stack.

Is there any reason for the update(s) to docker and buildx but not to docker-compose?

Describe the solution you'd like

• Update to the latest v2 of docker-compose
• Published protocol regarding how Buildkite will manage updates to the docker/buildx/docker-compose combination in the future

Describe alternatives you've considered

🤷🏼

Additional context

Latest release @ https://github.com/docker/compose/releases/tag/v2.5.1

[moskyb]

Hey @jufemaiz!

This is definitely something that we're planning on doing, but updating the major version isn't as simple as changing the number on line 6 there - the install process itself has changed, and the way you invoke docker compose has also changed (docker-compose up in v1.x, docker compose up (with a space) in v2.x).

There are workarounds available that mean we'd be able to do this without it being a breaking change to the elastic stack, but not in such a way that i can crank it out in an afternoon - we'll have to think about it a little bit, and prioritise it with other Agent and Elastic Stack work in our backlog.

[jufemaiz]

Thanks @moskyb. I realise it's a little more than a single line change, but it's probably a good one to have some formal policy/procedure in place wrt managing the releases for the future. :)

Look forward to hearing what that solution may be!

[moskyb]

Our policy so far has been "if a customer needs an updated version of an elastic stack dependency, they'll raise an issue/PR and we can prioritise that as necessary", as well as "if it becomes obvious to us that we need to update an elastic stack dependency, we'll do it, independent of customer need". For the moment at least, i don't see us moving away from those any time soon. The reality is that keeping these deps as up to date as possible is a lot of busywork for the amount of people we currently have working on Elastic Stack maintenance (basically me + 33% of @pda), given the manual nature of the upgrades.

Ideally I'd like to get to a place where something like dependabot handles this, but i don't see it happening any time soon, just because the AMI provisioning process is so manual at the moment, even through packer.

If you have further thoughts on better ways to do this though, i'd be really interested to hear them! These things are by no means set in stone, and i'd love to hear your ideas on this.

[moskyb]

For bikkies reading this: internal tracking for this issue is at PDP-42

[yob]

An interesting thing about this bump:

Docker Compose V2 is a major version bump release of Docker Compose. It has been completely rewritten from scratch in Golang (V1 was in Python).

Hopefully this means the banana pants hoops we had to jump through to install docker compose in the linux-amd64 AMI can be removed:

https://github.com/buildkite/elastic-ci-stack-for-aws/blob/6598b36b85d4c7df78a5c1567ee85693e51cbb08/packer/linux/scripts/install-docker.sh#L40-L53=

[moskyb]

Yep! I'm pretty psyched to just curl a binary instead of using the banana pants hoops, it should both simplify and speed up our provisioning process

[sj26]

If you need a workaround for the moment I think you could use something like this in your bootstrap script:

mkdir -p /usr/local/libexec/docker/cli-plugins
curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/libexec/docker/cli-plugins/docker-compose
chmod +x /usr/local/libexec/docker/cli-plugins/docker-compose

[tetienne]

@moskyb Except for the docker-compose plugin, what would be the breaking changes to this upgrade? Where docker-compose is used into the stack?

[pda]

@tetienne I believe the docker-compose plugin is the main use-case and backwards-compatibility concern, along with ad-hoc usage of docker-compose in customer's pipelines.

I'm not sure where the docker-compose plugin is at re. latest docker compose compatibility, but I think we'd need a substantial grace period between adding that support (if it's not there already) and upgrading buildkite/elastic-ci-stack-for-aws in a way that breaks the old docker-compose style.

That said, I gather there exists backwards-compatibility shims that make docker compose available as and behave like docker-compose, so there's probably a way to make the change without significant risk of breaking anything. It will just need some careful doing / testing.

[tetienne]

@pda Thx for the explanation. According to the blog post announcing the V2, the docker-compose alias is well present. So apparently there is a little risk the upgrade break something, but indeed making some tests before will be safer.

[christophermaier]

v1 and v2 of Docker Compose can coexist just fine; both could be shipped in the AMI for a transition period to allow people to migrate. I suspect the Docker Compose Buildkite plugin could be tweaked to take a new version argument whose value is either 1 or 2, defaulting to 1 initially, then down the road switching the default to 2, then finally removing the version argument altogether (all with plenty of advance communication, of course).

[jesseduffield]

One specific use case I have for this upgrade is the ability to use ssh: default inside a docker compose file (which is only supported in a recent compose version). This allows us to use the ssh agent of the host machine during a docker build step. Currently, we need to copy an SSH private key file into the image in a multi-stage build, which is cumbersome and potentially insecure.

[triarius]

Thanks for the suggestions everyone! Docker Compose v2 now lives alongside Docker Compose v1 in the elastic stack v5.12.0: https://github.com/buildkite/elastic-ci-stack-for-aws/releases/tag/v5.12.0