runner-images icon indicating copy to clipboard operation
runner-images copied to clipboard

intermittent error running docker compose up

Open adoroszlai opened this issue 7 months ago • 18 comments

Description

Ubuntu 24.04 runner image version 20250504.1.0 has Docker Compose 2.35.1, which is affected by https://github.com/docker/compose/issues/12747

Image version 20250427.1.0 had Docker Compose 2.27.1, which did not have the bug.

Please roll back to previous image version or upgrade Docker Compose to 2.36.0.

Platforms affected

  • [ ] Azure DevOps
  • [x] GitHub Actions - Standard Runners
  • [ ] GitHub Actions - Larger Runners

Runner images affected

  • [ ] Ubuntu 22.04
  • [x] Ubuntu 24.04
  • [ ] macOS 13
  • [ ] macOS 13 Arm64
  • [ ] macOS 14
  • [ ] macOS 14 Arm64
  • [ ] macOS 15
  • [ ] macOS 15 Arm64
  • [ ] Windows Server 2019
  • [ ] Windows Server 2022
  • [ ] Windows Server 2025

Image version and build link

20250504.1.0

https://github.com/apache/ozone/actions/runs/14856073950/job/41804480380?pr=8248#step:13:21

Is it regression?

yes

Expected behavior

docker compose up works

Actual behavior

docker compose up intermittently fails with fatal error: concurrent map writes

Repro steps

https://github.com/docker/compose/issues/12747#issuecomment-2810593465

adoroszlai avatar May 07 '25 17:05 adoroszlai

Hi @adoroszlai Thank you for bringing this issue to our attention. We will look into this issue and will update you after investigating.


RaviAkshintala avatar May 08 '25 05:05 RaviAkshintala

Hi @adoroszlai - Could you kindly proceed with the resolution steps provided by the official team? This serves as a good workaround, allowing us to avoid upgrading to the new version, and I believe this issue might be temporary. Please feel free to reach out if you have any questions

kishorekumar-anchala avatar May 09 '25 14:05 kishorekumar-anchala

@kishorekumar-anchala Can you please clarify which resolution steps you mean? I don't see any in the comments on the Docker Compose bug report.

I have worked around the bug by limiting Docker Compose parallel operations to 1 (export COMPOSE_PARALLEL_LIMIT=1). However, this is not optimal, since it needs to be set for each affected workflow by all users of Docker Compose. I think it would be much better in the long run to upgrade to a version with the fix.

adoroszlai avatar May 09 '25 17:05 adoroszlai

@kishorekumar-anchala Can you please clarify which resolution steps you mean? I don't see any in the comments on the Docker Compose bug report.

To minimize concurrent issues, kindly consider pre-pulling the images and retrying until the new version rollout is completed.

For reference, please find the comment provided by @samifruit514: https://github.com/docker/compose/issues/12747#issuecomment-2810593465

kishorekumar-anchala avatar May 12 '25 09:05 kishorekumar-anchala

Hi @adoroszlai - Could you please confirm whether the above workaround has been tried? I would appreciate it if you could verify once it's tested.

kishorekumar-anchala avatar May 14 '25 07:05 kishorekumar-anchala

@kishorekumar-anchala,

As I mentioned:

I have worked around the bug by limiting Docker Compose parallel operations to 1 (export COMPOSE_PARALLEL_LIMIT=1).

This is much simpler than separate pull with retry.

adoroszlai avatar May 14 '25 09:05 adoroszlai

Could you please confirm one last thing before we proceed? Have you tested this with the newer version mentioned above, and did it work as expected?

cc @adoroszlai

kishorekumar-anchala avatar May 14 '25 10:05 kishorekumar-anchala

Hi @adoroszlai - Please confirm ?

kishorekumar-anchala avatar May 16 '25 04:05 kishorekumar-anchala

Have you tested this with the newer version mentioned above

Not tested, but the fix is present in 2.36.0.

adoroszlai avatar May 16 '25 18:05 adoroszlai

I'm also affected. According to the docker-compose team this is verified and working.

https://github.com/docker/compose/pull/12752 - they claim to have tested it.

FredTheDino avatar May 19 '25 13:05 FredTheDino

This needs fixing on ubuntu 22.04 runners also - we are encountering this intermittently in our CI runs. I also submitted a PR #12237 to address this.

chakru-r avatar May 24 '25 06:05 chakru-r

Hi @adoroszlai -The latest version of Docker Compose has been released, including the latest image rollout. Could you kindly verify and confirm?

kishorekumar-anchala avatar May 30 '25 10:05 kishorekumar-anchala

@kishorekumar-anchala how do we verify this? Docker compose is presenting with no such service test even though it was there and working previously on ubuntu-latest.

apriendeau avatar May 30 '25 18:05 apriendeau

Hello @apriendeau - could you please elaborate the problem ?

@adoroszlai - could you please check and confirm ?

cc @FredTheDino @chakru-r

kishorekumar-anchala avatar Jun 02 '25 04:06 kishorekumar-anchala

Hi @adoroszlai - Could you please confirm if the error still persists?

kishorekumar-anchala avatar Jun 04 '25 13:06 kishorekumar-anchala

Still broken. The documentation still says it's running version 2.35.1 (https://github.com/actions/runner-images/blob/ubuntu24/20250511.1/images/ubuntu/Ubuntu2404-Readme.md).

Hopefully this gets fixed soon. This is a huge issue.

fishcharlie avatar Jun 04 '25 22:06 fishcharlie

Hi @fishcharlie - The latest version has been updated. You might be working with an older codebase, please consider forking the repository and building from there. Let me know if you need any help

https://github.com/kishorekumar-anchala/runner-images-kishore/blob/main/images/ubuntu/Ubuntu2404-Readme.md

kishorekumar-anchala avatar Jun 05 '25 05:06 kishorekumar-anchala

@kishorekumar-anchala Do you work for GitHub or are you affiliated with them? All that needs to happen here is GitHub needs to make this update on their runner and deploy it. It has nothing to do with forking the repo. That has never been GitHub's advice when images become outdated.

fishcharlie avatar Jun 05 '25 13:06 fishcharlie

Hi @fishcharlie - I understand. The deployment was successfully completed last week, and the attached file originates from that deployment.

@adoroszlai - Since we haven't received any response from your end, we are considering the issue resolved and proceeding with closure. If you have any concerns, please feel free to let us know.

kishorekumar-anchala avatar Jun 06 '25 07:06 kishorekumar-anchala