aws-codebuild-docker-images icon indicating copy to clipboard operation
aws-codebuild-docker-images copied to clipboard

CodeBuild crashes on `PROVISIONING` phase with Fedora42 (Adams) but not Fedora42 (Rawhide) or Fedora40

Open BwL1289 opened this issue 6 months ago • 4 comments

A custom Docker image based on Fedora 42 (GA, Adams release) causes AWS CodeBuild (ARM64) to fail during the PROVISIONING phase with the error:

SINGLE_BUILD_CONTAINER_DEAD: Build container found dead before completing the build. Build container died because it was out of memory, or the Docker image is not supported.

This image works when built and run locally using Docker Desktop (ARM64/M4) with docker run and docker buildx. There are no proxy-related environment variables set (see below).

To Reproduce: Steps to reproduce the behavior: 1. Create a CodeBuild project using this image: fedora:42@sha256:dab5a4cfc24901a7bd1920c841d6f4b94b18f26d909a9a4891c7b9393ac02c25 (Fedora 42 GA “Adams”) 2. Set compute type to 8 vCPUs, 16 GiB memory (ARM64) 3. Enable privilegedMode: true in the environment configuration 4. Push a build and observe failure during the provisioning phase (no buildspec.yml is executed) since it fails on PROVISIONING phase

Expected behavior: The image should provision and execute the build like it does locally. Fedora 42 GA is a stable release and should behave consistently across environments if no proxy or networking issues are present.

What's strange is that fedora rawhide or fedora40 does not crash (see below).

Logs: Error message from CodeBuild UI:

SINGLE_BUILD_CONTAINER_DEAD: Build container found dead before completing the build. Build container died because it was out of memory, or the Docker image is not supported. ErrorCode: 500

Inside the image when tested locally:

[root@container /]# cat /etc/os-release
NAME="Fedora Linux"
VERSION="42 (Container Image)"
ID=fedora
VERSION_ID=42
CPE_NAME="cpe:/o:fedoraproject:fedora:42"
PRETTY_NAME="Fedora Linux 42 (Container Image)"

Proxy variables are not set:

$ echo $HTTP_PROXY

$ echo $HTTPS_PROXY

$ echo $http_proxy

$ echo $https_proxy

$ echo $NO_PROXY

$ echo $no_proxy

Platform (please complete the following information): • OS: Linux ARM64 • Docker image: fedora:42@sha256:dab5a4cfc24901a7bd1920c841d6f4b94b18f26d909a9a4891c7b9393ac02c25 • Privileged mode: true • Local Docker version (working): Docker 27.5.1, containerd v1.7.25 • Compute: 8 vCPUs, 16 GiB memory

Additional context • The exact same build process works locally with ARM64 Docker. • The exact same buildspec.yml works when using a Fedora42 Rawhide image (fedora:42:sha256:63a62239306df94a1a3bdd6b60acc926651d5b05bd98b6a2667e2a29fb6632fa), CodeBuild does not fail, but that image includes Python 3.14, which breaks compatibility with many wheels on PyPI. • The exact same buildspec.yml works when using Fedora40 (fedora:40@sha256:42ac07cdb7ad84f5c5668db6a2637a56677de9b82fba5300ff9c708f277ec41b). This is the workaround I am currently using, but I'd prefer to use Fedora42 since all docker images being built in the pipeline are using Fedora42. • This image works locally with --platform linux/arm64 and starts dockerd successfully. • The container likely crashes before the build phases are started, as no logs from buildspec.yml are emitted. • No proxy configuration or custom entrypoints are being used. • Suggestion: Fedora 42 GA may be incompatible with Firecracker microVM constraints, seccomp profiles, or default device mounts in CodeBuild.

Please advise whether this image can be supported or if there’s a known limitation when using Fedora 42 in CodeBuild ARM64 environments.

BwL1289 avatar Jun 13 '25 20:06 BwL1289

I can confirm CodeBuild also works with "fedora:40@sha256:42ac07cdb7ad84f5c5668db6a2637a56677de9b82fba5300ff9c708f277ec41b". There seems to be something broken with Fedora42 and CodeBuild.

BwL1289 avatar Jun 13 '25 21:06 BwL1289

any progress on this issue?

furiousdavid avatar Jun 16 '25 23:06 furiousdavid

Similar issues with an image built using Nix.

Have you tried switching to lambda execution environment? In my case I get another error that wget/curl is missing (even though its there)

Quiark avatar Jun 19 '25 04:06 Quiark

I have not (can't use lambda execution env for our use case).

This needs to get fixed. Please thumbs up the issue so AWS prioritizes.

BwL1289 avatar Jun 19 '25 04:06 BwL1289

CC @leoherran-aws @xinyu-aws

BwL1289 avatar Jul 03 '25 18:07 BwL1289

Closing this. For context, I updated the image sha. Maybe dockerhub throttling?

BwL1289 avatar Jul 08 '25 04:07 BwL1289