Manifest push fails when using Gitea container registry with Docker BuildKit caching

[julmb]

Description

I am trying to use a Gitea container registry with Docker BuildKit caching (https://docs.docker.com/build/cache/). More specifically, I am trying to use the registry cache backend (https://docs.docker.com/build/cache/backends/registry/). Here, intermediate layers produced during a build are uploaded to the registry under a separate tag to be reused in subsequent builds.

I am using the following command:

docker buildx build --push --tag try.gitea.io/jules/suicune:latest --cache-to type=registry,ref=try.gitea.io/jules/suicune:build,mode=max .

This builds the image and successfully pushes it to the registry. It then starts pushing the cache layers to the registry. Pushing the layers works, but the final step of pushing the manifest fails:

#26 exporting cache to registry
#26 writing layer sha256:bab2cfbf1e9bee2409320e78b3cb9c3c2b9f2c54089bd89f42d011ad9bec699e 1.1s done
#26 writing layer sha256:c540ddb30ad1b73ba05d74a94bae43ced662b72ccca916592ab2037ae9d5cdd8
#26 writing layer sha256:c540ddb30ad1b73ba05d74a94bae43ced662b72ccca916592ab2037ae9d5cdd8 0.3s done
#26 writing layer sha256:ccb8642b226bb4d4492d41ea386268ca14457afe7eeafa621ae5dc701d39d01b
#26 writing layer sha256:ccb8642b226bb4d4492d41ea386268ca14457afe7eeafa621ae5dc701d39d01b 0.1s done
#26 writing layer sha256:dfab772f9f48504aee8bc3eed3f7f99a585e6e3d4c63d21a075521a2d796a90c
#26 writing layer sha256:dfab772f9f48504aee8bc3eed3f7f99a585e6e3d4c63d21a075521a2d796a90c 0.3s done
#26 writing layer sha256:e1a7258d27e602de318bdf87fdc38ef8c33c7c832728f727fad2a4db8b39300e
#26 writing layer sha256:e1a7258d27e602de318bdf87fdc38ef8c33c7c832728f727fad2a4db8b39300e 0.4s done
#26 writing layer sha256:e32bc7a16024c28d44887fdf36e51dea3448ffa1488fcc6f0892e49f53b0ab0c
#26 writing layer sha256:e32bc7a16024c28d44887fdf36e51dea3448ffa1488fcc6f0892e49f53b0ab0c 0.3s done
#26 writing layer sha256:ee12cf5233c9d50764aee90249dfbe32f3322a2d952963250fcf4c454858c5e3
#26 writing layer sha256:ee12cf5233c9d50764aee90249dfbe32f3322a2d952963250fcf4c454858c5e3 0.3s done
#26 writing layer sha256:f20bbea5142a083a99655706934104b8ee7664639d4c8ebc0ff403161e543ca2
#26 writing layer sha256:f20bbea5142a083a99655706934104b8ee7664639d4c8ebc0ff403161e543ca2 57.1s done
#26 ...

#29 [auth] *:: jules/suicune:pull,push token for try.gitea.io
#29 DONE 0.0s

#26 exporting cache to registry
#26 writing config sha256:daab72fe0a1a74867c48380ab59eae3d459e6c2e3b92227504bb991021adbd43
#26 writing config sha256:daab72fe0a1a74867c48380ab59eae3d459e6c2e3b92227504bb991021adbd43 1.0s done
#26 writing cache manifest sha256:f6ac364184a1cc56a1d30179846230533ec50f23e594596ac49f0f45e02b2bfc
#26 preparing build cache for export 220.1s done
#26 writing cache manifest sha256:f6ac364184a1cc56a1d30179846230533ec50f23e594596ac49f0f45e02b2bfc 0.2s done
#26 ERROR: error writing manifest blob: failed commit on ref "sha256:f6ac364184a1cc56a1d30179846230533ec50f23e594596ac49f0f45e02b2bfc": unexpected status from PUT request to https://try.gitea.io/v2/jules/suicune/manifests/build: 400 Bad Request

The Gitea logs (LEVEL = trace) show the following:

2024/01/29 08:00:27 ...s/process/manager.go:188:Add() [T] Start 65b75b1b-4: HEAD: /v2/jules/suicune/manifests/build (request)
2024/01/29 08:00:27 ...eb/routing/logger.go:47:func1() [T] router: started   HEAD /v2/jules/suicune/manifests/build for 172.25.0.1:33980
2024/01/29 08:00:27 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/jules/suicune/manifests/build for 172.25.0.1:33980, 404 Not Found in 0.9ms @ container/container.go:612(container.HeadManifest)
2024/01/29 08:00:27 ...s/process/manager.go:231:remove() [T] Done 65b75b1b-4: HEAD: /v2/jules/suicune/manifests/build
2024/01/29 08:00:27 ...s/process/manager.go:188:Add() [T] Start 65b75b1b-5: PUT: /v2/jules/suicune/manifests/build (request)
2024/01/29 08:00:27 ...eb/routing/logger.go:47:func1() [T] router: started   PUT /v2/jules/suicune/manifests/build for 172.25.0.1:33992
2024/01/29 08:00:27 ...eb/routing/logger.go:102:func1() [I] router: completed PUT /v2/jules/suicune/manifests/build for 172.25.0.1:33992, 400 Bad Request in 1.0ms @ container/container.go:527(container.UploadManifest)
2024/01/29 08:00:27 ...s/process/manager.go:231:remove() [T] Done 65b75b1b-5: PUT: /v2/jules/suicune/manifests/build

This happens both with my own instance and the one at https://try.gitea.io/. The exact same command works fine if Docker Hub is used as the registry.

I have looked at container/container.go:527 but the line numbers seem to be inaccurate so I cannot tell exactly where the 400 Bad Request comes from.

I have also come across https://forum.gitea.com/t/multiarch-docker-images-with-buildkit-and-cache-upload-fail/6213, which might be related.

Gitea Version

1.21.4

Can you reproduce the bug on the Gitea demo site?

Yes

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

I am running Gitea using the official gitea/gitea Docker image.

Database

None

[bcbnz]

I just hit the same problem trying to push Docker images from an Actions build with 1.21.5. The manifest push fails with buildx, but using the (deprecated) legacy build command without buildx installed works. This means you cannot use the docker/build-push-action in a workflow as it relies on buildx being available. The easy workaround is to call docker build and docker push:

    steps:
      - name: Check out repository
        uses: actions/checkout@v3

      - name: Log in to local registry
        uses: docker/login-action@v3
        with:
          registry: my.gitea.com
          username: ${{ gitea.repository_owner }}
          password: ${{ secrets.REGISTRY_TOKEN }} # gitea.token doesn't work for push, provide a personal access token

      - name: Build Docker image
        run: docker build -t my.gitea.com/user/img1 img1

      - name: Push Docker image
        run: docker push my.gitea.com/user/img1

[julmb]

I just hit the same problem trying to push Docker images from an Actions build with 1.21.5. The manifest push fails with buildx, but using the (deprecated) legacy build command without buildx installed works. This means you cannot use the docker/build-push-action in a workflow as it relies on buildx being available.

This is not the problem I am having. Both docker buildx build on the command line as well as the docker/build-push-action Github/Gitea action work just fine for me. The problem only occurs when using the --cache-to directive to push intermediate layers to a Gitea registry for caching.

[ItsJustRuby]

@julmb Applying the fix described in https://stackoverflow.com/a/77766002 fixed this issue for me:

The solution was to add these options to --cache-to: mode=max,image-manifest=true,oci-mediatypes=true,type=registry,ref=

[julmb]

Thank you so much for this hint! It really works!

Apparently there is a lot of information on it out there that I somehow never came across: https://github.com/moby/buildkit/issues/2251 https://github.com/moby/buildkit/pull/3724 https://gitlab.com/gitlab-org/container-registry/-/issues/407

From reading some of this, it sounds like the default behavior of BuildKit is to produce cache images that are not compliant with the OCI Image Spec. If this is the case, then I think the Gitea registry is correct to reject these. GitLab has apparently been accepting these noncompliant images due to a missing check and now that is causing issues.

The correct usage pattern here seems to be using the new image-manifest option to make BuildKit produce compliant cache images, which has solved my issue and allowed me to push cache images to the gitea registry.

Unless someone with more expertise on the subject wants to chime in, I would consider this resolved.