build-push-action Merge Docker manifests from different runners

Description

I've followed the steps here - https://docs.docker.com/build/ci/github-actions/multi-platform/ but I've been unable to get it to work for a custom registry. Its also extremely difficult to both understand and debug

adding an option in this action such as

'merge-manifests' that replaces all of that code would make this far simpler. With more runners allowing for ARM builds, this would let people avoid QEMU emulation and parallelize their build configurations far more easily

Dec 20 '24 18:12 diehard2

I've followed the steps here - https://docs.docker.com/build/ci/github-actions/multi-platform/ but I've been unable to get it to work for a custom registry.

Can you create a bug report with workflow and logs so we can take a look?

Its also extremely difficult to both understand and debug

Yes this has been raised in https://github.com/docker/build-push-action/issues/671 as well. We also want to ease distributed builds by using subactions for both build-push-action and bake-action.

'merge-manifests' that replaces all of that code would make this far simpler.

Can you explain how this merge-manifests input would work?

this would let people avoid QEMU emulation

This will not avoid QEMU on GitHub hosted runners, it will just build on a dedicated runner.

Jan 06 '25 10:01 crazy-max

I ran into a similar issue now that the arm64 linux runners ( ubuntu-24.04-arm) are online. Previously arm64 image were out of the question as qemu was too slow and MacOS M1 runner cannot build docker containers.

Using Noelware/docker-manifest-action seems to work.

workflow.yml

jobs:
  build:
    strategy:
      matrix:
        config:
          - {arch: 'arm64'}
          - {arch: 'amd64'}
    runs-on: ${{ matrix.config.arch == 'arm64' && 'ubuntu-24.04-arm' || 'ubuntu-latest' }}
    steps:
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Checkout code    
        uses: actions/checkout@v4
      - name: Build and Push Docker images for all Container Registries
        uses: docker/build-push-action@v6
        with:
          tags: |
            ghcr.io/${{ github.repository }}/build:latest-${{matrix.config.arch}}
          file: Dockerfile 
          push: true
  merge-docker-manifest:
    runs-on: ubuntu-latest
    needs: build
    steps:
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Create and push manifest images
        uses: Noelware/docker-manifest-action@master # or use a pinned version in the Releases tab
        with:
          inputs: ghcr.io/${{ github.repository }}/build:latest
          images: ghcr.io/${{ github.repository }}/build:latest-amd64,ghcr.io/${{ github.repository }}/build:latest-arm64           
        push: true

Jan 18 '25 18:01 junghans

now that the amd64 linux runners ( ubuntu-24.04-arm) are online

I guess you meant arm64 and yes we are going to update our docs related to these new runners but will not be up until these runners are GA.

Using Noelware/docker-manifest-action seems to work.

Your workflow creates multiple tags with architecture suffix which is not the desired behavior. In our case https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners we are pushing by digest to avoid that. Don't think Noelware/docker-manifest-action supports merging these manifests though as it's using docker manifest command: https://github.com/Noelware/docker-manifest-action/blob/a34eba526b9e4f02939cd3385b52f9cdef9bdf99/src/index.ts#L69

Jan 18 '25 18:01 crazy-max

Your workflow creates multiple tags with architecture suffix which is not the desired behavior.

I don't think that's the case, but that the arguments are very unclearly named. As I understand it, the architecture suffix tags are the inputs that get merged into one final manifest without a specific arch suffix.

Jan 18 '25 20:01 bo0tzz

now that the amd64 linux runners ( ubuntu-24.04-arm) are online

I guess you meant arm64 and yes we are going to update our docs related to these new runners but will not be up until these runners are GA.

Yes, typo!

Using Noelware/docker-manifest-action seems to work.

Your workflow creates multiple tags with architecture suffix which is not the desired behavior. In our case https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners we are pushing by digest to avoid that. Don't think Noelware/docker-manifest-action supports merging these manifests though as it's using docker manifest command: https://github.com/Noelware/docker-manifest-action/blob/a34eba526b9e4f02939cd3385b52f9cdef9bdf99/src/index.ts#L69

Yeah, I would like to avoid having the extra tags, but right now I can live with that.

Jan 19 '25 14:01 junghans

I dont have a method for creating both via a single job but if you are ok with a more verbose method you can split it out into 3 jobs.

The first job is a docker-amd64 which runs on the normal ubuntu-24.04 runner and builds as the name suggests amd64 images to your normal tags
The second job is a docker-arm64 which runs on the new ubuntu-24.04-arm runner and builds your arm64 images to your tags but with a suffix of arm64
The third job will then run after the two previous jobs are completed and combine the manifests together.

I only push 1 and 2 to the github container registry and then in step 3 i push the combined manifests to dockerhub. This lets me keep dockerhub clean as the arm64 tags wont exist since they arent needed individually.

Here is the code i use to combine the manifests

for TAG in $(jq -r '.tags[]' <<< "$DOCKER_METADATA_OUTPUT_JSON"); do
  echo "Creating manifest tag $TAG"
  # Replace ghcr.io/${{ github.actor }} with docker.io/${{ secrets.DOCKER_USERNAME }} if DOCKER_USERNAME is set
  if [ "${{ secrets.DOCKER_USERNAME }}" != "" ]; then
    DOCKERHUB_TAG=$(echo "--tag $TAG" | sed "s/ghcr.io\/${{ github.actor }}/${{ secrets.DOCKER_USERNAME }}/")
  else
    DOCKERHUB_TAG=""
  fi
  
  docker buildx imagetools create --append "${TAG}-arm64" --tag "${TAG}" ${DOCKERHUB_TAG}
done

You can look at my full github action here https://github.com/luigi311/tanoshi-builder/blob/d2d3e89df3d08d5e967befbf30cf04091e08e5a9/.github/workflows/ci.yml

I've been doing this for a while with a self hosted runner since qemu building the images were taking forever. This is pretty verbose though so hopefully theres a better way to do this in the future with a single matrix or something.

Feb 07 '25 04:02 luigi311

No real reason other than I didn't know it existed lol. I'll see if I can swap over to that to make it hopefully easier to understand.

Feb 14 '25 14:02 luigi311

You also can't use the provenance and sbom feature since using the example in their documentation makes you lose these information right away: https://docs.docker.com/build/ci/github-actions/attestations/

[!NOTE] Note that adding attestations to an image means you must push the image to a registry directly, as opposed to loading the image to the local image store of the runner. This is because the local image store doesn't support loading images with attestations.

Sep 28 '25 16:09 kimdre

This is because the local image store doesn't support loading images with attestations.

While that is the case, it can be easily mitigated with alternative export/import options which should be fine for a CI runner?

You can even run a registry locally such as with Zot (single binary or container), which can be ephemeral in the CI just to provide the needed conveniences. Pair that with a tool like oras CLI for managing OCI manifests/artifacts if you like and it should go fairly smoothly :)

Sep 28 '25 21:09 polarathene

I mean I can see that the attestation manifest is generated and it seems like it also gets pushed but I have no idea where it ends up or how I can add it to the merge job:

#23 [linux/amd64] generating sbom using docker.io/docker/buildkit-syft-scanner:stable-1
#23 0.056 time="2025-09-28T17:16:51Z" level=info msg="starting syft scanner for buildkit v1.9.0"
#23 DONE 1.2s

#24 exporting to image
#24 exporting layers
#24 exporting layers 3.5s done
#24 exporting manifest sha256:caf96e6a14dfba25af6ca2ba78b70bd719440c39b606a078781e6435db45e6e8 done
#24 exporting config sha256:7193a231e32f543f3f5f2e0dfa0e5f810f726af873fa4d7f59dac26ee46c8172 done
#24 exporting attestation manifest sha256:2a1fbced7d1432be4c76d0fdb665fdda70a5a83df83042dd37f916ce6f32bad6
#24 ...

#25 [auth] kimdre/doco-cd:pull,push token for ghcr.io
#25 DONE 0.0s

#24 exporting to image
#24 exporting attestation manifest sha256:2a1fbced7d1432be4c76d0fdb665fdda70a5a83df83042dd37f916ce6f32bad6 done
#24 exporting manifest list sha256:bfd0c4e643cdf658c88115fa6f88ea7795e42282d659880f89f3bf2d2b7e28c4 done
#24 pushing layers
#24 pushing layers 1.5s done
#24 pushing manifest for ghcr.io/kimdre/doco-cd
#24 pushing manifest for ghcr.io/kimdre/doco-cd 0.8s done
#24 DONE 6.4s

Sep 29 '25 07:09 kimdre

Here is a snippet of building an image and pushing it to a registry (localhost:5000 is a zot container in this case), and fetching the manifest from that image published, parsing it with jq to look at the annotations metadata with a Docker specific entry to use it's digest for retrieving the associated attestation manifest:

docker buildx build --builder custom-builder \
  --output 'type=image,push=true,oci-mediatypes=true,oci-artifact=true' \
  --platform 'linux/amd64,linux/arm64' \
  --provenance true \
  --tag localhost:5000/hello/world:example \
  .

oras manifest fetch localhost:5000/hello/world:example \
  | jq -r '[ .manifests[]
    | select(.annotations | has("vnd.docker.reference.digest"))
    | .digest
  ] | first' \
  | oras manifest fetch "localhost:5000/hello/world@$(cat -)"

I haven't looked into this for a while, but I provide rather verbose information and commands in the following comments that should detail plenty to answer any questions you have?:

https://github.com/project-zot/zui/issues/476#issuecomment-2712247630
https://github.com/project-zot/zui/issues/476#issuecomment-2712465919
https://github.com/project-zot/zui/issues/475#issuecomment-2715702195 (shows difference of oci-artifact enabled/disabled)
https://github.com/project-zot/zui/issues/475#issuecomment-2715897490

More detailed guidance

Here is an example of using a different output type with buildx to export locally, and access the same associated manifest entry without a registry involved:

# Basic image to build:
$ echo 'FROM alpine' > Dockerfile

# To use `--output 'type=oci` you'll need the `docker-container` builder driver with buildx:
$ docker buildx create --name=via-container --driver=docker-container --platform 'linux/amd64,linux/arm64'

# Output build to `oci-export/hello-world/` dir (and do not archive contents into a tar file)
$ docker buildx build \
  --output 'type=oci,dest=./oci-export/hello-world/,tar=false,oci-mediatypes=true,oci-artifact=true' \
  --platform 'linux/amd64,linux/arm64' \
  --provenance true \
  --tag localhost:5000/hello/world:example \
  --builder via-container \
  .

# This is the main image index:
$ jq . oci-export/hello-world/index.json

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.index.v1+json",
      "digest": "sha256:4790efc3ed2a32fa46e07ac3293e8cc0ba58369eb21bdbebf9b8546d0a1b03c4",
      "size": 1607,
      "annotations": {
        "io.containerd.image.name": "localhost:5000/hello/world:example",
        "org.opencontainers.image.created": "2025-09-30T05:17:32Z",
        "org.opencontainers.image.ref.name": "example"
      }
    }
  ]
}

# And here is the image manifest index (from the digest value above),
# it shows the digest for the two platforms (AMD64 + ARM64) we built
# and two separate provenance attestation digests, all linked to the built image `hello/world:example`
$ jq . oci-export/hello-world/blobs/sha256/4790efc3ed2a32fa46e07ac3293e8cc0ba58369eb21bdbebf9b8546d0a1b03c4

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:d09089c164ac7e39cde68f17574b58459114754a7d79fd7979164bea3d478bfc",
      "size": 668,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:d16e7719f24d4d1f2c6ecb695463fb0e702da45de53dae8ba305682de2abc212",
      "size": 668,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:f7a1e0d8020e3ebc42ba4f8645e873088f99f59a162275759363931a635c5b90",
      "size": 566,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:d09089c164ac7e39cde68f17574b58459114754a7d79fd7979164bea3d478bfc",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:a4b725746113d09298347ab48247e3aa40d3e8b14578b93e7ebaec59028fec4e",
      "size": 566,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:d16e7719f24d4d1f2c6ecb695463fb0e702da45de53dae8ba305682de2abc212",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    }
  ]
}

As you can see the last two "images" are for the Docker added attestations. This is only relevant to retain for some Docker software/tooling that relies on this approach to get the attestations instead of two other methods that OCI image registries may support as part of the OCI spec (as the image was built with OCI equivalent support enabled, that may be enough for you).

Those attestations refer to their own digest for more info, and also are annotated with a reference to their associated AMD64/ARM64 image from the first two images in that manifest list.

Manual inspection to get the manifest shown above from the known image index and then show the attestation manifest:

# Show the first provenance attestation found
# (Docker specific annotation prior to OCI artifact support, you cannot opt-out of it's generation):
$ cat oci-export/hello-world/index.json \
| jq -r .manifests[0].digest \
| cut -f2 -d ":" \
| cat "oci-export/hello-world/blobs/sha256/$(cat -)" \
\
| jq -r '[ .manifests[]
    | select(.annotations | has("vnd.docker.reference.digest"))
    | .digest
  ] | first' \
| cut -f2 -d ":" \
| cat "oci-export/hello-world/blobs/sha256/$(cat -)"

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:60072579ad378ab2f8d388e7f5b37dffefddde7b0134bf1c15d04e1b82cc54e4",
    "size": 167
  },
  "layers": [
    {
      "mediaType": "application/vnd.in-toto+json",
      "digest": "sha256:5712d4fec8cb55936b36c4e4df967af517c03970fbc5bee89bf014fbb9004613",
      "size": 3253,
      "annotations": {
        "in-toto.io/predicate-type": "https://slsa.dev/provenance/v0.2"
      }
    }
  ]
}

For multi-platform images there will be an attestation per platform (referenced via annotation). This is Docker's own approach before OCI artifacts were standardized (hence the unknown platform info which causes some registries to list invalid unknown platforms for images).

$ cat oci-export/hello-world/index.json | jq -r .manifests[0].digest | cut -f2 -d ":" | cat "oci-export/hello-world/blobs/sha256/$(cat -)" | jq -r '[ .manifests[]
    | select(.annotations | has("vnd.docker.reference.digest"))
  ]'

[
  {
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "digest": "sha256:f7a1e0d8020e3ebc42ba4f8645e873088f99f59a162275759363931a635c5b90",
    "size": 566,
    "annotations": {
      "vnd.docker.reference.digest": "sha256:d09089c164ac7e39cde68f17574b58459114754a7d79fd7979164bea3d478bfc",
      "vnd.docker.reference.type": "attestation-manifest"
    },
    "platform": {
      "architecture": "unknown",
      "os": "unknown"
    }
  },
  {
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "digest": "sha256:a4b725746113d09298347ab48247e3aa40d3e8b14578b93e7ebaec59028fec4e",
    "size": 566,
    "annotations": {
      "vnd.docker.reference.digest": "sha256:d16e7719f24d4d1f2c6ecb695463fb0e702da45de53dae8ba305682de2abc212",
      "vnd.docker.reference.type": "attestation-manifest"
    },
    "platform": {
      "architecture": "unknown",
      "os": "unknown"
    }
  }
]

Use oras manifest fetch for simpler CLI command than the example above:

# Install oras:
# https://oras.land/docs/installation#release-artifacts
$ RELEASE_URL_ORAS='https://github.com/oras-project/oras/releases/download/v1.3.0/oras_1.3.0_linux_amd64.tar.gz'
$ curl -fsSL "${RELEASE_URL_ORAS}" | tar -xz -C /usr/local/bin --no-same-owner oras

# This will get the same image manifest list shown earlier (2 platforms + 2 attestations):
$ oras manifest fetch --oci-layout oci-export/hello-world:example

# Still a tad verbose to fetch (manifest list), parse the manifest via jq, and fetch another manifest (attestation):
$ oras manifest fetch --oci-layout oci-export/hello-world:example \
  | jq -r '[ .manifests[]
    | select(.annotations | has("vnd.docker.reference.digest"))
    | .digest
  ] | first' \
  | oras manifest fetch --oci-layout "oci-export/hello-world@$(cat -)"

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:60072579ad378ab2f8d388e7f5b37dffefddde7b0134bf1c15d04e1b82cc54e4",
    "size": 167
  },
  "layers": [
    {
      "mediaType": "application/vnd.in-toto+json",
      "digest": "sha256:5712d4fec8cb55936b36c4e4df967af517c03970fbc5bee89bf014fbb9004613",
      "size": 3253,
      "annotations": {
        "in-toto.io/predicate-type": "https://slsa.dev/provenance/v0.2"
      }
    }
  ]
}

That is a tad awkward still if you need to rely on that, but for most image registries the OCI artifact support should work for the attestations, and you could identify artifacts via oras discover instead.

Just preserve the manifest entries output by buildx for your image platform and associated attestation if you want to merge those into one.

Otherwise if you don't need to preserve that Docker specific attestation approach, you only need to know the image digests for each platform and can create a new index from that (see this guidance with oras manifest index create, you can also update an existing one and append to it).

I've probably made it sound more complicated than it is 😅

OCI image export type or a registry like DockerHub does seem to limit you for discovering the artifacts like attestation, and would require that longer approach shown above (with filtering based on platform image digest):

# The DockerHub registry for this image has the Docker attestation entries per platform:
$ oras manifest fetch docker.io/library/alpine:latest | jq
# ...

# But no referrers API/tag OCI support to discover the artifacts:
$ oras discover docker.io/library/alpine:latest --distribution-spec v1.1-referrers-api --format json

{
  "reference": "docker.io/library/alpine@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1",
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "digest": "sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1",
  "size": 9218,
  "referrers": []
}

So in that case you may have two separate runners build each platform and push the individual images, if you did the oci image layout output though, you could restore those on the merge runner, or extract their image manifest list/index (platform image + attestation) and merge the two like this:

# `<(...)` runs the command and redirects file output to a file-descriptor,
# this provides the output of the command as a file-name to the `yq` command to read in and merge
# yq is used here because jq lacks the equivalent merge capability to combine the manifests array key.
yq eval-all '. as $item ireduce ({}; . *+ $item )' <(oras manifest fetch --oci-layout oci-export/hello-world-amd64:example) <(oras manifest fetch --oci-layout oci-export/hello-world-arm64:example)

With that you'll have the desired JSON and can just push that to the registry, oras provides another subcommand for that 👍

Sep 30 '25 10:09 polarathene