docker
The metadata output has gotten big for multilevel images in v5.6.0
Contributing guidelines
- [X] I've read the contributing guidelines and wholeheartedly agree
I've found a bug, and:
- [X] The documentation does not mention anything about my problem
- [X] There are no open or closed issues that are related to my problem
Description
Hi :wave:
The recently released version 5.6.0 broke our CI because the format of the metadata GHA output changed. I understand that this is probably caused by dependencies of this action, but I was not sure where else to open the issue --- happy to be redirected elsewhere.
Concretely, one of our steps started to fail since we've been parsing the metadata output and we were passing it via an environment variable like this:
- name: Set output variables
run: |
.github/workflows/extract-image-names.sh | tee -a "${GITHUB_OUTPUT}" | awk -F'=' '{print $2}' | jq
env:
BAKE_METADATA: ${{ steps.build-upload.outputs.metadata }}
Because of the size, we started to hit this error:
Error: An error occurred trying to start process '/usr/bin/bash' with working directory '/home/runner/work/aiidalab-docker-stack/aiidalab-docker-stack'. Argument list too long
We've been able to workaround this by saving the metadata output into a file first like this
- name: Set output image names
run: |
cat << EOF > bake_metadata.json
${{ steps.build-upload.outputs.metadata }}
EOF
images=$(.github/workflows/extract-image-names.sh bake_metadata.json)
echo "images=${images}" >> "${GITHUB_OUTPUT}"
Nevertheless, even with this workaround, it might be nice to try to prune the metadata output a bit to make it a bit more readable in the GHA logs. Specifically, there is now a section {"buildx.build.provenance": {"invocation": {...}} that is huge for multilevel builds.
Here's the output from the base image, nothing out of the ordinary
"base": {
"buildx.build.provenance": {
"buildType": "https://mobyproject.org/buildkit@v1",
"materials": [
{
"uri": "pkg:docker/jupyter/[email protected]?platform=linux%2Famd64",
"digest": {
"sha256": "f7022a83184c73b04fdb16d3f2f2b6b014c819ac086b87496f1c0641a0614f3a"
}
}
],
"invocation": {
"configSource": {
"entryPoint": "Dockerfile"
},
"parameters": {
"frontend": "dockerfile.v0",
"args": {
"build-arg:AIIDA_VERSION": "2.5.1",
"build-arg:BASE": "jupyter/minimal-notebook:python-3.9.13"
},
"locals": [
{
"name": "context"
},
{
"name": "dockerfile"
}
]
},
"environment": {
"platform": "linux/amd64"
}
}
},
"buildx.build.ref": "builder-df0f1d9d-0da6-4602-89bf-f0dd8e3e962d/builder-df0f1d9d-0da6-4602-89bf-f0dd8e3e962d0/2yfr5apv8yx146zzg287f66an",
"containerimage.config.digest": "sha256:dc3491813cf2fa44b8ed0dc6a914b221a7d1668544193496a516de248224a29e",
"containerimage.descriptor": {
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"digest": "sha256:a4ecb05d8859237d79ff229f84d1835f9d639aba923c3e8292318c42763b7267",
"size": 6356,
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
"containerimage.digest": "sha256:a4ecb05d8859237d79ff229f84d1835f9d639aba923c3e8292318c42763b7267",
"image.name": "ghcr.io/aiidalab/base"
},
Here's the second part from the image built on top of base image
"base-with-services": {
"buildx.build.provenance": {
"buildType": "https://mobyproject.org/buildkit@v1",
"materials": [
{
"uri": "pkg:docker/docker/dockerfile@1",
"digest": {
"sha256": "fe40cf4e92cd0c467be2cfc30657a680ae2398318afd50b0c80585784c604f28"
}
},
{
"uri": "pkg:docker/jupyter/[email protected]?digest=sha256:f7022a83184c73b04fdb16d3f2f2b6b014c819ac086b87496f1c0641a0614f3a&platform=linux%2Famd64",
"digest": {
"sha256": "f7022a83184c73b04fdb16d3f2f2b6b014c819ac086b87496f1c0641a0614f3a"
}
}
],
"invocation": {
"configSource": {
"entryPoint": "Dockerfile"
},
"parameters": {
"frontend": "gateway.v0",
"args": {
"build-arg:AIIDA_VERSION": "2.5.1",
"build-arg:PGSQL_VERSION": "15",
"build-arg:RMQ_VERSION": "3.9.13",
"cmdline": "docker/dockerfile:1",
"context:base": "input:0-base",
"frontend.caps": "moby.buildkit.frontend.contexts+forward",
"input-metadata:0-base": "{\"containerimage.config\":\"eyJhcmNoaXRlY3R1cmUiOiJhbWQ2NCIsIm9zIjoibGludXgiLCJyb290ZnMiOnsidHlwZSI6ImxheWVycyIsImRpZmZfaWRzIjpbInNoYTI1NjoxN2Y2MjNhZjAxZTI3N2M1ZmZlNjc3OWFmODE2NDkwN2RlMDJkOWFmN2EwZTE2MTY2MmZjNzM1ZGQ2NGYxMTdiIiwic2hhMjU2OjIxMjU2Y2ZiZjVjZDc4MjNjNjk3OTNiZmZlMmEyMmZjNWFiN2U2Mzg0MzZiNzY3MTFjMmIwZTNkYjgwNjExMTUiLCJzaGEyNTY6NWVhNjgxOTRjM2JmMmRjM2U0YzIxMDFiMDdlMWEwMGMyYmI4MmEyOGUyZDk0ZTYwNzNiNTFhOTNhZGE0NzE3YiIsInNoYTI1Njo1ZjcwYmYxOGEwODYwMDcwMTZlOTQ4YjA0YWVkM2I4MjEwM2EzNmJlYTQxNzU1YjZjZGRmYWYxMGFjZTNjNmVmIiwic2hhMjU2OjRhZmE4YzMzY2QyYzZkYTJhNmEyMzllMTcxZGJjMTgyNWVmMzIyYWM3ZmRjMWUwZWFmZGZiZjYyYzkwMTEzNDIiLCJzaGEyNTY6MGNhMzhlZjY2ZGZkZDI2ODRjMjhiMGE4MTQ2YTc2MGQ3YTdhYmJhNmQzYWZlZGRmMjk4NGM4MWFiZWNjZGJiNCIsInNoYTI1NjpjNzZlZTJkMDU0YjBkZWVlZDgzOTc0ZGE5M2Q0ZDBjMzg2NDJmY2JmMGNiYjQ2NTRhZjAzNjUzNzE0MWNmOWFhIiwic2hhMjU2OjRiOGE0YzlkNjM1OTNmMDNjMWI4ZjU1ZGQ4MDE5OGM2YTc3NWQ4ZTFkNjg0MGUzNzRmZTNlMTZlZjg0Y2IxM2IiLCJzaGEyNTY6NWY3MGJmMThhMDg2MDA3MDE2ZTk0OGIwNGFlZDNiODIxMDNhMzZiZWE0MTc1NWI2Y2RkZmFmMTBhY2UzYzZlZiIsInNoYTI1Njo3NWM2OGY2YjA4YjJhNGU2NjMxYTA1MDMwZDA0NTliMjA5ZjZmODdiYTU4ZmI1NWExNzliM2MxYzI1ZWFjYTQ2Iiwic2
(I elided the full output)
It would be great if the "input-metadata:0-base": "{\"containerimage.config\": section was not there.
Repository URL
https://github.com/aiidalab/aiidalab-docker-stack
Workflow run URL
https://github.com/aiidalab/aiidalab-docker-stack/actions/runs/10283635135/job/28465139810
YAML workflow
- name: Build and upload to ghcr.io
id: build-upload
uses: docker/[email protected]
with:
push: true
# Using provenance to disable default attestation so it will build only desired images:
# https://github.com/orgs/community/discussions/45969
provenance: false
set: |
*.platform=${{ inputs.platforms }}
*.output=type=registry,push-by-digest=true,name-canonical=true
*.cache-to=type=gha,scope=${{ github.workflow }},mode=max
*.cache-from=type=gha,scope=${{ github.workflow }}
files: |
docker-bake.hcl
build.json
.github/workflows/env.hcl
- name: Set output image names
id: bake_metadata
# bake-action metadata output has gotten too big, so we first write it
# to a file. See https://github.com/aiidalab/aiidalab-docker-stack/issues/491
- name: Set output variables
run: |
.github/workflows/extract-image-names.sh | tee -a "${GITHUB_OUTPUT}" | awk -F'=' '{print $2}' | jq
env:
BAKE_METADATA: ${{ steps.build-upload.outputs.metadata }}
Workflow logs
No response
BuildKit logs
No response
Additional info
No response
Thanks for reporting, what outputs are being created in Set output variables step?
Can you show what returns .github/workflows/extract-image-names.sh?
I think it would be better to rely on https://github.com/actions/github-script instead of shell for this use case.
Edit:
Thanks for reporting, what outputs are being created in
Set output variablesstep?Can you show what returns
.github/workflows/extract-image-names.sh?
Nevermind found it: https://github.com/aiidalab/aiidalab-docker-stack/blob/4b5e560db34ded20f5173dba53ecfef247c45c74/.github/workflows/extract-image-names.sh#L39-L46
I think something like this could to the trick (not tested):
- name: Set output variables
uses: actions/github-script@v7
with:
script: |
for (const key in JSON.parse(process.env.BAKE_METADATA || '{}', 'utf-8')) {
if (data.hasOwnProperty(key)) {
const entry = data[key];
core.setOutput(`${key.toUpperCase().replace(/-/g, "_")}_IMAGE`, `${entry["image.name"]}@${entry["containerimage.digest"]}`);
}
}
env:
BAKE_METADATA: ${{ steps.build-upload.outputs.metadata }}
Hi @crazy-max,
I've finally got around to trying your solution from https://github.com/docker/bake-action/issues/239#issuecomment-2285948802 (thanks so much for providing it! 💟 ).
Unfortunately, I am running into the same problem as in the original report:
Error: An error occurred trying to start process '/home/runner/runners/2.322.0/externals/node20/bin/node' with working directory '/home/runner/work/aiidalab-docker-stack/aiidalab-docker-stack'. Argument list too long
Would you mind re-opening this issue?
Here's the slightly modified version of your suggestion that I tried:
- name: Set output variables
id: set-output
uses: actions/github-script@v7
with:
# We need to produce the following JSON string from the bake action output,
# which is then used in follow-up steps to uniquelly identify the built
# images by their digests, and not by their tags. See e.g. test.yml
# {
# "BASE_IMAGE": "ghcr.io/aiidalab/base@sha256:bc53c...",
# "BASE_WITH_SERVICES_IMAGE":"ghcr.io/aiidalab/base-with-services@sha256:0df1...",
# "FULL_STACK_IMAGE":"ghcr.io/aiidalab/full-stack@sha256:dd04...",
# "LAB_IMAGE":"ghcr.io/aiidalab/lab@sha256:e8c7b3a662660ad20fef7...",
# }
script: |
const bake_output = JSON.parse(process.env.BAKE_METADATA, 'utf-8');
images = {};
for (const key in bake_output) {
const image = bake_output[key];
// turn e.g. 'full-stack' to 'FULL_STACK_IMAGE' key
const image_envvar = `${key.toUpperCase().replace(/-/g, "_")}_IMAGE`;
// create full canonical path to the image using its digest, i.e.
// ghcr.io/aiidalab/base@sha256:cdad93278a...
const image_digest = `${image["image.name"]}@${image["containerimage.digest"]}`;
images[image_envvar] = image_digest;
}
console.log(images);
return images;
env:
BAKE_METADATA: ${{ steps.build-upload.outputs.metadata }}
Here are the logs: https://github.com/aiidalab/aiidalab-docker-stack/actions/runs/13140307956/job/36665668019
I also tried directly injecting ${{ steps.build-upload.outputs.metadata }} into the JSON.parse function as a string like this, with no luck and the same error: (logs)
- name: Set output variables
id: set-output
uses: actions/github-script@v7
with:
script: |
const bake_output = JSON.parse('${{ steps.build-upload.outputs.metadata }}', 'utf-8');
images = {};
for (const key in bake_output) {
const image = bake_output[key];
// turn e.g. 'full-stack' to 'FULL_STACK_IMAGE' key
const image_envvar = `${key.toUpperCase().replace(/-/g, "_")}_IMAGE`;
// create full canonical path to the image using its digest, i.e.
// ghcr.io/aiidalab/base@sha256:cdad93278a...
const image_digest = `${image["image.name"]}@${image["containerimage.digest"]}`;
images[image_envvar] = image_digest;
}
console.log(images);
return images;