docker
allow to create manifests of local images
Description
docker manifest create does not allow to create a manifest from unpushed images.
Steps to reproduce the issue:
docker buildsome images locallydocker manifest createusing the images you just created
Describe the results you received:
no such manifest: registry/mylocalimg:tag
Describe the results you expected:
It should create the manifest locally, and when I push it (docker manifest push), it should push everything.
Additional information you deem important (e.g. issue happens only occasionally):
Output of docker version:
Client:
Cloud integration: 1.0.17
Version: 20.10.8
API version: 1.41
Go version: go1.16.6
Git commit: 3967b7d
Built: Fri Jul 30 19:55:20 2021
OS/Arch: darwin/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.8
API version: 1.41 (minimum version 1.12)
Go version: go1.16.6
Git commit: 75249d8
Built: Fri Jul 30 19:52:31 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.9
GitCommit: e25210fe30a0a703442421b0f60afac609f950a3
runc:
Version: 1.0.1
GitCommit: v1.0.1-0-g4144b63
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Output of docker info:
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Build with BuildKit (Docker Inc., v0.6.3)
compose: Docker Compose (Docker Inc., v2.0.0)
scan: Docker Scan (Docker Inc., v0.8.0)
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 8
Server Version: 20.10.8
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: e25210fe30a0a703442421b0f60afac609f950a3
runc version: v1.0.1-0-g4144b63
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 5.10.47-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.939GiB
Name: docker-desktop
ID: R5K7:TTMY:2XBT:BBBZ:4F6B:YEEA:OJCY:YTEV:4FEM:22F3:LKOY:2JJR
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
n/a
I'm not sure this would be possible, as manifests are a reference to images in the registry, and reference the image's digest. This digest is calculated when the image is pushed. So, it would be "somewhat" possible to do this, but would involve the manifest push command to "automate" the steps;
- look up local images referenced (
foo:bar,foo:baz) - push
foo:barand get the digest - push
foo:bazand get the digest - update the digest in the manifest
- push the manifest
Using buildx to build the multi-arch image and/or the GitHub action for this is probably a better solution. Alternatively, docker buildx imagetools create (with the --append option) could possibly work as well to append new architectures to a manifest; https://docs.docker.com/engine/reference/commandline/buildx_imagetools_create/
Alternatively,
docker buildx imagetools create(with the--appendoption) could possibly work as well to append new architectures to a manifest; https://docs.docker.com/engine/reference/commandline/buildx_imagetools_create/
Dear @thaJeztah, how can one do it? We must use different --build-arg for different --platform, so we can't build all platforms in one run. I'd say, out of documentation and playing around I see that it requires images to be pushed to a registry as well.
Here's an example.
We have two different commands to build an images
docker buildx build --platform=linux/amd64 --build-arg=REPOSITORY='https://s3.amazonaws.com/package_release' --tag=clickhouse/clickhouse-server:head-amd64 --build-arg=VERSION='22.4.1.678' --progress=plain docker/server
docker buildx build --platform=linux/arm64 --build-arg=REPOSITORY='https://s3.amazonaws.com/package_aarch64' --tag=clickhouse/clickhouse-server:head-arm64 --build-arg=VERSION='22.4.1.678' --progress=plain docker/server
So we have two different images. Is there any way to merge them together before pushing remotely?
upd
Ok, according to https://github.com/docker/buildx/issues/805#issuecomment-946427262 it's possible to push images by digest, save sha256 id with --iidfile head-arm64, and then one can use them in docker buildx imagetools create -t clickhouse/clickhouse-server:head -f head-arm64 -f head-amd64. Am I right?
upd2
No, I was wrong. The way to go is by using --metadata-file and reading containerimage.digest from there. The images should be pushed as --output=type=image,push-by-digest=true --tag=clickhouse/clickhouse-server, then a merged manifest is created as docker buildx imagetools create --tag clickhouse/clickhouse-server:head sha256:7ba33340ad98c15a90b30961552e7daab8a3936f4747347a82a9014736cc4abd sha256:ddbe9e633c9bc71de8f2d3814c6875a287b9c97d739fa9ed9893c8911b3e8852
@thaJeztah another question regarding your point
This digest is calculated when the image is pushed.
I highly doubt it's the currently correct statement. When I use docker buildx build --metadata-file ... w/o pushing an image, I have digests there:
> grep containerimage.digest tmp/docker_images_check/*
tmp/docker_images_check/head-alpine-amd64: "containerimage.digest": "sha256:3483a523269858a88d10bbd43f46a0a2544cccff8c072a5a8c2926e234a10a3a",
tmp/docker_images_check/head-alpine-arm64: "containerimage.digest": "sha256:bf813839db941477e7c8ebde94361e3612885187040ec6bbf4e367996f605694",
tmp/docker_images_check/head-amd64: "containerimage.digest": "sha256:7ea5a2790493219904ccc78ed6012de052571c4e040d7e35919d78be408c2792",
tmp/docker_images_check/head-arm64: "containerimage.digest": "sha256:8c72952b2c22795dce3014dd752a96f2e587696b3254ef0bf65f4fcb84ae9577",
I can use them to refer remote digests, and it works if unnamed images are pushed to the repo:
docker buildx imagetools create --tag clickhouse/clickhouse-server:head sha256:7ea5a2790493219904ccc78ed6012de052571c4e040d7e35919d78be408c2792 sha256:8c72952b2c22795dce3014dd752a96f2e587696b3254ef0bf65f4fcb84ae9577
If I can use them, why wouldn't docker build imagetools create be able to push all at once? I guess it's just a meaning of some batch of code.
I'd have to double check with the @docker/build team. Some things may depend on what kind of builder you're using (container-builder or docker engine embedded buildkit); there's multiple digests; digest of the local image (which is calculated over the image layers in the local image store, before compressing; these are stable and reproducible), and digests of the image layers (blobs) after compressing; those are not guaranteed to be stable, as compression may produce different results in many circumstances (this is the reason the containerd store defaults to storing both compressed and un-compressed artifacts of images after pulling).
As to:
We have two different commands to build an images
docker buildx build --platform=linux/amd64 --build-arg=REPOSITORY='https://s3.amazonaws.com/package_release' --tag=clickhouse/clickhouse-server:head-amd64 --build-arg=VERSION='22.4.1.678' --progress=plain docker/server docker buildx build --platform=linux/arm64 --build-arg=REPOSITORY='https://s3.amazonaws.com/package_aarch64' --tag=clickhouse/clickhouse-server:head-arm64 --build-arg=VERSION='22.4.1.678' --progress=plain docker/server
I'm wondering if for that specific example, the Dockerfile itself would be able to pick the right option based on one of the automatic platform ARGs, and conditionally pick the right value for REPOSITORY
But perhaps buildx bake would allow them to be set for each variant (@crazy-max ?), so that all can be built and pushed as a multi-arch image (each variant with the correct options).
You are right, after docker buildx prune the new built images are different
diff -ru1 tmp/docker_images_check/head-alpine-amd64 tmp/docker_images_check_cached/head-alpine-amd64
--- tmp/docker_images_check/head-alpine-amd64 2022-03-31 14:50:29.816595096 +0200
+++ tmp/docker_images_check_cached/head-alpine-amd64 2022-03-31 10:46:31.912117357 +0200
@@ -22,12 +22,12 @@
},
- "containerimage.config.digest": "sha256:2abc7dfcc4c4b268e24655d9bf1027564fb782a8cce3d70b5105e4e1d14ce534",
+ "containerimage.config.digest": "sha256:5452ea43ee5aa45a6e1e26182b04a0df9bdb8da02e1a3321139200c7cf06a80f",
"containerimage.descriptor": {
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
- "digest": "sha256:32470e84612dfaebc4306e332a30158e855b6c6b079a4f6c0aea006db72db8c6",
+ "digest": "sha256:3483a523269858a88d10bbd43f46a0a2544cccff8c072a5a8c2926e234a10a3a",
"size": 1781,
"annotations": {
- "org.opencontainers.image.created": "2022-03-31T12:50:26Z"
+ "org.opencontainers.image.created": "2022-03-31T08:46:28Z"
}
},
- "containerimage.digest": "sha256:32470e84612dfaebc4306e332a30158e855b6c6b079a4f6c0aea006db72db8c6",
+ "containerimage.digest": "sha256:3483a523269858a88d10bbd43f46a0a2544cccff8c072a5a8c2926e234a10a3a",
"image.name": "docker.io/clickhouse/clickhouse-server:head-alpine-amd64"
diff -ru1 tmp/docker_images_check/head-alpine-arm64 tmp/docker_images_check_cached/head-alpine-arm64
--- tmp/docker_images_check/head-alpine-arm64 2022-03-31 14:51:15.940363221 +0200
+++ tmp/docker_images_check_cached/head-alpine-arm64 2022-03-31 10:47:16.782102949 +0200
@@ -22,12 +22,12 @@
},
- "containerimage.config.digest": "sha256:ebf0b472412a991c8441b1da479a898b90e045ffbcefa2b5b08ee6cc57f8b60b",
+ "containerimage.config.digest": "sha256:ebee291992d7fd02dccba1520eaf80442f50855ea82035b8c7b7d8e4c10eb273",
"containerimage.descriptor": {
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
- "digest": "sha256:53dfe410eaa5a3a2414f099be9d360add6e769653344a0261fe84dff07c91361",
+ "digest": "sha256:bf813839db941477e7c8ebde94361e3612885187040ec6bbf4e367996f605694",
"size": 1781,
"annotations": {
- "org.opencontainers.image.created": "2022-03-31T12:51:12Z"
+ "org.opencontainers.image.created": "2022-03-31T08:47:13Z"
}
},
- "containerimage.digest": "sha256:53dfe410eaa5a3a2414f099be9d360add6e769653344a0261fe84dff07c91361",
+ "containerimage.digest": "sha256:bf813839db941477e7c8ebde94361e3612885187040ec6bbf4e367996f605694",
"image.name": "docker.io/clickhouse/clickhouse-server:head-alpine-arm64"
diff -ru1 tmp/docker_images_check/head-amd64 tmp/docker_images_check_cached/head-amd64
--- tmp/docker_images_check/head-amd64 2022-03-31 14:47:24.888169443 +0200
+++ tmp/docker_images_check_cached/head-amd64 2022-03-31 10:44:51.898729939 +0200
@@ -17,12 +17,12 @@
},
- "containerimage.config.digest": "sha256:a0306caba2fedb10ebd5dc10b0f94a36ac71bb007abcb09eb60c5207b1d9dfe9",
+ "containerimage.config.digest": "sha256:d84561301369020bb5e98927796ea9fecbf7a31952db8dc334f96d8addbc0d31",
"containerimage.descriptor": {
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
- "digest": "sha256:34908b2909e4e3bb8bf5c6d27e35a6eae272892534af6c2e550a5b58ab41cfa1",
+ "digest": "sha256:7ea5a2790493219904ccc78ed6012de052571c4e040d7e35919d78be408c2792",
"size": 2201,
"annotations": {
- "org.opencontainers.image.created": "2022-03-31T12:47:20Z"
+ "org.opencontainers.image.created": "2022-03-31T08:44:48Z"
}
},
- "containerimage.digest": "sha256:34908b2909e4e3bb8bf5c6d27e35a6eae272892534af6c2e550a5b58ab41cfa1",
+ "containerimage.digest": "sha256:7ea5a2790493219904ccc78ed6012de052571c4e040d7e35919d78be408c2792",
"image.name": "docker.io/clickhouse/clickhouse-server:head-amd64"
diff -ru1 tmp/docker_images_check/head-arm64 tmp/docker_images_check_cached/head-arm64
--- tmp/docker_images_check/head-arm64 2022-03-31 14:49:35.582748688 +0200
+++ tmp/docker_images_check_cached/head-arm64 2022-03-31 10:45:44.452107433 +0200
@@ -17,12 +17,12 @@
},
- "containerimage.config.digest": "sha256:dfc487c2ddef3ff512df0bae810a49ae52e2e29dae529ac3fd0602f525b81f35",
+ "containerimage.config.digest": "sha256:9df4519cb13cc2897568cb8607c4279aef6e31f173c8f42a0842ffd325b571bb",
"containerimage.descriptor": {
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
- "digest": "sha256:1df207d7ea2ffa7f293e8156d29114c8221706e8a87b27466c81b165f7b26eeb",
+ "digest": "sha256:8c72952b2c22795dce3014dd752a96f2e587696b3254ef0bf65f4fcb84ae9577",
"size": 2201,
"annotations": {
- "org.opencontainers.image.created": "2022-03-31T12:49:31Z"
+ "org.opencontainers.image.created": "2022-03-31T08:45:41Z"
}
},
- "containerimage.digest": "sha256:1df207d7ea2ffa7f293e8156d29114c8221706e8a87b27466c81b165f7b26eeb",
+ "containerimage.digest": "sha256:8c72952b2c22795dce3014dd752a96f2e587696b3254ef0bf65f4fcb84ae9577",
"image.name": "docker.io/clickhouse/clickhouse-server:head-arm64"
On one hand... But on another, I still don't see an issue. In any case, the operation seems atomic. I push either everything at once or nothing.
This requirement could be a prerequisite for manually pushing multi-arch images without -amd64/-arm64 alike tags pushed at first.
Buildx supports doing it internally but docker manifest requires images to be pushed first. A common case is that users build multi-arch images on different workers (GitHub Actions runners). While images can be shared by save/load, this process doesn't create a manifest and thus cannot create a multi-arch image (manifest) locally and push.
Said you can do docker buildx build --platfrom ... --push ., but you cannot do:
docker load -i amd64-image.tar
docker load -i arm64-image.tar
// do something to merge the images and push
I didn't experiment locally but would push w/o tagging and latter using preserved sha256 work?
@Felixoid That should satisfy my use case at some point. How can I find the sha256 programatically? (so that I can integrate it into the CD process)
In our case, the push is done here https://github.com/ClickHouse/ClickHouse/blob/1e4fe038f562029fc24a0e7a33e5d428ea0474f9/tests/ci/docker_server.py#L243-L245, and the metadata file is preserved in https://github.com/ClickHouse/ClickHouse/blob/1e4fe038f562029fc24a0e7a33e5d428ea0474f9/tests/ci/docker_server.py#L264
So later the metadata is read and the container's digest used to combine the manifest in https://github.com/ClickHouse/ClickHouse/blob/1e4fe038f562029fc24a0e7a33e5d428ea0474f9/tests/ci/docker_server.py#L294
for reference if anyone needs it https://github.com/backube/volsync/pull/1613
With the containerd image-store enabled, the local image store keeps multi-arch images by default;
- to enable on docker engine; https://docs.docker.com/engine/storage/containerd/
- and docker desktop (enabled by default on fresh installs) https://docs.docker.com/desktop/features/containerd/
In that case, you can build multiple architectures in a single build (docker build --platform linux/arm64,linux/amd64 ...); which can be either pushed directly (--push), or stored locally before pushing;
docker build --platform linux/arm64,linux/amd64 -t myimage .
# ...
docker image ls --tree myimage
IMAGE ID DISK USAGE CONTENT SIZE EXTRA
myimage:latest b4a38934dd47 287MB 112MB
├─ linux/arm64 704dba99917f 231MB 56MB
└─ linux/amd64 8197ca2e73e2 56.3MB 56.3MB
It's evident that the manifest from the single docker build --platform a,b,c,d command is created. This feature is at least 3 or 4 years old, AFAIK.
This feature request is about building multiple separate images locally, and then pushing all of them together as the single manifest.
For example, my case specifically concerned building different architects with different --build-arg parameters. I couldn't use --platform a,b,f, because they require different URLs to download the artifacts.