cli Docker Containerd snapshotter Overlayfs, docker save fails for image pulled using --platform=linux/arm64 on amd64 host.

Description

docker save fails for image pulled using --platform=linux/arm64 on amd64 host.

I pulled image as, docker pull --platform=linux/arm64 nginx:latest

docker images --no-trunc
REPOSITORY   TAG       IMAGE ID                                                                  CREATED       SIZE
nginx        latest    sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3   5 weeks ago   273MB

docker save -o nginx.tar sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3**
Error response from daemon: unable to create manifests file: content digest sha256:88a0a069d5e9865fcaaf8c1e53ba6bf3d8d987b0fdc5e0135fec8ce8567d673e: not found

Not able to do docker save because of above error

Extra info, Although, docker save works with arm64 arch image if image is pulled using mainfest digest as below,

docker pull nginx:latest@sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171

docker images --no-trunc
REPOSITORY   TAG       IMAGE ID                                                                  CREATED       SIZE
nginx        <none>    sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171   5 weeks ago   273MB


 docker save -o nginx.tar sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171 - success

Reproduce

Same as above description.

Expected behavior

No response

docker version

docker version
Client:
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:13:00 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:13:08 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.8
  GitCommit:        7eba5930496d9bbe375fdf71603e610ad737d2b2
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
[root@localhost sha256]#

docker info

# docker info
Client:
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc.)
    Version:  v0.8.0
    Path:     /usr/libexec/docker/cli-plugins/docker-app
  buildx: Build with BuildKit (Docker Inc.)
    Version:  v0.3.1-tp-docker
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 4
 Server Version: 25.0.3
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7eba5930496d9bbe375fdf71603e610ad737d2b2
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.4.17-2136.323.8.2.el7uek.x86_64
 Operating System: Oracle Linux Server 7.9
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.544GiB
 Name: localhost.localdomain
 ID: 0844a04b-e797-4332-881c-74867bca5af0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTPS Proxy: http://10.115.27.54:3128/
 No Proxy: localhost,127.0.0.1,localaddress,.localdomain.com
 Username: [email protected]
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

[root@localhost sha256]#

Additional Info

No response

Sep 24 '24 12:09 chetanshivaji

Thanks for reporting! Yes, this is needs some work. I was curious if this was fixed on current master of the engine, and if PR would have fixed it;

https://github.com/moby/moby/pull/48295
(related to) https://github.com/docker/cli/pull/5331

But it looks like, while the error is slightly different 27.3 and "master" (looks like it now includes a NotFound prefix; possibly originating from containerd error-defs), the issue is still the same;

docker pull --platform=linux/s390x nginx:latest
latest: Pulling from library/nginx
a95d04829aec: Download complete
42b2ec246294: Download complete
95fe27c895a8: Download complete
6982f08f280c: Download complete
b2fff712f8b4: Download complete
1cdb42c01f48: Download complete
f9cf33a9c2ac: Download complete
Digest: sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3
Status: Downloaded newer image for nginx:latest
docker.io/library/nginx:latest

docker image ls --tree
WARNING: This is an experimental feature. The output may change and shouldn't be depended on.

IMAGE                   ID             DISK USAGE   CONTENT SIZE   IN USE
nginx:latest            04ba374043cc        246MB         65.3MB
├─ linux/s390x          0fa1ac852fea        246MB         65.3MB
├─ linux/amd64          88a0a069d5e9           0B             0B
├─ linux/arm/v5         6289301f3b91           0B             0B
├─ linux/arm/v7         d719acf8bcaf           0B             0B
├─ linux/arm64/v8       9f661996f4d1           0B             0B
├─ linux/386            67e126c43a8d           0B             0B
├─ linux/mips64le       83277e45a916           0B             0B
└─ linux/ppc64le        964b97f05849           0B             0B

docker save -o nginx.tar nginx:latest
Error response from daemon: unable to create manifests file: NotFound: content digest sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171: not found

daemon logs (current master) look like;

DEBU[2024-09-24T18:49:43.377261423Z] Calling HEAD /_ping                           spanID=c4ad8816f0c9fe7c traceID=d6ece5243ab000edd0dee3f6857257da
DEBU[2024-09-24T18:49:43.378616590Z] Calling GET /v1.48/images/get?names=nginx%3Alatest  spanID=0d7b5e44f6d43c8a traceID=d9a86fa6439b281f1f9320fbb2ffdcf8
DEBU[2024-09-24T18:49:43.385920632Z] export requested                              img="{docker.io/library/nginx:latest map[] {application/vnd.oci.image.index.v1+json sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3 10272 [] map[] [] <nil> } 2024-09-24 18:46:01.821534085 +0000 UTC 2024-09-24 18:46:01.821534085 +0000 UTC}" name="nginx:latest" resolveErr="<nil>" spanID=0d7b5e44f6d43c8a specificDigestResolved=false traceID=d9a86fa6439b281f1f9320fbb2ffdcf8
DEBU[2024-09-24T18:49:43.401433173Z] export image                                  name="docker.io/library/nginx:latest" spanID=0d7b5e44f6d43c8a target="{application/vnd.oci.image.index.v1+json sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3 10272 [] map[] [] <nil> }" traceID=d9a86fa6439b281f1f9320fbb2ffdcf8

The --platform flag (from https://github.com/docker/cli/pull/5331) helps in this case, but either the error message should be better (i.e., "we have the image, but not in the default platform"), or it should export only the available one;

docker save -o nginx.tar --platform=linux/s390x nginx:latest

ls -la nginx.tar
-rw------- 1 root root 65330176 Sep 24 18:47 nginx.tar

Sep 24 '24 18:09 thaJeztah

Also tried to find what the sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171 digest refers to, and it looks indeed the default (arm64) image;

docker buildx imagetools inspect nginx:latest
...
...
  Name:        docker.io/library/nginx:latest@sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/arm64/v8
  Annotations:
    org.opencontainers.image.url:             https://hub.docker.com/_/nginx
    org.opencontainers.image.version:         1.27.1
    com.docker.official-images.bashbrew.arch: arm64v8
    org.opencontainers.image.base.digest:     sha256:ace984fea4de21d4ad6d0047c3c6ab7ade9fcd9dee3e44c5fe6349d7dd063d86
    org.opencontainers.image.base.name:       debian:bookworm-slim
    org.opencontainers.image.created:         2024-09-05T12:38:34Z
    org.opencontainers.image.revision:        e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a
    org.opencontainers.image.source:          https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian

Full output;

docker buildx imagetools inspect nginx:latest
Name:      docker.io/library/nginx:latest
MediaType: application/vnd.oci.image.index.v1+json
Digest:    sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3

Manifests:
  Name:        docker.io/library/nginx:latest@sha256:88a0a069d5e9865fcaaf8c1e53ba6bf3d8d987b0fdc5e0135fec8ce8567d673e
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/amd64
  Annotations:
    org.opencontainers.image.created:         2024-09-04T23:10:40Z
    org.opencontainers.image.revision:        e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a
    org.opencontainers.image.source:          https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian
    org.opencontainers.image.url:             https://hub.docker.com/_/nginx
    org.opencontainers.image.version:         1.27.1
    com.docker.official-images.bashbrew.arch: amd64
    org.opencontainers.image.base.digest:     sha256:903d3225acecaa272bbdd7273c6c312c2af8b73644058838d23a8c9e6e5c82cf
    org.opencontainers.image.base.name:       debian:bookworm-slim

  Name:        docker.io/library/nginx:latest@sha256:a6c3d2371006c10c41f4d3be8db4561b8f21c7cda4e8a3ab464521a00c336826
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    com.docker.official-images.bashbrew.arch: amd64
    vnd.docker.reference.digest:              sha256:88a0a069d5e9865fcaaf8c1e53ba6bf3d8d987b0fdc5e0135fec8ce8567d673e
    vnd.docker.reference.type:                attestation-manifest

  Name:        docker.io/library/nginx:latest@sha256:6289301f3b918b98577fe0f80c13761aae5791b0059ed9e1d554edeae7c54b82
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/arm/v5
  Annotations:
    org.opencontainers.image.url:             https://hub.docker.com/_/nginx
    org.opencontainers.image.version:         1.27.0
    com.docker.official-images.bashbrew.arch: arm32v5
    org.opencontainers.image.base.digest:     sha256:aef22c7b68e2f7707fabdff982d4d4928ca261775bae0ddb7f7f8358ed0ef10f
    org.opencontainers.image.base.name:       debian:bookworm-slim
    org.opencontainers.image.created:         2024-08-13T10:44:11Z
    org.opencontainers.image.revision:        a6f7d140744f8b15ff4314b8718b3f022efc7f43
    org.opencontainers.image.source:          https://github.com/nginxinc/docker-nginx.git#a6f7d140744f8b15ff4314b8718b3f022efc7f43:mainline/debian

  Name:        docker.io/library/nginx:latest@sha256:3e576b43defa219d279961a042477c8df1b2e46d15ec8cd98827eae750e33fa6
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    com.docker.official-images.bashbrew.arch: arm32v5
    vnd.docker.reference.digest:              sha256:6289301f3b918b98577fe0f80c13761aae5791b0059ed9e1d554edeae7c54b82
    vnd.docker.reference.type:                attestation-manifest

  Name:        docker.io/library/nginx:latest@sha256:d719acf8bcaf35a0a81288ea2a60a0fcab0b4cd65a0b6f68cadb060c5861f16e
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/arm/v7
  Annotations:
    org.opencontainers.image.url:             https://hub.docker.com/_/nginx
    org.opencontainers.image.version:         1.27.1
    com.docker.official-images.bashbrew.arch: arm32v7
    org.opencontainers.image.base.digest:     sha256:756f2240fee673990dccc7bdd288792e149ce50072d5cd29d328e4cd80aa6979
    org.opencontainers.image.base.name:       debian:bookworm-slim
    org.opencontainers.image.created:         2024-09-05T04:28:40Z
    org.opencontainers.image.revision:        e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a
    org.opencontainers.image.source:          https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian

  Name:        docker.io/library/nginx:latest@sha256:3a254dd24b20ecb921ff418f880892562a5d11a4de7e08465ab8296e4bc6a327
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    com.docker.official-images.bashbrew.arch: arm32v7
    vnd.docker.reference.digest:              sha256:d719acf8bcaf35a0a81288ea2a60a0fcab0b4cd65a0b6f68cadb060c5861f16e
    vnd.docker.reference.type:                attestation-manifest

  Name:        docker.io/library/nginx:latest@sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/arm64/v8
  Annotations:
    org.opencontainers.image.url:             https://hub.docker.com/_/nginx
    org.opencontainers.image.version:         1.27.1
    com.docker.official-images.bashbrew.arch: arm64v8
    org.opencontainers.image.base.digest:     sha256:ace984fea4de21d4ad6d0047c3c6ab7ade9fcd9dee3e44c5fe6349d7dd063d86
    org.opencontainers.image.base.name:       debian:bookworm-slim
    org.opencontainers.image.created:         2024-09-05T12:38:34Z
    org.opencontainers.image.revision:        e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a
    org.opencontainers.image.source:          https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian

  Name:        docker.io/library/nginx:latest@sha256:0df868782671055c19e022b02b723375891f278fb785c0a89b4107b815c3ca1c
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    vnd.docker.reference.type:                attestation-manifest
    com.docker.official-images.bashbrew.arch: arm64v8
    vnd.docker.reference.digest:              sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171

  Name:        docker.io/library/nginx:latest@sha256:67e126c43a8d738f2d05a7a7baa94a26b4d3f3f4cfb70b1aa49a52e700ef0709
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/386
  Annotations:
    org.opencontainers.image.url:             https://hub.docker.com/_/nginx
    org.opencontainers.image.version:         1.27.1
    com.docker.official-images.bashbrew.arch: i386
    org.opencontainers.image.base.digest:     sha256:a09e6f30093763451525093f6c1f425fcb751de9afc5c7944a28e4e48250f4c9
    org.opencontainers.image.base.name:       debian:bookworm-slim
    org.opencontainers.image.created:         2024-09-05T00:14:49Z
    org.opencontainers.image.revision:        e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a
    org.opencontainers.image.source:          https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian

  Name:        docker.io/library/nginx:latest@sha256:96d7658c6592eee20fcb834cc3319cb7cbdaf487614132c3ff4da35b63e3baa9
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    vnd.docker.reference.type:                attestation-manifest
    com.docker.official-images.bashbrew.arch: i386
    vnd.docker.reference.digest:              sha256:67e126c43a8d738f2d05a7a7baa94a26b4d3f3f4cfb70b1aa49a52e700ef0709

  Name:        docker.io/library/nginx:latest@sha256:83277e45a9163e419c5e676f23498233b4e8f283daed1174db46a6e2d7860efa
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/mips64le
  Annotations:
    org.opencontainers.image.base.name:       debian:bookworm-slim
    org.opencontainers.image.created:         2024-09-05T11:19:25Z
    org.opencontainers.image.revision:        e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a
    org.opencontainers.image.source:          https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian
    org.opencontainers.image.url:             https://hub.docker.com/_/nginx
    org.opencontainers.image.version:         1.27.1
    com.docker.official-images.bashbrew.arch: mips64le
    org.opencontainers.image.base.digest:     sha256:460b58fe46216c353ee64696b20a853678b1a08da92a64a1caf5e6faf997b329

  Name:        docker.io/library/nginx:latest@sha256:6f8fb2d834e6f7d91039dd2ddb653b809cfdb60f2646d839d17c714d8a2e2b9b
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    com.docker.official-images.bashbrew.arch: mips64le
    vnd.docker.reference.digest:              sha256:83277e45a9163e419c5e676f23498233b4e8f283daed1174db46a6e2d7860efa
    vnd.docker.reference.type:                attestation-manifest

  Name:        docker.io/library/nginx:latest@sha256:964b97f05849b3ca619df5bea76d98b53196260cf8cd9b6c03ccbb8393dc8613
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/ppc64le
  Annotations:
    com.docker.official-images.bashbrew.arch: ppc64le
    org.opencontainers.image.base.digest:     sha256:f39559bf405679e55f74c361f383e0cfe19b01934a53eca2149ddd966604293b
    org.opencontainers.image.base.name:       debian:bookworm-slim
    org.opencontainers.image.created:         2024-09-05T00:43:55Z
    org.opencontainers.image.revision:        e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a
    org.opencontainers.image.source:          https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian
    org.opencontainers.image.url:             https://hub.docker.com/_/nginx
    org.opencontainers.image.version:         1.27.1

  Name:        docker.io/library/nginx:latest@sha256:a72749cca16dd47485c9220a98b74b04d6f564ef8ec6c189e85c56e04c7e4dc6
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    com.docker.official-images.bashbrew.arch: ppc64le
    vnd.docker.reference.digest:              sha256:964b97f05849b3ca619df5bea76d98b53196260cf8cd9b6c03ccbb8393dc8613
    vnd.docker.reference.type:                attestation-manifest

  Name:        docker.io/library/nginx:latest@sha256:0fa1ac852feac85b15bdec82773c1b19eafdde054536e2f7f4c05cecbff96433
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/s390x
  Annotations:
    com.docker.official-images.bashbrew.arch: s390x
    org.opencontainers.image.base.digest:     sha256:d11dd86eb4857cdeac7b99af3884fa13e8a3d18ce550f77cee3cca64a735f233
    org.opencontainers.image.base.name:       debian:bookworm-slim
    org.opencontainers.image.created:         2024-09-05T02:49:36Z
    org.opencontainers.image.revision:        e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a
    org.opencontainers.image.source:          https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian
    org.opencontainers.image.url:             https://hub.docker.com/_/nginx
    org.opencontainers.image.version:         1.27.1

  Name:        docker.io/library/nginx:latest@sha256:7df7564b1cfdd1f098b8ffb3d91a91ed248a1d54826a8260b52ff225b70cc353
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    com.docker.official-images.bashbrew.arch: s390x
    vnd.docker.reference.digest:              sha256:0fa1ac852feac85b15bdec82773c1b19eafdde054536e2f7f4c05cecbff96433
    vnd.docker.reference.type:                attestation-manifest

Sep 24 '24 19:09 thaJeztah

Thanks for the details @thaJeztah.

This seems to be an issue with image pulls with --platform mismatching to host arch.

docker inspect 04ba374043cc|grep -ani arch

74:        "Architecture": "386",

docker images --no-trunc

REPOSITORY            TAG       IMAGE ID                                                                  CREATED             SIZE
nginx                 latest    sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3   5 weeks ago         252MB

docker save -o nginx386arch.tar sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3

Error response from daemon: unable to create manifests file: content digest sha256:88a0a069d5e9865fcaaf8c1e53ba6bf3d8d987b0fdc5e0135fec8ce8567d673e: not found

Sep 25 '24 14:09 chetanshivaji

@chetanshivaji yes, correct; the issue is that the image pulled does not match the default platform. When running docker save, the initial part "succeeds" (multi-arch image exists), but it then tries to save the default (native) platform, and that is missing.

So at least the error-message should be more clear, but we should also look if the request is non-ambiguous (i.e., image exists, and there's only a single platform locally -> in that case we could save the image, regardless if it's the default platform).

Sep 25 '24 15:09 thaJeztah

x-posting here from https://github.com/docker/cli/issues/5493#issuecomment-2391039818, because I somewhat assumed that this never worked with the containerd image store enabled, but it looks like older versions did handle this case.

With the added note that;

It's still possible that this was an intentional change in behavior (i.e., default to native platform instead of "pick whatever is found"), but at least the error-reporting should be more clear on that if that's the case.

docker version --format '{{.Server.Version}}'
# 24.0.9

docker info --format '{{ .Driver }} {{ .DriverStatus }}'
# overlayfs [[driver-type io.containerd.snapshotter.v1]]

docker pull --platform=linux/amd64 alpine

docker images
# REPOSITORY   TAG       IMAGE ID       CREATED         SIZE
# alpine       latest    beefdbd8a1da   4 minutes ago   12.1MB

docker save -o foo.tar beefdbd8a1da

docker save beefdbd8a1da | tar -tf-
# blobs/
# blobs/sha256/
# blobs/sha256/33735bd63cf84d7e388d9f6d297d348c523c044410f553bd878c6d7829612735
# blobs/sha256/43c4264eed91be63b206e17d93e75256a6097070ce643c5e8f0379998b44f170
# blobs/sha256/91ef0af61f39ece4d6710e465df5ed6ca12112358344fd51ae6a3b886634148b
# index.json
# manifest.json
# oci-layout

Oct 03 '24 10:10 thaJeztah

I think the problem may be in this area; https://github.com/moby/moby/blob/2d049d11a963e6721d1e65f22baf43ede4e518d0/daemon/containerd/image_exporter.go#L56-L67

// This makes the exported archive also include `manifest.json`
// when the image is a manifest list. It is needed for backwards
// compatibility with Docker image format.
// The containerd will choose only one manifest for the `manifest.json`.
// Our preference is to have it point to the default platform.
// Example:
//  Daemon is running on linux/arm64
//  When we export linux/amd64 and linux/arm64, manifest.json will point to linux/arm64.
//  When we export linux/amd64 only, manifest.json will point to linux/amd64.
// Note: This only matters when importing this archive into non-containerd Docker.
// Importing the same archive into containerd, will not restrict the platforms.
archive.WithPlatform(pm),

For backwards-compatiblity, it's expected to have a manifest.json, which defaults to the native platform. In this case, it may find the manifest, but it's empty (?), or at least, the list of records is non-empty, and it may now expect the native platform to be there (in this case it isn't), and fail because of that; https://github.com/moby/moby/blob/2d049d11a963e6721d1e65f22baf43ede4e518d0/vendor/github.com/containerd/containerd/images/archive/exporter.go#L336-L350

if !eo.skipDockerManifest && len(dManifests) > 0 {
    tr, err := manifestsRecord(ctx, store, dManifests)
    if err != nil {
        return fmt.Errorf("unable to create manifests file: %w", err)
    }

    records = append(records, tr)
}

if len(algorithms) > 0 {
    records = append(records, directoryRecord("blobs/", 0755))
    for alg := range algorithms {
        records = append(records, directoryRecord("blobs/"+alg+"/", 0755))
    }
}

Oct 07 '24 14:10 thaJeztah

If the native platform is not available, the pm matcher will match any platform. The WithPlatform was specifically added to handle this situation, but looks like something changed (on our side or containerd). Will take a look.

Oct 22 '24 09:10 vvoland

Any updates? I have same problem.

I use MacOS arm64. I can download and run this linux/amd64 container, but i can't save it.

  ~  INSERT  docker save -o milvus-2.5.9-linux-amd64.tar  milvusdb/milvus:v2.5.9                                                                                           11:23:16 
Error response from daemon: unable to create manifests file: NotFound: content digest sha256:c12c50588b890eec30910a54c3f3b1a963b8452dd2b73b3dc3d2db96d8b08e86: not found

  ~  INSERT  docker run milvusdb/milvus:v2.5.9                                                                                                                         1 ↵  11:23:19 
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
tini (tini version 0.19.0 - git.de40ad0)
Usage: tini [OPTIONS] PROGRAM -- [ARGS] | --version

Execute a program under the supervision of a valid init process (tini)

Command line options:

  --version: Show version and exit.
  -h: Show this help message and exit.
  -s: Register as a process subreaper (requires Linux >= 3.4).
  -p SIGNAL: Trigger SIGNAL when parent dies, e.g. "-p SIGKILL".
  -v: Generate more verbose output. Repeat up to 3 times.
  -w: Print a warning when processes are getting reaped.
  -g: Send signals to the child's process group.
  -e EXIT_CODE: Remap EXIT_CODE (from 0 to 255) to 0.
  -l: Show license and exit.

Environment variables:

  TINI_SUBREAPER: Register as a process subreaper (requires Linux >= 3.4).
  TINI_VERBOSITY: Set the verbosity level (default: 1).
  TINI_KILL_PROCESS_GROUP: Send signals to the child's process group.

version info

Client:
 Version:           28.0.4
 API version:       1.48
 Go version:        go1.23.7
 Git commit:        b8034c0
 Built:             Tue Mar 25 15:06:09 2025
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.40.0 (187762)
 Engine:
  Version:          28.0.4
  API version:      1.48 (minimum version 1.24)
  Go version:       go1.23.7
  Git commit:       6430e49
  Built:            Tue Mar 25 15:07:18 2025
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.7.26
  GitCommit:        753481ec61c7c8955a23d6ff7bc8e4daed455734
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Apr 14 '25 03:04 JrDw0

Temporarily please specify the platform explicitly when saving:

docker save --platform linux/amd64 -o milvus-2.5.9-linux-amd64.tar  milvusdb/milvus:v2.5.9

Apr 14 '25 09:04 vvoland

Took a crack at this.

PR with fix here: https://github.com/containerd/containerd/pull/11980

Jun 13 '25 19:06 ctalledo

cli cli copied to clipboard

Docker Containerd snapshotter Overlayfs, docker save fails for image pulled using --platform=linux/arm64 on amd64 host.

Description

Reproduce

Expected behavior

docker version

docker info

Additional Info

cli
cli copied to clipboard