Error "Manifest does not match provided manifest digest" when using @sha256:digest in docker-daemon

[airadier]

I reported this in skopeo (see https://github.com/containers/skopeo/issues/1051) but the fix might be required in containers/image instead.

When running a skopeo inspect docker-daemon:repo@sha256:digest it fails with a "Manifest does not match provided manifest digest" error.

After some tests, the cause might be that the contents of the manifest are different (testing with skopeo inspect --raw for docker:// source and for docker-daemon: source. My guess in here is that for docker-daemon: source the manifest is automatically generated by Skopeo or the underlying libraries, while for docker:// source, the manifest is downloaded "as is". The docker:// version, when retrieved from the registry, is indented and formatted, so the hash differs with the hash of the locally generated manifest. See my tests:

airadier@ender:~$ docker pull alpine@sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65
sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65: Pulling from library/alpine
df20fa9351a1: Pull complete
Digest: sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65
Status: Downloaded newer image for alpine@sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65

airadier@ender:~$ ./skopeo inspect docker-daemon:alpine@sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65
FATA[0000] Error parsing manifest for image: Manifest does not match provided manifest digest sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65

airadier@ender:~$ ./skopeo inspect --raw docker-daemon:alpine@sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65
{"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.v2+json","config":{"mediaType":"application/vnd.docker.container.image.v1+json","size":1509,"digest":"sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e"},"layers":[{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":5844992,"digest":"sha256:50644c29ef5a27c9a40c393a73ece2479de78325cae7d762ef3cdc19bf42dd0a"}]}

airadier@ender:~$ ./skopeo inspect --raw docker-daemon:alpine@sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65 | shasum -a 256
0fe895ba713b184489c17550d67464dd67fade3f92f3f0c1904dc8df6750d609  -

airadier@ender:~$ ./skopeo inspect --raw docker://alpine@sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1509,
      "digest": "sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2797541,
         "digest": "sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c"
      }
   ]
}

airadier@ender:~$ ./skopeo inspect --raw docker://alpine@sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65 | shasum -a 256
a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65  -

I would expect that it is possible to do a skopeo inspect docker-daemon:repo@sha256:digest instead of failing. So some proposals would be:

• Fix the manifest generation, so it matches the remote manifest format. I am not sure if this is in general a good idea. Is there an standard for manifest formatting?
• Skip the check that seems to be done in https://github.com/containers/image/blob/21244c96ad792ef415068dc1bc1ab82dffb68dc3/image/unparsed.go#L59, as the digest for the manifest generated on the fly won't probably match the real manifest in the remote repository.

The issue also applies for other commands like skopeo copy:

kopeo copy docker-daemon:alpine@sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65 oci:test.tar
FATA[0000] Error determining manifest MIME type for docker-daemon:alpine@sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65: Manifest does not match provided manifest digest sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd6

and anything using FromUnparsedImage (https://github.com/containers/image/blob/1a0dda734e3bed323a9674e274b379cf841fe8b1/image/sourced.go#L68) which in turn uses:

https://github.com/containers/image/blob/21244c96ad792ef415068dc1bc1ab82dffb68dc3/image/unparsed.go#L59

[airadier]

I can see that not only the format of the auto-generated digest differs from the real manifest. Also the layer contents and digest.

i.e. locally generated:

"layers":[{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":5844992,"digest":"sha256:50644c29ef5a27c9a40c393a73ece2479de78325cae7d762ef3cdc19bf42dd0a"}]

real manifest:

   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2797541,
         "digest": "sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c"
      }
   ]

Probably because the locally generated version is using the uncompressed version of the layer (is the mediaType correct?) and the real manifest is using the distribution .tar.gz layer, the digest of the blob in the remote registry.

So, as this is expected, does it make sense to enable the manifest digest verification when using the docker-daemon:// source?

[mtrmac]

Thanks for your report. You’re right, the original compressed representation can’t be regenerated.