Unknown media type during manifest conversion: "application/vnd.docker.image.rootfs.diff.tar.gzip"

[crisbal]

Description

Steps to reproduce the issue:

I have an image built with Kaniko in CI, that I am trying to use as base image for another that I am building locally with podman build.

The local image has the following Containerfile

FROM ci-registry.company.com/image_build_with_kaniko
RUN echo "Hello"

If I build the image with podman build -f Containerfile -t tmp-image . I get the following error: (Adding --format=docker gets the same error)

Trying to pull ci-registry.company.com/image_build_with_kaniko
Getting image source signatures
Copying blob c408bb006db1 done  
Copying blob 60c686673fbf done  
Copying blob 46da271691fa done  
Copying blob 1aa347c632b6 done  
Copying config 04ee292f7a done  
Writing manifest to image destination
Storing signatures
Error: error creating build container: error preparing image configuration: error converting image 
"containers-storage:[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@04ee292f7a5549e765c99205acc567738a09eb084409cd71f6600facd3743c51" 
from "application/vnd.oci.image.manifest.v1+json" to "application/vnd.docker.distribution.manifest.v2+json": 
Unknown media type during manifest conversion: "application/vnd.docker.image.rootfs.diff.tar.gzip"

Describe the results you expected:

The build should continue, maybe? What is going wrong here?

Output of podman version if reporting a podman build issue:

3.4.4-1

Output of cat /etc/*release:

cat /etc/os-release
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"

Output of uname -a:

Linux PC 5.4.72-microsoft-standard-WSL2 #1 SMP Wed Oct 28 23:40:43 UTC 2020 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

Did not touch this / Default one

[vrothberg]

Thanks for reaching out, could you share the output of the following command? skopeo inspect --raw docker://ci-registry.company.com/image_build_with_kaniko | jq .

That will print the image's manifest.

[crisbal]

Here it is:

{
  "schemaVersion": 2,
  "mediaType": "",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "size": 1778,
    "digest": "sha256:04ee292f7a5549e765c99205acc567738a09eb084409cd71f6600facd3743c51"
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 64239542,
      "digest": "sha256:60c686673fbf60314729fe8eacc2b514df4f3235cfe77642b81413b989ba6abe"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 268790402,
      "digest": "sha256:1aa347c632b602157e2e5b521ddb459bf82084746bcc8a4b4689c361dfb57348"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 238,
      "digest": "sha256:c408bb006db1d5cee3f5e30b7b864c471e23c5c5ea1b703f787c499eb82fd2d4"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 12903731,
      "digest": "sha256:46da271691fa1d20103f0c04a60028ab50ec59cac5e28696fb418fc45d41e7a8"
    }
  ]
}

[vrothberg]

Thank you! That's what I suspected. The image claims to be an OCI one but lists Docker layers (application/vnd.docker.image.rootfs.diff.tar.gzip) which I think doesn't conform with the OCI image spec.

@nalind WDYT?

[crisbal]

@vrothberg

The ci-registry.company.com/image_build_with_kaniko Dockerfile has this structure (some minor things omitted that hopefully should not matter):

FROM ubuntu:20.04 AS installer

WORKDIR /installer
ARG INSTALLER
COPY $INSTALLER .
RUN bash $INSTALLER /opt/application/1.0.0

################################################################################

ARG BASE_IMAGE
FROM ${BASE_IMAGE}

COPY --from=installer /opt/application /opt/application
RUN ln -s /opt/application/1.0.0 /opt/application/stable

RUN echo "source /opt/application/stable/activate.sh" >> /root/.bashrc
# non-oci! --format=docker required
SHELL ["/bin/bash", "-c"]
CMD ["/bin/bash"]

[vrothberg]

Thanks, @crisbal.

I suggest reaching out our friends over at kaniko to resolve the issue; the layers should be converted to OCI ones during build or when pushing to the registry.

The OCI image spec states that "an encountered mediaType that is unknown to the implementation MUST be ignored", but that is mostly for supporting OCI artifacts I presume. In this case (i.e., as a base image during build), where it's clear that those are layers I think that Buildah is right to reject the image.

[vrothberg]

Cc @mtrmac @rhatdan

[nalind]

Since we build the manifest from scratch whenever we commit an image, we shouldn't be producing manifests that mark layers with types from different specs. I'm not opposed to just accepting every layer blob type that's known to us (this may turn out to be another strictness check that we added to the image library that we need to walk back) if it's truly as simple as adding more case clauses in the manifest conversion logic.

[crisbal]

Thanks, I reported it over to Kaniko, should I close this issue or it is something that you still want to fix on your side?

[mtrmac]

Since we build the manifest from scratch whenever we commit an image, we shouldn't be producing manifests that mark layers with types from different specs

This is not just a build issue; a skopeo copy --format … can trigger the same code conversion path, which will fail on such MIME type mixes the same way.

At least this particular code path (c/image/image/$type1.convertToManifest$type2) can be trivially taught to handle more MIME types; I’d need to take a deeper look about possible variants of this situation that involve also compression/decompression.

Let’s see what Kaniko says, hopefully these images will turn out to be so rare that we can just continue rejecting them.

[crisbal]

hopefully these images will turn out to be so rare that we can just continue rejecting them

It looks like to me that the issue can be triggered easily but at the same time I wonder why it never came out, it is possible it is just a weird combination of steps.

So it is just:

• Have a base image that has MediaType OCI
• Build on top of the base image an intermediate image with Kaniko (this will get MediaType OCI and Docker Layers)
• Attempt to build an image using as base the intermediate Kaniko image with buildah build

[remivoirin]

Thanks @crisbal for reporting directly to Kaniko. Do you think this is directly related and should be added to the issue on their side? We have images built with Kaniko that can be run by Podman 3.3.1 on CentOS Stream 8, but not pushed to another registry because of this.

$ skopeo inspect --raw docker://<kaniko built image> | jq .
{
  "schemaVersion": 2,
  "mediaType": "",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "size": 2229,
    "digest": "sha256:1d9384a1e8cf5636c4c525b02d3eb2c5e2a6300987717cf00b798db54aabd955"
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 79848180,
      "digest": "sha256:78846bc60c09f099fe07532fd402aff11d9704b96310c7bcf0a7ee20085774a1"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 334659526,
      "digest": "sha256:e8430b5e4f32476399aedbebd2bce8e31406d9f53e61bce5149b0f77fc3e11df"
    },
...

When trying to push:

$ podman push <kaniko built image>
Getting image source signatures
Copying blob 59a6252e61ff [--------------------------------------] 0.0b / 0.0b
Error: creating an updated image manifest: preparing updated manifest,
layer "sha256:b208550f9bbbc279a3ea5d174d86f9aadd31da9ae83f3496e3e92d2ea864cef6":
unsupported MIME type for compression: application/vnd.docker.image.rootfs.diff.tar.gzip

[crisbal]

Hey @remivoirin, the skopeo inspect part shows the same issue as my case. I will let the experts speak about the podman push but it looks related to me.

Anyway you could join into the issue on Kaniko repo so maybe more noise helps in getting the issue sorted.

[remivoirin]

Thanks, will do!

[mtrmac]

We have images built with Kaniko that can be run by Podman 3.3.1 on CentOS Stream 8, but not pushed to another registry because of this.

Yes, that’s the same underlying cause. The Kaniko-built images are inconsistent, and basically any kind of consumption (not just builds using Buildah) could, in principle, cause them to be rejected. (The actual implementation only rejects them on some code paths, but that’s just an implementation detail, not a maintained feature.)

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@nalind @mtrmac @vrothberg @crisbal What should we do with this issue?

[mtrmac]

I dunno. https://github.com/GoogleContainerTools/kaniko/issues/1836 is labeled “help wanted”, so one or both of:

• Contribute a fix in Kaniko
• Modify c/image to accept (and canonicalize??) such images.

Obviously doing neither, and hoping Kaniko fixes it without help, is attractive.

[crisbal]

I would be happy to help but unfortunately I don't think I am skilled enough with the languages, concepts and technologies involved.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[K-A-R-I-M]

I found a potential solution to solve this issue.

If you build your base image with podman build -t base_image . as usual, you will obtain an OCI mediaType manifest.

{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:a5d8d05897e1e276091ddedaf52f71a9de72c79a0fba215e9d6a0c3197cbd2c2",
    "size": 1553
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar",
      "digest": "sha256:59880d7ef6d202374f5f4775b63d61a1e2930c432c3a4f08237ed0f7f0c8e704",
      "size": 165120000
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar",
      "digest": "sha256:a11181b4bd92e5bf3c9589104bafa2e147d4cc5ee6b6c7f68a8d0baf7b1454cc",
      "size": 154247680
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar",
      "digest": "sha256:869d2f57befc8c54774604bb3646484f3e5691d4b1ff1da24b4e89f51573f03e",
      "size": 1978880
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar",
      "digest": "sha256:151e4cb91d30564ead5de52542bc9b40e2eb71c54b48fc394d5eebb0586c6353",
      "size": 251123200
    }
  ]
}

then you do:

podman push -f v2s2 registry.company.ch/base_image

when you push it to your registry.

This will allow you to have a Docker mediaType manifest.

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1555,
    "digest": "sha256:a78f1cad57997e34791a23c64c031d0221cc6420fe5f284acfd1b369c9a4dda3"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 64239979,
      "digest": "sha256:07132c1cc1a35ac077e1e2dd3de6875ebb1c74940398b56de41f214d6b3ebf51"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 97080546,
      "digest": "sha256:d10c72bf83d8e121602feee8a3b2a525c6426ce0b8058ac417182c78c7142444"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 515713,
      "digest": "sha256:86d3ca747a85bba71e083de35aaa72aa9ecc47c379f9664d26643a4c3e7e37be"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 122365225,
      "digest": "sha256:88e39919aa9eb54475ed8b43fc2d034642f91a2dba5b04a822ea25926d1ed490"
    }
  ]
}

which will no more be a problem for Kaniko and its way of adding Docker mediaType layers in your manifest.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Since you found a solution for this problem, I am closing the issue, reopen if this is a mistake.