containerd fix:save image err because content digest not found

image 1

"Manifest": {
            "schemaVersion": 2,
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "config": {
                "mediaType": "application/vnd.docker.container.image.v1+json",
                "digest": "sha256:141ba1659ac87bc83c33fb756ec91e105f5d196672a8f56c99eda2aa50c7f128",
                "size": 5668
            },
            "layers": [
                {
                    "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                    "digest": "sha256:3d0bcbd3ba78828654b1de97afb12337e1f2f01b8fd4e18bb4e0befc0cab35e2",
                    "size": 3756561
                },
                {
                    "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                    "digest": "sha256:5241a4e84f4eb9a11685a4b5e4a1d451512e92d70277ad077224eebb57bf1499",
                    "size": 4743936
                },

            "diff_ids": [
                    "sha256:7c76c8b384d5823f659d49db27f4e880ed1102c76e717160b9085855233135c8",
                    "sha256:bc20b96670742fb3960fa29572bd477bf72892d4ba702fa7943f5c9d74032327",
                ]

image 2

"Manifest": {
            "schemaVersion": 2,
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "config": {
                "mediaType": "application/vnd.docker.container.image.v1+json",
                "digest": "sha256:9f8f2ee9ddac537d25f24b49936115bf9a4f93ee3d004e7256daae9ace83567e",
                "size": 4124
            },

            "layers": [
                {
                    "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                    "digest": "sha256:f904ad7dc96d0e260d9b0b77f50ae0ba638cbd327d24ead122353d4fe9b0d0a2",
                    "size": 3629086

                },
                {
                    "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                    "digest": "sha256:2a5f6f535b245c6a1b6046cc44b43b2389fcdb5b35332be58f3f113ecc8e0b4d",
                    "size": 80143

                },
                "diff_ids": [
                    "sha256:7c76c8b384d5823f659d49db27f4e880ed1102c76e717160b9085855233135c8",
                    "sha256:91a32fe855d4fba03c5c811a4758219067ab85b0a27a41e7d37b345adf028cd0",
                ]

image1 blob sha256:3d0bcbd3ba78828654b1de97afb12337e1f2f01b8fd4e18bb4e0befc0cab35e2 and image 2 blob: sha256:f904ad7dc96d0e260d9b0b77f50ae0ba638cbd327d24ead122353d4fe9b0d0a2 have same chainid/diffid 7c76c8b384d5823f659d49db27f4e880ed1102c76e717160b9085855233135c8

if down load image 1 and then down load image 2 blob sha256:9f8f2ee9ddac537d25f24b49936115bf9a4f93ee3d004e7256daae9ace83567e will skip to be download. we can create container successfully, but save image 2 will show blob is lost.

nerdctl 1.7

./nerdctl  -n k8s.io save  test:v1 -o 123.tar 
FATA[0001] failed to get reader: content digest sha256:f904ad7dc96d0e260d9b0b77f50ae0ba638cbd327d24ead122353d4fe9b0d0a2: not found

@AkihiroSuda @apostasie

Jun 18 '25 11:06 ningmingxiao

Hi @ningmingxiao. Thanks for your PR.

I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Jun 18 '25 11:06 k8s-ci-robot

@dmcgowan @mxpv @mikebrow would you mind taking a look at this PR?

Jun 20 '25 02:06 ningmingxiao

It seems to have the similar logic as shown in the PR https://github.com/containerd/containerd/pull/8877

Aug 06 '25 08:08 ChengyuZhu6

Yes! I'm not sure whether it is a bug, at least we should add some log. I create a new pr https://github.com/containerd/containerd/pull/12140. use nerdctl new version commit will auto download it.

Aug 07 '25 08:08 ningmingxiao

Hi everyone, I think this is a very impactful issue on usage. Can we fix it?

Sep 15 '25 06:09 reckless-huang

you can use nerdctl new version to save image @reckless-huang

Sep 16 '25 16:09 ningmingxiao

You can use the new version of nerdctl to save image @reckless-huang

The new version still has the same issue. Most of the time I save an image because it no longer exists in the repository, so I need to export or retag it. But now, whether I do either, it will re-fetch the metadata and then fail

Sep 17 '25 03:09 reckless-huang

It's strange that Docker does not re-fetch manifest and other information during retag and save operations

Sep 17 '25 03:09 reckless-huang

docker will save all blob info @reckless-huang

Sep 17 '25 03:09 ningmingxiao

Docker will save all blob info @reckless-huang

These strange operations of containerd are actually not that convenient

Sep 17 '25 03:09 reckless-huang

Additionally, containerd should have pulled all the blobs. The issue is that even though I have all the blobs locally, it still tries to fetch metadata. Can we provide an option to allow retagging and saving when metadata cannot be obtained?

Sep 17 '25 03:09 reckless-huang

Sorry, we're here in containerd, I should submit an issue in nerdctl

Sep 17 '25 04:09 reckless-huang

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Oct 25 '25 13:10 k8s-ci-robot