layer is not pulled when using the 'nerdctl pull'

[yankay]

Description

Relate to the https://github.com/containerd/nerdctl/issues/2327

The content digest sha256:xxx: not found is becuase of the nerdctl does not pull the layer.

In my environment:

/usr/local/bin/nerdctl --namespace=nerdctl-test  save -o /tmp/haha.tar ghcr.io/stargz-containers/alpine:3.13-org
FATA[0000] failed to get reader: content digest sha256:ca3cd42a7c9525f6ce3d64c1a70982613a8235f0cc057ec9244052921853ef15: not found

And try to pull with nerdctl

root@kay201:~/oss/containerd# nerdctl  -n nerdctl-test pull ghcr.io/stargz-containers/alpine:3.13
-org
ghcr.io/stargz-containers/alpine:3.13-org:                                        resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:ec14c7992a97fc11425907e908340c6c3d6ff602f5f13d899e6b7027c9b4133a:    exists         |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:e103c1b4bf019dc290bcc7aca538dc2bf7a9d0fc836e186f5fa34945c5168310: exists         |++++++++++++++++++++++++++++++++++++++|
config-sha256:49f356fa4513676c5e22e3a8404aad6c7262cc7aaed15341458265320786c58c:   exists         |++++++++++++++++++++++++++++++++++++++|
elapsed: 2.9 s                                                                    total:   0.0 B (0.0 B/s)

It skips the layer file. And the content digest is still not found.

And then try to pull with ctrctl

root@kay201:~/oss/containerd#  ctr  -n nerdctl-test i pull ghcr.io/stargz-containers/alpine:3.13-org
ghcr.io/stargz-containers/alpine:3.13-org:                                        resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:ec14c7992a97fc11425907e908340c6c3d6ff602f5f13d899e6b7027c9b4133a:    exists         |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:e103c1b4bf019dc290bcc7aca538dc2bf7a9d0fc836e186f5fa34945c5168310: exists         |++++++++++++++++++++++++++++++++++++++|
config-sha256:49f356fa4513676c5e22e3a8404aad6c7262cc7aaed15341458265320786c58c:   exists         |++++++++++++++++++++++++++++++++++++++|
layer-sha256:ca3cd42a7c9525f6ce3d64c1a70982613a8235f0cc057ec9244052921853ef15:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 1.7 s                                                                    total:  2.7 Mi (1.6 MiB/s)
unpacking linux/amd64 sha256:ec14c7992a97fc11425907e908340c6c3d6ff602f5f13d899e6b7027c9b4133a...
done: 9.968427ms

The layer file can be pulled and the content digest can be found.

Steps to reproduce the issue

1. nerdctl -n nerdctl-test pull ghcr.io/stargz-containers/alpine:3.13-org
2. ctr -n nerdctl-test content rm sha256:ca3cd42a7c9525f6ce3d64c1a70982613a8235f0cc057ec9244052921853ef15
3. nerdctl -n nerdctl-test pull ghcr.io/stargz-containers/alpine:3.13-org
4. nerdctl --namespace=nerdctl-test save -o /tmp/haha.tar ghcr.io/stargz-containers/alpine:3.13-org

FATA[0000] failed to get reader: content digest sha256:ca3cd42a7c9525f6ce3d64c1a70982613a8235f0cc057ec9244052921853ef15: not found

Describe the results you received and expected

The content digest exist, like using the ctrctl

What version of nerdctl are you using?

root@kay201:~/oss/containerd# nerdctl version Client: Version: v1.4.0-82-g5195b20.m OS/Arch: linux/amd64 Git commit: 5195b2096db36d7ca6e09ae8028d7e018d2d83ff.m buildctl: Version: v0.11.6 GitCommit: 2951a28cd7085eb18979b1f710678623d94ed578

Server: containerd: Version: v1.7.1 GitCommit: 1677a17964311325ed1c31e2c0a3589ce6d5c30d runc: Version: 1.1.7 GitCommit: v1.1.7-0-g860f061b

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

No response

[manishmalik]

Any update on this? Seems like I am facing similar issue. Any workaround to pull the image again preferably via nerdctl?

[AkihiroSuda]

ghcr.io/stargz-containers/alpine:3.13-org

Is this issue specific to ghcr.io/stargz-containers? Does docker.io/library/alpine work?

cc @ktock

[yankay]

ghcr.io/stargz-containers/alpine:3.13-org

Is this issue specific to ghcr.io/stargz-containers? Does docker.io/library/alpine work?

cc @ktock

Hi @AkihiroSuda

It's the same with docker.io/library/alpine

The reproduce methods:

nerdctl -n nerdctl-test pull alpine:3.18.4
ctr -n nerdctl-test content rm sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa
nerdctl -n nerdctl-test pull alpine:3.18.4
nerdctl --namespace=nerdctl-test save -o /tmp/haha.tar alpine:3.18.4

[halohsu]

I also have the same problem.

I downloaded the cephcsi image from Docker on my local macOS in order to use it in the production cluster. I then uploaded it through SFTP and it was used by Containerd. It worked!