containerd
cannot run image based on sha (specific to local unpushed images)
Description
I have standard lima with nerdctl, default image. I load in a simple image:
$ nerdctl image ls
REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
abc foo 28eec4daefa7 About a minute ago linux/arm64 1.8 MiB 940.9 KiB
I try to run the image using the name:tag, that works:
$ nerdctl run --rm -it abc:foo /bin/sh
#
It also works using the canonical name:
$ nerdctl run --rm -it docker.io/library/abc:foo /bin/sh
#
But it fails if I use the sha, whether with or without the tag:
$ nerdctl run --rm docker.io/library/abc:foo@sha256:28eec4daefa7838fa3b8638eff27de54d7cb31552730d28cc6c030ecde962836 /bin/sh
docker.io/library/abc@sha256:28eec4daefa7838fa3b8638eff27de54d7cb31552730d28cc6c030ecde962836: resolving |--------------------------------------|
elapsed: 1.1 s total: 0.0 B (0.0 B/s)
docker.io/library/abc@sha256:28eec4daefa7838fa3b8638eff27de54d7cb31552730d28cc6c030ecde962836: resolving |--------------------------------------|
elapsed: 2.2 s total: 0.0 B (0.0 B/s)
INFO[0002] trying next host error="pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed" host=registry-1.docker.io
FATA[0002] failed to resolve reference "docker.io/library/abc@sha256:28eec4daefa7838fa3b8638eff27de54d7cb31552730d28cc6c030ecde962836": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
$ nerdctl run --rm docker.io/library/abc@sha256:28eec4daefa7838fa3b8638eff27de54d7cb31552730d28cc6c030ecde962836 /bin/sh
docker.io/library/abc@sha256:28eec4daefa7838fa3b8638eff27de54d7cb31552730d28cc6c030ecde962836: resolving |--------------------------------------|
elapsed: 1.5 s total: 0.0 B (0.0 B/s)
docker.io/library/abc@sha256:28eec4daefa7838fa3b8638eff27de54d7cb31552730d28cc6c030ecde962836: resolving |--------------------------------------|
elapsed: 2.0 s total: 0.0 B (0.0 B/s)
INFO[0002] trying next host error="pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed" host=registry-1.docker.io
FATA[0002] failed to resolve reference "docker.io/library/abc@sha256:28eec4daefa7838fa3b8638eff27de54d7cb31552730d28cc6c030ecde962836": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
The hash definitely exists in containerd and nerdctl knows it, e.g. nerdctl inspect abc:foo:
[
{
"Id": "sha256:78a332f4204f86ea549209154aa4c800cd26c0c4bf44dfead02bb3f7a2f8bcc9",
"RepoTags": [
"abc:foo"
],
"RepoDigests": [
"abc@sha256:28eec4daefa7838fa3b8638eff27de54d7cb31552730d28cc6c030ecde962836"
],
"Comment": "This is an apko single-layer image",
"Created": "0001-01-01T00:00:00Z",
"Author": "apko",
"Config": {
"AttachStdin": false,
"Env": [
"GOPATH=/home/build/.cache/go",
"HOME=/home/build"
]
},
"Architecture": "arm64",
"Os": "linux",
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:ca3f2d6ea1960d6bddd093dceef35436bbcb5e58b93f576633362e137c18ec01"
]
},
"Metadata": {
"LastTagTime": "0001-01-01T00:00:00Z"
}
}
]
and:
$ ctr --address /proc/3245/root/run/containerd/containerd.sock image ls
REF TYPE DIGEST SIZE PLATFORMS LABELS
docker.io/library/abc:foo application/vnd.docker.distribution.manifest.v2+json sha256:28eec4daefa7838fa3b8638eff27de54d7cb31552730d28cc6c030ecde962836 940.9 KiB linux/arm64 -
How do I run it based on hash?
I can replicate it slightly differently if I use an image that is in a registry:
$ nerdctl pull alpine:3.17
docker.io/library/alpine:3.17: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:af06af3514c44a964d3b905b498cf6493db8f1cde7c10e078213a89c87308ba0: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:d3156fec8bcbc7b491a4edc271a7734dcfa186fc73282d4e120eeaaf2ce95c43: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:261da4162673b93e5c0e7700a3718d40bcc086dbf24b1ec9b54bca0b82300626: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 5.7 s total: 3.1 Mi (558.9 KiB/s)
$ nerdctl image ls
REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
alpine 3.17 8914eb54f968 2 seconds ago linux/arm64/v8 7.8 MiB 3.1 MiB
$ # it finds it when referenced by tag
$ nerdctl run --rm alpine:3.17 echo hello
hello
$# if I ask for it by tag and hash, or just the hash, it goes back to registry even though the hash is there
$ nerdctl run --rm alpine:3.17@sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4 echo hello
docker.io/library/alpine@sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:af06af3514c44a964d3b905b498cf6493db8f1cde7c10e078213a89c87308ba0: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:d3156fec8bcbc7b491a4edc271a7734dcfa186fc73282d4e120eeaaf2ce95c43: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 1.6 s total: 0.0 B (0.0 B/s)
hello
$ # if I try it again, now it finds it
$ nerdctl run --rm alpine:3.17@sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4 echo hello
hello
$ # so by hash, it only works when the tag is _not_ there
$ nerdctl image ls
REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
alpine 3.17 8914eb54f968 About a minute ago linux/arm64/v8 7.8 MiB 3.1 MiB
alpine <none> 8914eb54f968 About a minute ago linux/arm64/v8 7.8 MiB 3.1 MiB
The above is not too bad, if wasteful, when dealing with published images. For images that are not published, i.e. loaded in, it is a failure.
Steps to reproduce the issue
- pull a known image
- try to run it by tag (or no tag) and hash
- it tries to pull again from the registry, even though it is local
Describe the results you received and expected
It should recognize a local digest before trying to pull.
What version of nerdctl are you using?
$ nerdctl version
Client:
Version: v1.0.0
OS/Arch: linux/arm64
Git commit: c00780a1f5b905b09812722459c54936c9e070e6
buildctl:
Version: v0.10.5
GitCommit: bc26045116045516ff2427201abd299043eaf8f7
Server:
containerd:
Version: v1.6.8
GitCommit: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
runc:
Version: 1.1.4
GitCommit: v1.1.4-0-g5fd4c4d1
Are you using a variant of nerdctl? (e.g., Rancher Desktop)
None
Host information
In lima default:
Client:
Namespace: default
Debug Mode: false
Server:
Server Version: v1.6.8
Storage Driver: overlayfs
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Log: fluentd journald json-file syslog
Storage: native overlayfs stargz fuse-overlayfs
Security Options:
apparmor
seccomp
Profile: default
cgroupns
rootless
Kernel Version: 5.19.0-26-generic
Operating System: Ubuntu 22.10
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 3.813GiB
Name: lima-melange-builder
ID: 48bb5881-01b0-4ea1-87a7-69822962ae46
WARNING: AppArmor profile "nerdctl-default" is not loaded.
Use 'sudo nerdctl apparmor load' if you prefer to use AppArmor with rootless mode.
This warning is negligible if you do not intend to use AppArmor.
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
➜ ~ snd images
REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
rabbitmq latest 90c62dd5e5fd 11 hours ago linux/amd64 232.4 MiB 96.9 MiB
➜ ~ snd run 90c62dd5e5fd echo hi
hi
➜ ~ snd run rabbitmq echo hi
hi
➜ ~ snd run rabbitmq:latest echo hi
hi
(snd = sudo nerdctl)
You are saying I should just give it the digest? No! It cannot be that simple! 🤦♂️
It does get things working (and thank you indeed for that), but should we support @sha256: format?
but should we support @sha256: format?
what do you mean by supporting sha256 format (e.g., REPO:TAG@sha256:xxxxxx... or sha256:xxxx....)?
It supports sha256 format, just don't need to explicitly specify the sha256: prefix.
➜ ~ snd image inspect 90c62dd5e5fd
[
{
"Id": "sha256:3ddcc140fe5c6dda21a48a3fcc99f120f686003106a767fc1279a5dfeac32b54",
"RepoTags": [
"rabbitmq:latest"
],
"RepoDigests": [
"rabbitmq@sha256:90c62dd5e5fdeda5ca57268c2ea7c7a648fed9fd73e7e545bbf36b936842ef4c"
],
....
# full sha256 id
➜ ~ snd run 90c62dd5e5fdeda5ca57268c2ea7c7a648fed9fd73e7e545bbf36b936842ef4c echo hi
hi
It's the same for other commands and for docker (when a sha256 id is needed, only the string (or a unique prefix) is needed)
➜ ~ d run 9b0d21eb242c echo hi
hi
➜ ~ d images | grep 9b0d21eb242c
golang 1.19-buster 9b0d21eb242c 4 weeks ago 935MB
(d = docker)
$ docker run -it --rm alpine:3.17
/ #
$ docker run -it --rm alpine:3.17@sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4
/ #
I took a look and it seems that it's because it didn't match during the image walk. The tag was not matched as part of the digest, so it only matches successfully when there is only the
I observed that Docker's behavior is as expected.
Please pay more attention to *ImageWalker.Walk. @Retrospection
I took a look and it seems that it's because it didn't match during the image walk. The tag was not matched as part of the digest, so it only matches successfully when there is only the tag.
I observed that Docker's behavior is as expected.
Please pay more attention to
*ImageWalker.Walk. @Retrospection
Thanks for your kind suggestion ~
Hey folks.
On current main, the following work:
nerdctl run -it --rm alpine:3.17
nerdctl run -it --rm alpine:3.17@sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4
nerdctl run -it --rm alpine@sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4
nerdctl run -it --rm sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4
nerdctl run -it --rm 8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4
nerdctl run -it --rm 8914eb54f9
Is there anything left to be done here or shall we close?
