kind
kind copied to clipboard
kubernetes-sigs
Kind Load Image Digest Doesn't Work
What happened: I loaded a docker image by digest and kind seemed to load the image correctly. But, unfortunately the image wasn't loaded.
Here is a redacted snippet of the load command and response:
]$ kind load docker-image redacted/org/grafana@sha256:72e1e8a30e96b9e237ab601be1a16b84933abfb6311b3420f07867f10bba7827
Image: "redacted/org/grafana@sha256:72e1e8a30e96b9e237ab601be1a16b84933abfb6311b3420f07867f10bba7827" with ID "sha256:5030e7c37d6d2958c32e056dc5b1941048140047c04a7fb6a9a3aa2ab1bc73ef" not yet present on node "kind-control-plane", loading...
]$ kind load docker-image redacted/org/grafana@sha256:72e1e8a30e96b9e237ab601be1a16b84933abfb6311b3420f07867f10bba7827
Image: "redacted/org/grafana@sha256:72e1e8a30e96b9e237ab601be1a16b84933abfb6311b3420f07867f10bba7827" with ID "sha256:5030e7c37d6d2958c32e056dc5b1941048140047c04a7fb6a9a3aa2ab1bc73ef" not yet present on node "kind-control-plane", loading...
]$ kind load docker-image redacted/org/grafana@sha256:72e1e8a30e96b9e237ab601be1a16b84933abfb6311b3420f07867f10bba7827
Image: "redacted/org/grafana@sha256:72e1e8a30e96b9e237ab601be1a16b84933abfb6311b3420f07867f10bba7827" with ID "sha256:5030e7c37d6d2958c32e056dc5b1941048140047c04a7fb6a9a3aa2ab1bc73ef" not yet present on node "kind-control-plane", loading...
What you expected to happen: I expected the image digest to be loaded into kind and to be accessible via it's digest.
How to reproduce it (as minimally and precisely as possible): Try to load an image digest into kind and try to use that image digest, or try to look for that image digest via crictl.
Anything else we need to know?:
Environment:
- kind version: (use
kind version): kind v0.11.1 go1.16.4 linux/amd64 - Kubernetes version: (use
kubectl version):
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.9", GitCommit:"94f372e501c973a7fa9eb40ec9ebd2fe7ca69848", GitTreeState:"clean", BuildDate:"2020-09-16T13:56:40Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.10", GitCommit:"f3add640dbcd4f3c33a7749f38baaac0b3fe810d", GitTreeState:"clean", BuildDate:"2020-05-20T13:51:56Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
- Docker version: (use
docker info):
Client:
Version: 20.10.7
API version: 1.41
Go version: go1.16.4
Git commit: f0df35096d
Built: Fri Jun 4 08:14:39 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server:
Engine:
Version: 20.10.7
API version: 1.41 (minimum version 1.12)
Go version: go1.16.4
Git commit: b0f5bc36fe
Built: Fri Jun 4 08:14:24 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.5.2
GitCommit: 36cc874494a56a253cd181a1a685b44b58a2e34a.m
runc:
Version: 1.0.0-rc95
GitCommit: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
docker-init:
Version: 0.19.0
GitCommit: de40ad0
- OS (e.g. from
/etc/os-release):
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
LOGO=archlinux
I expected the image digest to be loaded into kind and to be accessible via it's digest.
You can load by a digest but you cannot reference it by the digest you pulled. You can list the images in the cluster and see that there is an image loaded, but not with the digest you desired.
This is because image digests hash the entire image including the metadata that contains the repo tags in the image, and they will not match.
Another common issue is referencing an image by the digest of a manifest list, but manifest lists can only be stored in a registry, not in docker/containerd.
Basically you're seeing what happens when you docker save + ctr images import, kind does nothing itself to break anything.
You can push to a local registry instead (see the docs for how to set one up), but the digests will again change because the hostname will change.
I think it may be possible to make this work with some manipulation of the image as loaded to have identical metadata but this has not ever worked in kind and is not a straightforward bug.
Lol, this is really strange design that digest of Docker images are registry-bound.
possibly related 🤦 https://github.com/kubernetes-sigs/kind/issues/2734
have not had time to dig back into this.
You can push to a local registry instead (see the docs for how to set one up), but the digests will again change because the hostname will change.
That's not accurate, I was thinking of the docker save format, not the registry API.
However, digests are frequently pinned to multi-arch images, but kind load docker-image must be for a particular architecture and not the manifest, because images in local storage / saved by docker save are for a specific arch (one of the manifests, behind the manifest list indirection).
https://github.com/kubernetes-sigs/kind/pull/2756 🤦♂️
See also: https://github.com/kubernetes-sigs/kind/pull/2955 (not released yet, but fixing a regression in image loading)
Hey @BenTheElder 👋
Just to chime in I think I'm seeing the same issue with kind v0.17.0 (I also tried building off main at 4a0de915). If I load the image via load docker-image:
kind load docker-image gcr.io/knative-releases/knative.dev/operator/cmd/operator@sha256:15fdcd6927ec3415fe2cca17f85c395b67522f716a255bffb23de855c0ccc6b6 --name kind
Image: "" with ID "sha256:1e7e67348c2fce975e89c1670537a68ff1e1131467d94f42d8d8fb8f9a15cb4b" not yet present on node "kind-control-plane", loading...
or from a tarball via load image-archive (note: kind show nothing on stdout, but does exit 0)
kind load image-archive gcr.io__knative-releases__knative.dev__operator__cmd__operator.tar --name kind
Neither work, as the image isn't present:
docker exec -it kind-control-plane crictl images
IMAGE TAG IMAGE ID SIZE
docker.io/kindest/kindnetd v20221004-44d545d1 7ba9b35cf55e6 23.7MB
docker.io/kindest/local-path-helper v20220607-9a4d8d2a 304f67e47fc80 2.75MB
ghcr.io/fluxcd/helm-controller v0.27.0 5fd6014d29477 71.4MB
registry.k8s.io/coredns/coredns v1.9.3 b19406328e70d 13.4MB
registry.k8s.io/kube-controller-manager v1.25.3 05b17bba8656e 62.3MB
docker.io/library/import-2023-01-19 <none> 1e7e67348c2fc 73.7MB
docker.io/kindest/local-path-provisioner v0.0.22-kind.0 7902f9a1c54fa 15.6MB
registry.k8s.io/etcd 3.5.4-0 8e041a3b0ba8b 81.1MB
registry.k8s.io/kube-apiserver v1.25.3 feafd6a91eb52 74.2MB
registry.k8s.io/kube-proxy v1.25.3 aa31a9b19ccdf 59.6MB
registry.k8s.io/kube-scheduler v1.25.3 253d0aeea8c69 50.6MB
registry.k8s.io/pause 3.7 e5a475a038057 268kB
I know this issue isn't closed so it makes sense it still doesn't work, just wanted to +1 the issue. I'd also be happy to contribute a fix if you can point me in the right direction of what changes need to be made 😄
The problem is that gcr.io/knative-releases/knative.dev/operator/cmd/operator@sha256:15fdcd6927ec3415fe2cca17f85c395b67522f716a255bffb23de855c0ccc6b6 is a multi-arch manifest-list, which is resolved by docker pull / docker save to the host-platform specific manifest / image.
$ crane manifest gcr.io/knative-releases/knative.dev/operator/cmd/operator@sha256:15fdcd6927ec3415fe2cca17f85c395b67522f716a255bffb23de855c0ccc6b6
{"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.list.v2+json","manifests":[{"mediaType":"application/vnd.docker.distribution.manifest.v2+json","size":957,"digest":"sha256:a71bad45fe686851238a86408e9cdfcb4dda14b6974e131911bc7fda27e3cc98","platform":{"architecture":"amd64","os":"linux"}},{"mediaType":"application/vnd.docker.distribution.manifest.v2+json","size":957,"digest":"sha256:403cb4e0b91443365813de9df862073302e47122f5441e63188748256fbd3426","platform":{"architecture":"arm","os":"linux"}},{"mediaType":"application/vnd.docker.distribution.manifest.v2+json","size":957,"digest":"sha256:e87490b5b47027452692589cf995c1432ee773293f91b2b0e87c3e40cf22a7f9","platform":{"architecture":"arm64","os":"linux"}},{"mediaType":"application/vnd.docker.distribution.manifest.v2+json","size":957,"digest":"sha256:c6ec9ae95baad76e76a02542bdc7a9d216cbec53909c34d724a10ba55fe83f4a","platform":{"architecture":"ppc64le","os":"linux"}},{"mediaType":"application/vnd.docker.distribution.manifest.v2+json","size":957,"digest":"sha256:22b8326043ba1173ee1a72c5ebf936a5e3f839225d1937d1dd5b842049d1672c","platform":{"architecture":"s390x","os":"linux"}}]}
kind load docker-image cannot load a multi-arch image manifest, docker pull / docker save do not support this. Docker stores one platform per ref locally.
I haven't had time to dig into if there's a reasonable approach to solve this in an expected and compatible way.
I think to do this sort of thing properly we'd need a different command with some custom logic to:
- pull to some cache directory instead of docker's existing image store
- record the original multi-arch image digest
- attach the multi-arch digest to the imported image
Which could get tricky fast ...
The OCI layout format using someting like crane pull has this information, but it's natively a directory structure not a single file like the docker archive tars, so plumbing it to the running node container is going to be clunky, and again we lose the expectation of de-duping storage on the host at least into your container runtime's image store (along with the existing controls for listing, removing, and pruning images etc)
We might be able to do something hackier and whenever we see we're loading a digest we instruct containerd to attach the digest ref to the imported image. Maybe by exec-ing ctr images tag.
Related to #1140? I am having trouble preloading images from the ingress-nginx chart since the chart specifies both a tag and a hash. If I preload the images with just the tag, then it looks like the layers are there on the node, but Kind still has to pull the manifest (taking ~1s but of course requiring the network); if I preload the images with both the tag and hash (verbatim as in the chart) then the pods fail to start ever (it seems like CRI gets confused). Can try to come up with a self-contained test case if this is unknown.
