kind
kind copied to clipboard
Cannot delete manually loaded docker images(`kind load docker-image`) inside control-plane container
What happened:
We cannot delete images loaded to Kind cluster using the command kind load docker-image ...
. The issue is that it increases the RAM usage every time new images are loaded into the cluster since we cannot delete old images.
What you expected to happen:
Inside the docker container control plane
crictl rmi IMAGE_ID
-> should delete a respective image, but not deleting. We also tried with a non-truncated IMAGE ID, but no success.
How to reproduce it (as minimally and precisely as possible):
- Load a docker image to kind cluster using the command
kind load docker-image
- Try to delete the loaded docker image -
crictl rmi IMAGE_ID
Anything else we need to know?:
We tried to debug the issue using debug flags everywhere. The issue is found to be empty RepoTags
for loaded images. This can be easily seen with the command crictl inspecti {ImageID}
Also, we tried using ctr
command, but no success.
Environment:
- kind version: 0.17.0:
- Runtime info (docker info):
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.10.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.16.0
Path: /usr/libexec/docker/cli-plugins/docker-compose
scan: Docker Scan (Docker Inc.)
Version: v0.23.0
Path: /usr/libexec/docker/cli-plugins/docker-scan
Server:
Server Version: 23.0.1
- OS :
Description: Ubuntu 20.04.5 LTS
Release: 20.04
Codename: focal
- Kubernetes version
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d", GitTreeState:"clean", BuildDate:"2022-12-08T19:58:30Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
### Tasks
You have to also delete the digest-only reference created by image importing.
As long as there is another reference to the image you're just deleting the tag.
/kind support /remove-kind bug
aside: it should be disk usage, not RAM, unless somehow your docker volumes are memory backed
@BenTheElder Tried out the following commands to delete digest reference, but no luck!
#crictl images --digests=true --no-trunc
docker.io/library/import-2023-09-26 <none> sha256:cf5db0e942a3bb33a6394331f7216ed7eceb94d0bfa117b628dc40d799150796 sha256:29504ba812afe36081ccbdf84f813675998aed17b66e36985035ded7f4296b82 128MB
#crictl rmi docker.io/library/import-2023-09-26@sha256:cf5db0e942a3bb33a6394331f7216ed7eceb94d0bfa117b628dc40d799150796
ERRO[0000] no such image docker.io/library/import-2023-09-26@sha256:cf5db0e942a3bb33a6394331f7216ed7eceb94d0bfa117b628dc40d799150796
FATA[0000] unable to remove the image(s)
I'm not sure if crictl supports this properly, you might have to use ctr -n=k8s.io
@vinayakbhat24 Did you manage to resolve the issue? If yes, could you please provide the command you use? Thanks.
@victork0917 I have not been able to resolve the issue yet.
Related: https://github.com/containerd/containerd/issues/7698