Cannot delete manually loaded docker images(`kind load docker-image`) inside control-plane container

[vinayakbhat24]

What happened: We cannot delete images loaded to Kind cluster using the command kind load docker-image .... The issue is that it increases the RAM usage every time new images are loaded into the cluster since we cannot delete old images.

What you expected to happen: Inside the docker container control plane crictl rmi IMAGE_ID -> should delete a respective image, but not deleting. We also tried with a non-truncated IMAGE ID, but no success.

How to reproduce it (as minimally and precisely as possible):

• Load a docker image to kind cluster using the command kind load docker-image
• Try to delete the loaded docker image - crictl rmi IMAGE_ID

Anything else we need to know?: We tried to debug the issue using debug flags everywhere. The issue is found to be empty RepoTags for loaded images. This can be easily seen with the command crictl inspecti {ImageID}

Also, we tried using ctr command, but no success.

Environment:

• kind version: 0.17.0:
• Runtime info (docker info):

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.16.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan
Server:
 Server Version: 23.0.1

• OS :

Description:	Ubuntu 20.04.5 LTS
Release:	20.04
Codename:	focal

• Kubernetes version

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d", GitTreeState:"clean", BuildDate:"2022-12-08T19:58:30Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7

### Tasks

[BenTheElder]

You have to also delete the digest-only reference created by image importing.

As long as there is another reference to the image you're just deleting the tag.

/kind support /remove-kind bug

[BenTheElder]

aside: it should be disk usage, not RAM, unless somehow your docker volumes are memory backed

[vinayakbhat24]

@BenTheElder Tried out the following commands to delete digest reference, but no luck!

#crictl images --digests=true --no-trunc
docker.io/library/import-2023-09-26                       <none>               sha256:cf5db0e942a3bb33a6394331f7216ed7eceb94d0bfa117b628dc40d799150796   sha256:29504ba812afe36081ccbdf84f813675998aed17b66e36985035ded7f4296b82   128MB

#crictl rmi docker.io/library/import-2023-09-26@sha256:cf5db0e942a3bb33a6394331f7216ed7eceb94d0bfa117b628dc40d799150796
ERRO[0000] no such image docker.io/library/import-2023-09-26@sha256:cf5db0e942a3bb33a6394331f7216ed7eceb94d0bfa117b628dc40d799150796 
FATA[0000] unable to remove the image(s)

[BenTheElder]

I'm not sure if crictl supports this properly, you might have to use ctr -n=k8s.io

[victork0917]

@vinayakbhat24 Did you manage to resolve the issue? If yes, could you please provide the command you use? Thanks.

[vinayakbhat24]

@victork0917 I have not been able to resolve the issue yet.

[rriski]

Related: https://github.com/containerd/containerd/issues/7698