[MinIO addon] console pod goes into ImagePullBackOff

[deusebio]

Summary

When enabling MinIO addon on MicroK8s 1.28-strict, the console pod never starts because of a ImagePullBackOff error due to the fact that it is unable to download the minio/console image from docker. I checked on DockerHub, and indeed, the minio/console page (which is indexed by google) does not seem to exist anymore.

Here is the kubectl describe command output

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  38s                default-scheduler  Successfully assigned minio-operator/console-78d567bfc8-vkbhp to pelican
  Normal   Pulling    19s (x2 over 37s)  kubelet            Pulling image "minio/console:v0.20.3"
  Warning  Failed     16s (x2 over 34s)  kubelet            Failed to pull image "minio/console:v0.20.3": failed to pull and unpack image "docker.io/minio/console:v0.20.3": failed to resolve reference "docker.io/minio/console:v0.20.3": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  Warning  Failed     16s (x2 over 34s)  kubelet            Error: ErrImagePull
  Normal   BackOff    5s (x2 over 34s)   kubelet            Back-off pulling image "minio/console:v0.20.3"
  Warning  Failed     5s (x2 over 34s)   kubelet            Error: ImagePullBackOff

The console pod is then on a ImagePullBackOff error loop

What Should Happen Instead?

The console pod should just start fine.

Reproduction Steps

1. sudo snap install microk8s --channel 1.28-strict/stable
2. microk8s enable minio

Introspection Report

inspection-report-20240423_091630.tar.gz

Can you suggest a fix?

I'm wondering whether the MinIO console image has been pulled out from DockerHub or if it got moved. If it was pull out and there is not alternative, we should probably either publish a Canonical supported one to have more control over it or disable the console to be run by default.

Are you interested in contributing with a fix?

If provided with guidance, I could give it a try. But I've never dug in the MicroK8s codebase, so if it's not a good-first issue (and given that this seems a bit critical/impactful to everyone using MinIO), maybe I'm not the best bet.

[eaudetcobello]

Hi @deusebio,

Thanks for reporting this issue. I'll be working on getting this fixed.

[nirmesh]

I am also facing same issue. is there any work around? When will it get fixed?

[sennid]

https://github.com/minio/console/issues/3318 - they no longer upload images for console since it's now part of minio/minio same binary

[abctaylor]

Confirmed I see this too

[eaudetcobello]

Hi everyone,

The minio team has deleted the console image from Dockerhub, which is problematic for the minio addon because it uses v4.5 of the minio-operator which relies on the console image. A temporary workaround is to find another source for the console image (eg. quay.io/minio/console) and modify the deployments so they use this image instead of just minio/console.

For a new cluster, you can replace line 89 of /var/snap/microk8s/common/addons/core/addons/minio/enable with this line: "${SNAP_COMMON}/plugins/kubectl-minio" init --console-image quay.io/minio/console:v0.23.0

Please note that these workarounds are not officially supported by Canonical, and the Microk8s team will be upgrading the minio addon version in the next patch release of Microk8s.

[oryza-anggara]

Is there any update?

How about set the image directly on the ReplicaSets? like.. kubectl set image replicaset.apps/console-78d567bfc8 -n=minio-operator console=quay.io/minio/console:v0.23.0

would it lead to further issue?

[cedricve]

Is there any update?

How about set the image directly on the ReplicaSets? like.. kubectl set image replicaset.apps/console-78d567bfc8 -n=minio-operator console=quay.io/minio/console:v0.23.0

would it lead to further issue?

This didnt work for me.

[cedricve]

Hi @deusebio,

Thanks for reporting this issue. I'll be working on getting this fixed.

Any idea on when this will be resolved? I believe this one is quiet important to support the addon community.

[cedricve]

The thing is that I'm currently unable to install the Minio operator as running it through the preferred installation method of Minio (with Kustomize) also gives an pthread error with microk8s (https://github.com/canonical/microk8s/issues/4550).

  root@microk8s:~# kubectl apply -k github.com/minio/operator\?ref=v6.0.1
  error: failed to run '/snap/microk8s/7040/usr/bin/git fetch --depth=1 https://github.com/minio/operator v6.0.1': /usr/lib/git-core/git-remote-https: symbol lookup error: /snap/core20/current/lib/x86_64-linux-gnu/libpthread.so.0: undefined symbol: __libc_pthread_init, version GLIBC_PRIVATE

I'm now forced to clone the repository manually.

git clone --depth 1 --branch v6.0.1 https://github.com/minio/operator.git && kubectl apply -k operator/