minikube icon indicating copy to clipboard operation
minikube copied to clipboard

Fix CRI-O image name resolution for localhost images

Open elasticdotventures opened this issue 8 months ago • 10 comments

Summary

Fixes CRI-O's inability to resolve locally built images when using podman driver, eliminating InvalidImageName errors.

Problem

When using minikube start --driver=podman --container-runtime=cri-o, locally built images fail to deploy with:

Failed to apply default image tag "/app4dog/middleware:latest": invalid reference format

Root Cause

CRI-O's registries.conf only searched docker.io, causing unqualified names to be incorrectly resolved.

Solution

Configure CRI-O during setup to:

  • Search localhost first, then docker.io for unqualified names
  • Mark localhost as insecure registry
  • Add proper unqualified-search-registries configuration

Impact

  • ✅ Fixes podman + CRI-O custom image deployment
  • ✅ Zero impact on Docker runtime (99.99% of users)
  • ✅ Surgical change - only modifies CRI-O registry config
  • ✅ Backwards compatible

Related Issues

  • Fixes #21251
  • Related to #21052 (separate podman-env connectivity issue)
  • Original report: https://github.com/elasticdotventures/minikube/issues/1

Testing

Enables successful deployment of locally built images referenced by unqualified names in K8s manifests.

elasticdotventures avatar Aug 06 '25 03:08 elasticdotventures

Hi @elasticdotventures. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot avatar Aug 06 '25 03:08 k8s-ci-robot

Can one of the admins verify this patch?

minikube-bot avatar Aug 06 '25 03:08 minikube-bot

Keywords which can automatically close issues and at(@) or hashtag(#) mentions are not allowed in commit messages.

The list of commits with invalid commit messages:

  • b063560 Fix CRI-O image name resolution for localhost images

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

k8s-ci-robot avatar Aug 06 '25 08:08 k8s-ci-robot

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: afbjorklund, elasticdotventures Once this PR has been reviewed and has the lgtm label, please assign spowelljr for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot avatar Aug 06 '25 18:08 k8s-ci-robot

/ok-to-test

medyagh avatar Aug 06 '25 18:08 medyagh

kvm2 driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21252 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 50.4s    │ 50.0s                  │
│ enable ingress │ 15.0s    │ 15.1s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 50.5s 47.5s 52.0s 50.2s 51.8s Times for minikube (PR 21252) start: 49.5s 48.9s 53.6s 48.9s 49.3s

Times for minikube (PR 21252) ingress: 14.9s 15.4s 14.9s 15.5s 14.9s Times for minikube ingress: 15.4s 15.4s 14.9s 14.4s 15.0s

docker driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21252 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 22.5s    │ 23.5s                  │
│ enable ingress │ 12.6s    │ 12.8s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 21.9s 21.1s 23.2s 21.5s 24.7s Times for minikube (PR 21252) start: 20.7s 22.4s 22.7s 26.0s 25.6s

Times for minikube ingress: 13.7s 10.7s 12.3s 13.2s 13.2s Times for minikube (PR 21252) ingress: 13.7s 10.3s 13.7s 12.8s 13.2s

docker driver with containerd runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21252 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 22.7s    │ 22.4s                  │
│ enable ingress │ 26.0s    │ 24.4s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 22.1s 22.2s 25.1s 22.1s 21.8s Times for minikube (PR 21252) start: 23.6s 20.6s 20.3s 22.3s 25.6s

Times for minikube ingress: 22.7s 38.7s 22.7s 23.2s 22.8s Times for minikube (PR 21252) ingress: 22.7s 23.3s 30.2s 23.2s 22.7s

minikube-pr-bot avatar Aug 06 '25 19:08 minikube-pr-bot

@elasticdotventures can you plz check the KVM_Linux_CRIO test failures? they usually fail much lower this is failing 127/259 https://storage.googleapis.com/minikube-builds/logs/21252/40841/KVM_Linux_crio.html

medyagh avatar Aug 06 '25 21:08 medyagh

I just want to be clear - this presumes and is blocked by #21054 This is downstream & should be compartmentalized. Unless somebody feels strongly - I'd prefer to keep them isolated.

elasticdotventures avatar Aug 07 '25 02:08 elasticdotventures

@medyagh I lost all momentum on this. Can you please look at https://github.com/kubernetes/minikube/pull/21054 If that gets merged then these tests will pass.

elasticdotventures avatar Sep 28 '25 07:09 elasticdotventures

@elasticdotventures: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-minikube-integration b0635602d18a39393ee3d6e1ff626d4fda018b53 link true /test pull-minikube-integration
integration-kvm-docker-linux-x86-64 b0635602d18a39393ee3d6e1ff626d4fda018b53 link true /test integration-kvm-docker-linux-x86-64
integration-docker-crio-linux-x86-64 b0635602d18a39393ee3d6e1ff626d4fda018b53 link true /test integration-docker-crio-linux-x86-64
integration-docker-docker-linux-x86-64 b0635602d18a39393ee3d6e1ff626d4fda018b53 link true /test integration-docker-docker-linux-x86-64
integration-docker-containerd-linux-x86-64 b0635602d18a39393ee3d6e1ff626d4fda018b53 link true /test integration-docker-containerd-linux-x86-64
integration-kvm-containerd-linux-x86-64 b0635602d18a39393ee3d6e1ff626d4fda018b53 link true /test integration-kvm-containerd-linux-x86-64
integration-none-docker-linux-x86-64 b0635602d18a39393ee3d6e1ff626d4fda018b53 link true /test integration-none-docker-linux-x86-64
integration-kvm-crio-linux-x86-64 b0635602d18a39393ee3d6e1ff626d4fda018b53 link true /test integration-kvm-crio-linux-x86-64
integration-docker-docker-linux-arm64 b0635602d18a39393ee3d6e1ff626d4fda018b53 link true /test integration-docker-docker-linux-arm64
integration-docker-docker-linux-x86 b0635602d18a39393ee3d6e1ff626d4fda018b53 link true /test integration-docker-docker-linux-x86
integration-kvm-docker-linux-x86 b0635602d18a39393ee3d6e1ff626d4fda018b53 link true /test integration-kvm-docker-linux-x86
integration-docker-crio-linux-x86 b0635602d18a39393ee3d6e1ff626d4fda018b53 link true /test integration-docker-crio-linux-x86
integration-docker-docker-linux-arm b0635602d18a39393ee3d6e1ff626d4fda018b53 link true /test integration-docker-docker-linux-arm
integration-kvm-containerd-linux-x86 b0635602d18a39393ee3d6e1ff626d4fda018b53 link true /test integration-kvm-containerd-linux-x86
integration-docker-containerd-linux-x86 b0635602d18a39393ee3d6e1ff626d4fda018b53 link true /test integration-docker-containerd-linux-x86
integration-kvm-crio-linux-x86 b0635602d18a39393ee3d6e1ff626d4fda018b53 link true /test integration-kvm-crio-linux-x86
integration-none-docker-linux-x86 b0635602d18a39393ee3d6e1ff626d4fda018b53 link true /test integration-none-docker-linux-x86

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

k8s-ci-robot avatar Dec 18 '25 21:12 k8s-ci-robot