podman-env support needs to update or deprecate

[elasticdotventures]

What Happened?

Minikube continues to embed podman 3.4.4, my system is podman 4.9.2 (ships standard with ubuntu noble 24.04) & my machine-default-image is podman 5.9.2

I can't reopen #15036 but here we are 3 years later. No progress. Same issue.

The discussion in issue/15036 speaks of a future to come re podman-env, and some other nonsense that failed to materialize. Minikube podman-env is effectively incompatible with all modern LTS versions of podman.

I want to understand why, I think minikube should at least output an explicit warning saying "hey, podman-env is irreprarably broked and will never be fixed, and we'll remove it someday" if that's true, if it's just docker forever, OR I want to at least (for now) link an issue .. maybe this one to the warning & workarounds so somebody else doesn't speak the ~6 hours I did going through upgrades, and ultimately being told "why don't you just use actual docker derp derp". What a waste of time, the docker mafia strikes again.

Yes, I can work around this with a local insecure registry or several other ways.

The specific error to reproduce:

minikube start --driver=podman --container-runtime=cri-o --kubernetes-version=v1.33.1

eval $(minikube -p minikube podman-env)

podman build -t middleware:latest -f ./middleware/Dockerfile ./middleware

Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: server API version is too old. Client "4.0.0" server "3.4.4": ssh://[email protected]:44503/run/podman/podman.sock

Attach the log file

 lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 24.04.2 LTS
Release:        24.04
Codename:       noble

Operating System

Ubuntu

Driver

QEMU

[elasticdotventures]

Related https://github.com/kubernetes/minikube/pull/20260 Related https://github.com/kubernetes/minikube/issues/15461

[elasticdotventures]

I see work has been done related the deprecating podman-env but appears it's review is stalled. I'm going to propose updating the podman-env docs to make it clear this will be sunset (if that is the final decision)

[elasticdotventures]

closing, this ticket, duplicate. https://github.com/kubernetes/minikube/pull/20884

[elasticdotventures]

Okay — @LJTian has closed #20884 (WIP) without progress. 👋🏻 @afbjorklund, any thoughts on the status of #15461?

I'm happy to investigate and submit a fresh solution. For those interested: I’ve successfully run Podman 4.9.2 with Minikube 1.36.0, but only by setting up an insecure registry. I couldn’t get it working with Podman + containerd — only with CRI-O.

FWIW, I had a discussion with the VSCode team about Podman support — the consensus was that there are numerous subtle but significant differences between Docker and Podman (too many to list). Trying to support both with a single module introduces complexity and risk. I believe maintaining a separate podman-env module is the better path — while recommending Docker as the default.

TL;DR: Separation of concerns. Docker users generally don’t want to read through Podman-specific errata, and Podman users deserve more detailed, focused documentation without breaking Docker flows or tests. A clean separation avoids cross-contamination and frustration.

Removing podman-env would be a step backward. Instead, it should be modernized:

• Keep it lean and focused, Reuse patterns from docker-env where feasible
• Support rootless and advanced Podman features
• Avoid muddying docker-env with Podman-specific edge cases
• Ensure test failures are isolated (Docker ≠ Podman and vice versa)

Podman aims for Docker compatibility, but the reality is messier — many projects and distros have different assumptions, timelines, and packaging quirks. Without someone from the Podman/CRI-O teams explicitly working to improve Docker compatibility in Minikube, I think the best approach is pragmatic: a dedicated, well-scoped podman-env.

Leaving this here for anyone who wants to weigh in or kick off a feasibility investigation. Happy to help move it forward.

[afbjorklund]

I think that minikube should update the podman version to v4.9, it is also obsolete - but less so than v3.4. The main requirement for it is that it works with cri-o (for loading and building images), and I suppose that still works to some extent. The main problem is that there is no client for it, and that it is not backwards-compatible. That is why it is better to use the docker client, since it negotiates versions and is compatible. Also avoids the need for a special podman-env, but that one can use docker-env for all runtimes. The preferred approach is minikube image, which can be tailored to minikube's needs.

I don't think that minikube needs to provide a fully functional Docker or Podman environment, if that is what you want there are other projects (such as Docker Machine or Podman Machine, or the newer Lima / Linux Machine) that does it better. The features that are needed are the ones missing from the CRI, and not delivered by the discontinued CBI. Most of the tutorials and whatnot just wants to load images from the disk cache, and occasionally build new images from a Dockerfile without having to push/pull to a registry or save/load to a disk. Then there are some helper functions like list and such.

[afbjorklund]

As for the stalled reviews etc, I would have to refer you to the current project reviewers and approvers as I have left.

EDIT: The main blocker back then, was that there was no Podman4 package available for Ubuntu 20.04, only Podman3. It is the same situation now with Podman5, but I suppose that can come later... Mostly missing quadlets anyway. (the quadlets, or "squarelets", are a more two-dimensional version of the kubelets that are managed by systemd)

[afbjorklund]

As far as I know, it is the same socket*. So even if there is only docker-env, we should be able to use it with podman too? * what I mean is that both Docker API (compat) and Podman API (libpod) is on the same sock, but with different paths

Like: CONTAINER_HOST=$DOCKER_HOST

Docker has no support for ssh keys though, so the workaround is to feed them to the ssh agent directly (no variable). But that workaround should also work with podman, you don't have to provide an identity if one is already available.

https://github.com/kubernetes/minikube/issues/15461#issuecomment-1336362019

[elasticdotventures]

Currently it's not possible to compile minikube on a machine which has a modern podman installed.

I've added a script to better detect podman during the docker-env in #21054 I'm not sure if that should be submitted separately or not, but seems like a good first step and progress for a Friday arvo! ;-)

[afbjorklund]

Currently it's not possible to compile minikube on a machine which has a modern podman installed.

You might want to open a separate issue about this, building minikube should not require/interfere with podman.

But I typically have both of them (docker engine and podman engine) installed, and they usually do get along...

Note that bugs with podman as a driver are (slightly) different from bugs with crio as a runtime (and podman helping)

[elasticdotventures]

🔍 Related Issue Found

Discovered a separate CRI-O bug while investigating podman-env: #21251

TL;DR: CRI-O incorrectly resolves → , causing errors.

This is separate from podman-env connectivity - it's a runtime image resolution bug. Both impact podman users but through different mechanisms.

Original report: https://github.com/elasticdotventures/minikube/issues/1

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[elasticdotventures]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten