kind
kind copied to clipboard
kubernetes-sigs
Podman restart support
What happened: Cluster does not work anymore after podman container is restarted (eg after host OS boot). The issue is fixed foe docker (https://github.com/kubernetes-sigs/kind/issues/148). Is there a plan to support restart for podman in the near future?
What you expected to happen: Cluster should run again after restarting podman container
How to reproduce it (as minimally and precisely as possible):
kind create cluster
podman stop kind-control-plane
podman start kind-control-plane
Anything else we need to know?:
Environment:
- kind version: (use
kind version): 0.11.0 - Kubernetes version: (use
kubectl version): kindest/node:v1.21.1 - Docker version: (use
docker info): podman version 3.1.2 - OS (e.g. from
/etc/os-release): Latest ArchLinux
podman doesn't handle restarts by design, it needs to use systemd files for managing containers on restarts.
https://github.com/containers/podman/blob/master/docs/source/markdown/podman-generate-systemd.1.md
Bear in mind that KIND wraps these containers technologies, if docker supports something out of the box and podman doesn't, is not likely that KIND is going to workaround it, it is by far out of scope of the project, however we work close and have a good relationship with both projects, collaborating and opening bugs if necessary.
Are you running podman as rootless? If podman support is "experimental", rootless is even "more experimental", so all the "advance" features may have bugs or simply not be supported at all ...
Podman also lacks a stable container network identifier which makes managing Kubernetes nodes across restarts problematic.
I don't think anyone is planning to work on this feature or has a plan for how it might be possible.
No, I am running podman as root. that is kind create cluster is run by root user.
Minikube supports podman and it can still do cluster start and stop using podman.
What makes kind different in this case?
After executing podman start kind-control-plane, can we just manually run a script on the running kind-control-plane container to start everything all over again?
Or would it be easier to add feature where all user data on kind-control-plane container is persisted in the host machine? This means if you delete and create cluster again, the new cluster will still have all the k8s objects from the previous cluster.
Minikube supports podman and it can still do cluster start and stop using podman.
Minikube supports podman and docker using a fork of the kind image yes.
What makes kind different in this case?
We don't work on that project. I don't work on podman support either. I can't tell you.
But I can tell you that podman lacks automatic restart for containers and lacks sufficient networking features to design robust restart. Node addresses will be random and restart support will be a roll of the dice. Stop and start is not what we mean when we say docker has restart support and has a different tracking issue that nobody has contributed to investigating this far. #1867
After executing podman start kind-control-plane, can we just manually run a script on the running kind-control-plane container to start everything all over again?
You're welcome to try but we have no such script.
Or would it be easier to add feature where all user data on kind-control-plane container is persisted in the host machine? This means if you delete and create cluster again, the new cluster will still have all the k8s objects from the previous cluster
Kubeadm doesn't support this AIUI. You can't just persist all data and then start a new cluster with it.
When stopping and starting or in docker restarting the data is persisted on any anonymous volume already. But not across clusters.
We are focused on making starting clusters cheap and quick so tests can be run from a clean state. We don't recommend keeping clusters permanently.
Thank you @BenTheElder for explaining things in the earlier post!
Do you think it will be (or maybe it is already) possible to declare required parameters in the config YAML file? Say if I wan to restart a multi-node cluster running on podman - in addition to the number of nodes I could declare static IP addresses per node... and so on. In other words if podman doesn't provide this functionality is there any way to allow users to make further configuration changes in order to compensate?
You have to use podman restart kind-control-plane.
podman start does not reattach the port forwarding.
Interestingly after an implicit stop, like rebooting, you have to start it and then restarting to make it work.
Hi @BenTheElder could you explain the "Node addresses will be random and restart support will be a roll of the dice"
I created an issue in Podman repository to be able to handle kind requirements but it's not clear what Kind is expecting from Podman side. https://github.com/containers/podman/issues/16797
Podman networking has changed a lot over the past few years but historically container IPs are random on startup and podman lacked an equivalent mechanism to docker's embedded DNS resolver with resolvable container names.
I don't think it's appropriate to file a bug against podman for kind unless there's a specific bug.
As you saw in #2998 the other reason we have't had a restart policy for podman is podman didn't support them meaningfully. That has changed a bit.
Hi @alvinsw
Even i'm also facing same error, after creating kind cluster using podman, when we are restring podman stop and start kind cluster not able to reach target endpoint. Almost we migrated docker podman around 1000 developers machine, this is something high priority. please let me if you get any workaround for this.
this is my support ticket - https://github.com/kubernetes-sigs/kind/issues/3473
Almost we migrated docker podman around 1000 developers machine, this is something high priority.
Unfortunately podman and docker are NOT direct substitutes and we don't have the bandwidth to spend on this ourselves currently.
In your issue, the containers are failing to start outright, at which point no kind code is even running, only podman/crun.
We'll continue to review suggested approaches to improving podman implementation in kind and the subsequent PRs.
Related: I think podman has had optional support for resolving container names for a while now, we could consider making this a pre-requesite and matching the docker behavior more closely.