kind
kind copied to clipboard
kubernetes-sigs
DOCKER_DEFAULT_PLATFORM breaks kind if it does not match the dockerd host platform
When running kind create cluster it fails with the following log message:
$ kind create cluster
Creating cluster "kind" ...
✓ Ensuring node image (kindest/node:v1.23.4) 🖼
✗ Preparing nodes 📦
ERROR: failed to create cluster: could not find a log line that matches "Reached target .*Multi-User System.*|detected cgroup v1"
Environment:
- kind version:
$ kind version
kind v0.12.0 go1.17.8 darwin/arm64
- Kubernetes version:
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:31:32Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"darwin/arm64"}
- Docker version: (use
docker info):
$ docker info
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc., v0.8.2)
compose: Docker Compose (Docker Inc., v2.4.1)
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
scan: Docker Scan (Docker Inc., v0.17.0)
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 10
Server Version: 20.10.14
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
runc version: v1.0.3-0-gf46b6ba
init version: de40ad0
Security Options:
seccomp
Profile: default
cgroupns
Kernel Version: 5.10.104-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 5
Total Memory: 7.765GiB
Name: docker-desktop
ID: LZ6O:NZMG:OHAC:45JC:ED7V:RX4B:O3X5:ENMG:CQ76:QAKS:S2UV:HWHJ
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5000
127.0.0.0/8
Live Restore Enabled: false
- OS:
macOS Monterey (v12.0) on Apple M1 Pro.
a similar one reported by fedora seems that was caused by the kernel version https://github.com/kubernetes-sigs/kind/issues/2689
I was experiencing the same issue Fixed for me by creating an architecture-specific image
Dockerfile
FROM --platform=arm64 kindest/node:v1.23.4
RUN arch
Built as
docker build -t tempkind .
Then
kind create cluster --image tempkind
hmm, we are publishing arm images https://hub.docker.com/layers/node/kindest/node/v1.23.4/images/sha256-0415a7bb1275c23c9315d97c5bfd7aecfdf8a8ea0562911f653711d947f7bec0?context=explore :thinking:
Originally reported issue doesn't appear to be related to availability of an image for arm64:
ERROR: failed to create cluster: could not find a log line that matches "Reached target .*Multi-User System.*|detected cgroup v1"
I have also created kind clusters on an M1 Mac successfully. I don't think this specific version, but normal creation works fine. Marco's issue appears to be something triggering a failure during the use of that image.
I confirm that following @lukevs instructions actually works. I created a new Docker build as he suggested, and the Kubernetes cluster started properly:
$ kind create cluster --image tempkind
Creating cluster "kind" ...
✓ Ensuring node image (tempkind) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:
kubectl cluster-info --context kind-kind
Not sure what to do next? 😅 Check out https://kind.sigs.k8s.io/docs/user/quick-start/
That doesn't make any sense, why isn't docker selecting the arm image from the multi-arch manifest? 🤔
What version of the docker desktop app is this? (Not dockerd / docker CLI, check the GUI / tray icon about info)
Is this behavior reproducible with other multi arch images?
This feels like a regression in the docker app, we know that M1 worked before and we didn't change much related to that on our end. We should double check the full default image reference in v0.12.0 though.
The only other explanation I have is v0.12 being bugged with the digest pointing to the amd64 image specifically. Will check for that. The image reference is truncated when printing the pull info unless more verbose logging is used IIRC.
For v0.12.0 the default image is https://github.com/kubernetes-sigs/kind/commit/ef21f8045c9379f52c433a4212d1fe8c61958a64#diff-643b1e9d9e446aa30da4407354de0098f24c947ac985213a06f73188c3e8e3fcR21
=> "kindest/node:v1.23.4@sha256:0e34f0d0fd448aa2f2819cfd74e99fe5793a6e4938b328f657c8e3f81ee0dfb9"
It's still crane digest kindest/node:v1.23.4 => sha256:0e34f0d0fd448aa2f2819cfd74e99fe5793a6e4938b328f657c8e3f81ee0df
And if you inspect that image you get:
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"manifests": [
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 743,
"digest": "sha256:1742ff7f0b79a8aaae347b9c2ffaf9738910e721d649301791c812c162092753",
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 742,
"digest": "sha256:0415a7bb1275c23c9315d97c5bfd7aecfdf8a8ea0562911f653711d947f7bec0",
"platform": {
"architecture": "arm64",
"os": "linux"
}
}
]
}
So docker should be auto-selecting an arm64 image on M1 ....
I don't currently have docker on M1, will need help narrowing down the source of this behavior https://github.com/kubernetes-sigs/kind/issues/2718#issuecomment-1101633715
Just as an FYI, Docker Desktop released a new version 4.7.1 (77678) but this problem still happens.
https://github.com/kubernetes-sigs/kind/issues/2718#issuecomment-1101960535 does docker run some-multi-arch-image also result in running the amd64 image instead of arm64?
@aojea and I cannot test this ourselves at the moment. But this workaround suggests a docker bug.
Just tested on a MacBook Air M1 - macOS 12.3.1
$ kind create cluster
Creating cluster "kind" ...
✓ Ensuring node image (kindest/node:v1.23.4) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:
kubectl cluster-info --context kind-kind
Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂
$ kind version
kind v0.12.0 go1.17.8 darwin/arm64
$ docker info
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc., v0.8.1)
compose: Docker Compose (Docker Inc., v2.3.3)
scan: Docker Scan (Docker Inc., v0.17.0)
Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 292
Server Version: 20.10.13
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
Default Runtime: runc
Init Binary: docker-init
containerd version: 2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc
runc version: v1.0.3-0-gf46b6ba
init version: de40ad0
Security Options:
seccomp
Profile: default
cgroupns
Kernel Version: 5.10.104-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 6
Total Memory: 3.826GiB
Name: docker-desktop
ID: OE44:7GZ6:JZKY:H6LU:LIFW:HL3U:2A37:4SG4:7QDJ:ZEWN:UWMY:NCL2
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5000
127.0.0.0/8
Live Restore Enabled: false
Looks like docker info returns the same kernel version as shown the original failure description.
$ docker images | grep kindest
kindest/node <none> f9d67d6e8342 6 weeks ago 1.21GB
$ docker inspect f9d67d6e8342
[
{
"Id": "sha256:f9d67d6e83427c29b4c9d413c2d79c5d5a5f3da57c39e59a0c892c183812624f",
"RepoTags": [],
"RepoDigests": [
"kindest/node@sha256:0e34f0d0fd448aa2f2819cfd74e99fe5793a6e4938b328f657c8e3f81ee0dfb9"
],
"Parent": "",
"Comment": "",
"Created": "2022-03-06T21:44:36.257726427Z",
"Container": "8ab6159dddf292aa03d0763d3ba61c4735fefcf6ed906bffaadc403bab93d4eb",
"ContainerConfig": {
"Hostname": "8ab6159dddf2",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"container=docker"
],
"Cmd": [
"infinity"
],
"Image": "docker.io/kindest/base:v20220305-b67a383f",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": [
"sleep"
],
"OnBuild": null,
"Labels": {},
"StopSignal": "SIGRTMIN+3"
},
"DockerVersion": "20.10.11",
"Author": "",
"Config": {
"Hostname": "8ab6159dddf2",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"container=docker"
],
"Cmd": null,
"Image": "docker.io/kindest/base:v20220305-b67a383f",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": [
"/usr/local/bin/entrypoint",
"/sbin/init"
],
"OnBuild": null,
"Labels": {},
"StopSignal": "SIGRTMIN+3"
},
"Architecture": "arm64",
"Os": "linux",
"Size": 1214940253,
"VirtualSize": 1214940253,
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/a736ff8cd71c7d07cdced41e53aad3d2b44dc98265b24e51baac8cff0f6eae2a/diff",
"MergedDir": "/var/lib/docker/overlay2/4fe148cc1e4c4b1539bbdc023b3033667b3f501d9ec3caea6185f81585d8ca45/merged",
"UpperDir": "/var/lib/docker/overlay2/4fe148cc1e4c4b1539bbdc023b3033667b3f501d9ec3caea6185f81585d8ca45/diff",
"WorkDir": "/var/lib/docker/overlay2/4fe148cc1e4c4b1539bbdc023b3033667b3f501d9ec3caea6185f81585d8ca45/work"
},
"Name": "overlay2"
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:d384c88eae37978ca8a95aff7b13c6987b0bdedec50fca7890c484b0918ccb73",
"sha256:db19631aac508ec95983ced710b2ca045ec593603644e6b31f833bfaae47092b"
]
},
"Metadata": {
"LastTagTime": "0001-01-01T00:00:00Z"
}
}
]
I had the same problem, but I then remembered I had added DOCKER_DEFAULT_PLATFORM=linux/amd64 to my ~/.zshrc. After unsetting the env-var it would still throw the same error. Only after removing the images completely (docker image rm ...) Docker would pull the correct image and the problem was gone.
@svmaris thank you! This was exactly my problem. I removed the environment variable, deleted the Docker images, and now it works like a charm.
I had the same problem, but I then remembered I had added DOCKER_DEFAULT_PLATFORM=linux/amd64 to my ~/.zshrc. After unsetting the env-var it would still throw the same error. Only after removing the images completely (docker image rm ...) Docker would pull the correct image and the problem was gone.
Thank you! That explains everything, including why the workaround of building an arm64 only image would work https://github.com/kubernetes-sigs/kind/issues/2718#issuecomment-1101496008
We should probably consider some follow up here ...
Is this env only read by the docker CLI? It seems so given your docker info shows aarch64 on the server. If that's the case, kind should maybe unset this when invoking docker
If not .. we should consider forcing --platform to be the server platform all over the place ... which would be pretty messy.
At minimum we should probably add a known issues entry if we don't plan to work around this.
Is this env only read by the docker CLI? It seems so given your docker info shows aarch64 on the server. If that's the case, kind should maybe unset this when invoking docker
Answering myself: yes https://docs.docker.com/engine/reference/commandline/cli/
DOCKER_DEFAULT_PLATFORM | Default platform for commands that take the --platform flag.
We should consider if it makes sense for kind to block this env from reaching docker calls invoked by kind, or else to check and warn if it doesn't seem to match the server info when doing certain operations (kind create cluster is probably the only one in particular).
I think it might make sense to have kind ignore broadly, like unsetting it in the kind binary or filtering it in execs to docker. Using exec(docker ...) is a current implementation detail, if we intended to support this https://github.com/kubernetes-sigs/kind/issues/2735 we'd have some kind specific option probably (KIND_DEFAULT_PLATFORM perhaps), and we don't, as it needs to match the host platform to work.
cc @aojea btw https://github.com/kubernetes-sigs/kind/issues/2738#issuecomment-1116712326 has some more details on why we can't leverage the docker qemu userspace emulation approach to multi-arch (which kind actually uses for compiling node images! it's just not sufficient for running kubernetes ...), and therefore should probably block this broken override.
Had the same issue above; but @lukevs solution didn't work for me, I was getting some other error:
Command Output: Unable to find image 'tmpkind' locally
docker: Error response from daemon: pull access denied for tmpkind, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.
Fixed it by setting DOCKER_BUILDKIT=1 as env variable. You also have to build the image with this env variable.
DOCKER_BUILDKIT=1 kind create cluster --image tmpkind
Creating cluster "kind" ...
✓ Ensuring node image (scgckind) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:
kubectl cluster-info --context kind-kind
Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂
Is this env only read by the docker CLI? It seems so given your docker info shows aarch64 on the server. If that's the case, kind should maybe unset this when invoking docker
Answering myself: yes https://docs.docker.com/engine/reference/commandline/cli/
DOCKER_DEFAULT_PLATFORM | Default platform for commands that take the --platform flag.
We should consider if it makes sense for kind to block this env from reaching
dockercalls invoked by kind, or else to check and warn if it doesn't seem to match the server info when doing certain operations (kind create clusteris probably the only one in particular).I think it might make sense to have kind ignore broadly, like unsetting it in the kind binary or filtering it in execs to docker. Using
exec(docker ...)is a current implementation detail, if we intended to support this #2735 we'd have some kind specific option probably (KIND_DEFAULT_PLATFORMperhaps), and we don't, as it needs to match the host platform to work.
Don't know if there is any use case for setting DOCKER_DEFAULT_PLATFORM but preventing it to reach Docker would introduce some non-linear behavior in Kind. I think the user should be allowed to make mistakes like this.
I'm therefore in favor of just putting a warning in case of mismatching values.
@ungps for now you should probably unset DOCKER_DEFAULT_PLATFORM when calling kind e.g. like DOCKER_DEFAULT_PLATFORM= kind create cluster, then it will work with the multi-arch images isntead.
Don't know if there is any use case for setting DOCKER_DEFAULT_PLATFORM but preventing it to reach Docker would introduce some non-linear behavior in Kind. I think the user should be allowed to make mistakes like this.
This is a fair point, though kind being implemented by way of exec(docker, ...) is a current leaky implementation detail FWIW, the client library wouldn't respect all these env AFAIK.
We could similarly instead accomplish this by setting --platform=$dockerd_server_platform when we call docker, but it's equivalent and simpler to just not pass the env through.
If we just do a warning, the user will have to actively alter their host configuration or at least override it when invoking kind, which is more friction.
A compromise might be detecting that the env is set and warning that kind ignores it and include a reference to this issue or some documentation.
It seems to be an unintuitive failure mode that took a month to pin down for the first user with more trickling in, so we should probably move towards one of these soon. cc @aojea /priority important-soon
This is a fair point, though kind being implemented by way of
exec(docker, ...)is a current leaky implementation detail FWIW, the client library wouldn't respect all these env AFAIK.
I overlooked this, then maybe it's better to completely sanitize the environment and let only the explicitly configured variables.
what is the reasons of setting DOCKER_DEFAULT_PLATFORM ?
is this because of this https://stackoverflow.com/questions/65612411/forcing-docker-to-use-linux-amd64-platform-by-default-on-macos?
that seems a temporary problem , if affirmative I think warning and link is the best way to proceed?
what is the reasons of setting DOCKER_DEFAULT_PLATFORM ? is this because of this https://stackoverflow.com/questions/65612411/forcing-docker-to-use-linux-amd64-platform-by-default-on-macos?
I think this is the primary reason
that seems a temporary problem , if affirmative I think warning and link is the best way to proceed?
I don't see any reason it would be temporary, as long as users are still developing amd64-only projects from macOS, where the host platform will be heading to arm64 + emulation for the forseeable future.
I overlooked this, then maybe it's better to completely sanitize the environment and let only the explicitly configured variables.
Yeah, I think as a general rule we probably should sanitize env going to exec, this particular issue just seems to have highlighted it.
If, in the future, this setting were actually supportable, we could consider intentionally passing through the env.
Just chiming in here... perhaps I should open a new issue, but it would be really powerful for us to be able to launch a KIND cluster on our M1 Mac's with both AMD64 and ARM64 Nodes. We are struggling to reproduce certain environments locally (like istio, which doesn't yet have a mullti-arch proxyv2 image), and being able to run a full multi-architecture local kind cluster would be an incredible value. Is that something that is remotely feasible in the future?
Just chiming in here... perhaps I should open a new issue, but it would be really powerful for us to be able to launch a KIND cluster on our M1 Mac's with both AMD64 and ARM64 Nodes. We are struggling to reproduce certain environments locally (like istio, which doesn't yet have a mullti-arch proxyv2 image), and being able to run a full multi-architecture local kind cluster would be an incredible value. Is that something that is remotely feasible in the future?
Unfortunately not. See https://github.com/kubernetes-sigs/kind/issues/2738#issuecomment-1116712326
The way docker run does multi-arch cannot run complex workloads like Kubernetes, only simple compute with limited syscalls. To run multi arch Kubernetes you need a host / kernel (could be a VM) actually running on each arch.
I would posit that you should be able to test with two separate clusters one of each arch though.
Docker desktop already employs a VM to get linux on Mac. In theory a second VM on amd64 would let you run an amd64 cluster. I haven't seen anyone make this work yet though. I think most tools don't support x86 VMs on M1 because it would be slow without x86 hardware virtualization support.
We do not have any intent of supporting multi-machine (you should use an actual non-local cluster installer for this) so it wouldn't be a single cluster using two VMs to host the respective architectures, and the qemu bimft_misc hack cannot run Kubernetes.
The simplest approach is to export DOCKER_DEFAULT_PLATFORM=linux/arm64 for Mac M1 or just empty that field as DOCKER_DEFAULT_PLATFORM=, and then run kind create cluster --config create-kind-cluster.yaml This fixed my issue.
I used the create-kind.yaml file as
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
image: kindest/node-arm64:v1.26.3@sha256:3dc28d458c92e3252c78752ade134d7231ce8e3565f2de4e437a77b171d046ea
- role: worker
image: kindest/node-arm64:v1.26.3@sha256:3dc28d458c92e3252c78752ade134d7231ce8e3565f2de4e437a77b171d046ea
- role: worker
image: kindest/node-arm64:v1.26.3@sha256:3dc28d458c92e3252c78752ade134d7231ce8e3565f2de4e437a77b171d046ea
- role: worker
image: kindest/node-arm64:v1.26.3@sha256:3dc28d458c92e3252c78752ade134d7231ce8e3565f2de4e437a77b171d046ea
