k3d
k3d copied to clipboard
k3d-io
[BUG] CoreDNS NodeHosts lost after adding a new node
What did you do
- How was the cluster created?
k3d cluster create
INFO[0016] Injecting records for hostAliases (incl. host.k3d.internal) and for 2 network members into CoreDNS configmap...
What did you do afterwards?
- Check configmap for coredns
The host host.k3d.internal is present on the cluster creation
NodeHosts: |
172.40.0.1 host.k3d.internal
172.40.0.2 k3d-local-serverlb
172.40.0.3 k3d-local-server-0
kind: ConfigMap
- Add a new node with
k3d node create newserver --cluster local --role server
INFO[0000] Adding 1 node(s) to the runtime local cluster 'local'...
INFO[0000] Using the k3d-tools node to gather environment information
INFO[0000] Starting new tools node...
INFO[0000] Starting Node 'k3d-local-tools'
INFO[0000] HostIP: using network gateway 172.40.0.1 address
INFO[0001] Starting Node 'k3d-newserver-0'
INFO[0015] Updating loadbalancer config to include new server node(s)
INFO[0015] Successfully configured loadbalancer k3d-local-serverlb!
INFO[0016] Successfully created 1 node(s)!
- Check the coreDNS configmap
NodeHosts: |
172.40.0.4 k3d-newserver-0
kind: ConfigMap
What did you expect to happen
The new host is added but others NodeHosts are not lost
Screenshots or terminal output
Which OS & Architecture
- Linux, Fedora 35
Which version of k3d
- output of
k3d version v5.3.0 k3s version v1.22.6-k3s1 (default)
Which version of docker
- output of
Version: 20.10.12
API version: 1.41
Go version: go1.16.12
Git commit: e91ed57
Built: Mon Dec 13 11:46:03 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.12
API version: 1.41 (minimum version 1.12)
Go version: go1.16.12
Git commit: 459d0df
Built: Mon Dec 13 11:43:48 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.12
GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc:
Version: 1.0.2
GitCommit: v1.0.2-0-g52b36a2
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Hi @lerminou , thanks for opening this issue! This is indeed pretty weird and I haven't seen this before. This would in fact mean, that K3s even deletes its own entries :thinking: I put this on the list to be checked before the next release :+1:
Hi @iwilltry42
Thanks for your time.
One more precision, My cluster was created with a 1.21 k3s image..
My second node was created with a 1.22 version ( I submitted another bug about that)
If a check k3s changelog. There are some changes about the coredns config.
I will try to reproduce it in a full 1.22 cluster
Hey @lerminou in context of #1032 I just did several node adds and my CoreDNS configmap now looks like this:
NodeHosts: |
172.21.0.2 k3d-test-server-0
172.21.0.5 k3d-testnode2-0
172.21.0.1 host.k3d.internal
172.21.0.6 k3d-testnode3-0
172.21.0.3 k3d-test-agent-0
172.21.0.4 k3d-test-serverlb
So I assume that the issue is limited to creating a new server node.
Which after taking another look at your original post raises a question:
You just ran k3d cluster create without any additional flags (Meaning: 1 server and no --cluster-init flag), right? :thinking:
That would mean, that the new server node shouldn't even be able to join the existing cluster (or induce some split-brain mode), since the original server-0 does not use etcd and thus cannot add another server node.
Hi @iwilltry42 I didn't fill the yaml config file but the cluster was created with the correct flag to use etcd.
I agree it only appears when adding a new node to an existing cluster
I didn't fill the yaml config file but the cluster was created with the correct flag to use etcd.
Huh? The --cluster-init flag should only be set when creating more than 1 server at cluster creation or when you set it manually :thinking:
Can you share the output of docker inspect <server-0-container>? (inspecting the initial server container created by k3d cluster create)
I agree it only appears when adding a new node to an existing cluster
Also when adding an agent node?
Here is the relevant informations from my config.yaml used when I create the cluster
apiVersion: k3d.io/v1alpha4
kind: Simple
metadata:
name: local
servers: 1
agents: 0
image: rancher/k3s:v1.21.10-k3s1
options:
k3s: # options passed on to K3s itself
extraArgs: # additional arguments passed to the `k3s server|agent` command; same as `--k3s-arg`
- arg: --tls-san=back-local.kube.com
nodeFilters:
- all
- arg: --cluster-init
nodeFilters:
- server:0
And the output of the container
[
{
"Id": "f03c57557f5e0e74efc7d9ac6c6bf672af1a9b35dbbb0e009e288cc8d1e2a153",
"Created": "2022-03-26T14:35:51.512776582Z",
"Path": "/bin/k3s",
"Args": [
"server",
"--tls-san=back-local.kube.com",
"--cluster-init",
"--cluster-domain=kube.com",
"--disable=traefik",
"--resolv-conf=/var/resolv.conf",
"--tls-san",
"back-local.kube.com"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 45282,
"ExitCode": 0,
"Error": "",
"StartedAt": "2022-03-26T14:35:52.169325626Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:ce2cc3439feeb2a54dc01daf414754a64897f815e4c9a5c59c02b6d9d5b8d38c",
"ResolvConfPath": "/home/docker-images/containers/f03c57557f5e0e74efc7d9ac6c6bf672af1a9b35dbbb0e009e288cc8d1e2a153/resolv.conf",
"HostnamePath": "/home/docker-images/containers/f03c57557f5e0e74efc7d9ac6c6bf672af1a9b35dbbb0e009e288cc8d1e2a153/hostname",
"HostsPath": "/home/docker-images/containers/f03c57557f5e0e74efc7d9ac6c6bf672af1a9b35dbbb0e009e288cc8d1e2a153/hosts",
"LogPath": "/home/docker-images/containers/f03c57557f5e0e74efc7d9ac6c6bf672af1a9b35dbbb0e009e288cc8d1e2a153/f03c57557f5e0e74efc7d9ac6c6bf672af1a9b35dbbb0e009e288cc8d1e2a153-json.log",
"Name": "/k3d-local-server-0",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/home/projects/k3d/manifests:/var/lib/rancher/k3s/server/manifests/",
"/home/projects/k3d/CA-kube.crt:/etc/ssl/certs/ca.pem",
"/home/projects/k3d/storage:/var/lib/rancher/k3s/storage",
"/home/projects/k3d/resolv.conf:/var/resolv.conf",
"/home:/projects",
"k3d-local-images:/k3d/images"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "default",
"PortBindings": null,
"RestartPolicy": {
"Name": "unless-stopped",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "host",
"Dns": null,
"DnsOptions": null,
"DnsSearch": null,
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"label=disable"
],
"Tmpfs": {
"/run": "",
"/var/run": ""
},
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": null,
"DeviceCgroupRules": null,
"DeviceRequests": null,
"KernelMemory": 0,
"KernelMemoryTCP": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": false,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": null,
"ReadonlyPaths": null,
"Init": true
},
"GraphDriver": {
"Data": {
"LowerDir": "/home/docker-images/overlay2/b99aeab69d19a2a6c3837ac92b6fb0b56724d203459a9f4de47a75b636a7e2d2-init/diff:/home/nicolasdubut/docker-images/overlay2/2a7d00212eac6ae53ab10ddee65285eb88f69c80f1b4fbc539787da3719a7001/diff:/home/nicolasdubut/docker-images/overlay2/5ac2e683a098eb7db25da3239d0b11620528e0cc7f208a9cb0c63f96392d1c80/diff:/home/nicolasdubut/docker-images/overlay2/a781511d2cea6e430e927f862a3b443862af1b4ef74b12029b03d499760cf968/diff",
"MergedDir": "/home/docker-images/overlay2/b99aeab69d19a2a6c3837ac92b6fb0b56724d203459a9f4de47a75b636a7e2d2/merged",
"UpperDir": "/home/docker-images/overlay2/b99aeab69d19a2a6c3837ac92b6fb0b56724d203459a9f4de47a75b636a7e2d2/diff",
"WorkDir": "/home/docker-images/overlay2/b99aeab69d19a2a6c3837ac92b6fb0b56724d203459a9f4de47a75b636a7e2d2/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "bind",
"Source": "/home/projects/k3d/storage",
"Destination": "/var/lib/rancher/k3s/storage",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "volume",
"Name": "3c31da469f55de07251b661d5f1c42a88ee6a479cb08cc35397037894eabe302",
"Source": "/home/docker-images/volumes/3c31da469f55de07251b661d5f1c42a88ee6a479cb08cc35397037894eabe302/_data",
"Destination": "/var/lib/cni",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
},
{
"Type": "volume",
"Name": "678124af0cdcc59b2714c59b91e94a8ac9a3bdd1042ca25dda118742b3aa2a39",
"Source": "/home/docker-images/volumes/678124af0cdcc59b2714c59b91e94a8ac9a3bdd1042ca25dda118742b3aa2a39/_data",
"Destination": "/var/lib/rancher/k3s",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
},
{
"Type": "bind",
"Source": "/home/projects/k3d/manifests",
"Destination": "/var/lib/rancher/k3s/server/manifests",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/home/projects/k3d/resolv.conf",
"Destination": "/var/resolv.conf",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/home",
"Destination": "/projects",
"Mode": "",
"RW": true,
"Propagation": "rslave"
},
{
"Type": "volume",
"Name": "k3d-local-images",
"Source": "/home/docker-images/volumes/k3d-local-images/_data",
"Destination": "/k3d/images",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
},
{
"Type": "volume",
"Name": "6554fec529be600aebc4219f07891c27ca6a12d1d51d36566b104ed65afd4996",
"Source": "/home/docker-images/volumes/6554fec529be600aebc4219f07891c27ca6a12d1d51d36566b104ed65afd4996/_data",
"Destination": "/var/lib/kubelet",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
},
{
"Type": "volume",
"Name": "e86a826c0420a7033f933d396b60ed6948c1d4a28b897b6a3e64dc47871827db",
"Source": "/home/docker-images/volumes/e86a826c0420a7033f933d396b60ed6948c1d4a28b897b6a3e64dc47871827db/_data",
"Destination": "/var/log",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
},
{
"Type": "bind",
"Source": "/home/projects/k3d/CA-kube.crt",
"Destination": "/etc/ssl/certs/ca.pem",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
}
],
"Config": {
"Hostname": "k3d-local-server-0",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"bar=baz",
"K3S_TOKEN=superSecretToken",
"K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/bin/aux",
"CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml"
],
"Cmd": [
"server",
"--tls-san=back-local.kube.com",
"--cluster-init",
"--cluster-domain=kube.com",
"--disable=traefik",
"--resolv-conf=/var/resolv.conf",
"--tls-san",
"back-local.kube.com"
],
"Image": "rancher/k3s:v1.21.10-k3s1",
"Volumes": {
"/var/lib/cni": {},
"/var/lib/kubelet": {},
"/var/lib/rancher/k3s": {},
"/var/log": {}
},
"WorkingDir": "",
"Entrypoint": [
"/bin/k3s"
],
"OnBuild": null,
"Labels": {
"app": "k3d",
"k3d.cluster": "local",
"k3d.cluster.imageVolume": "k3d-local-images",
"k3d.cluster.network": "k3d-local",
"k3d.cluster.network.external": "false",
"k3d.cluster.network.id": "d29ad6e8de48162f149e07fd956c5deb59f335dea88ccd6c49475ecb368e6507",
"k3d.cluster.network.iprange": "172.40.0.0/16",
"k3d.cluster.token": "superSecretToken",
"k3d.cluster.url": "https://k3d-local-server-0:6443",
"k3d.node.staticIP": "172.40.0.3",
"k3d.role": "server",
"k3d.server.api.host": "back-local.kube.com",
"k3d.server.api.hostIP": "0.0.0.0",
"k3d.server.api.port": "36145",
"k3d.version": "v5.3.0",
"org.opencontainers.image.created": "2022-02-24T23:35:30Z",
"org.opencontainers.image.revision": "471f5eb3dbfeaee6b6dd6ed9ab4037c10cc39680",
"org.opencontainers.image.source": "https://github.com/k3s-io/k3s.git",
"org.opencontainers.image.url": "https://github.com/k3s-io/k3s"
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "e0e8f724abe0e51bcf123e5b21e07b40f64c28ec2cc7294eae16668f1b9b5367",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/var/run/docker/netns/e0e8f724abe0",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"k3d-local": {
"IPAMConfig": {
"IPv4Address": "172.40.0.3"
},
"Links": null,
"Aliases": [
"f03c57557f5e",
"k3d-local-server-0"
],
"NetworkID": "d29ad6e8de48162f149e07fd956c5deb59f335dea88ccd6c49475ecb368e6507",
"EndpointID": "95d2c18a81c178b1beac39b6be429755409c9bb7c9c76a05301a1c63a805664e",
"Gateway": "172.40.0.1",
"IPAddress": "172.40.0.3",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:28:00:03",
"DriverOpts": null
}
}
}
}
]
When I create a new AGENT node in this cluster, the nodeHost is not modified:
projects k3d k3d node create newserver --cluster local --role agent
INFO[0000] Adding 1 node(s) to the runtime local cluster 'local'...
INFO[0000] Using the k3d-tools node to gather environment information
INFO[0000] Starting new tools node...
INFO[0000] Starting Node 'k3d-local-tools'
INFO[0000] HostIP: using network gateway 172.40.0.1 address
INFO[0000] Starting Node 'k3d-newserver-0'
INFO[0009] Successfully created 1 node(s)!
when I create a new SERVER node, the configMap is broken
Okay, now that makes more sense. From your initial post I couldn't see that you were using a config file. I can verify that that's an issue. It's quite weird though that K3s overwrites the already deployed CoreDNS config :thinking: Maybe that's worth an upstream issue.
This is indeed the intended behavior on K3s' end (thanks @brandond for the explanation :pray: ) However, there are many ways this can be tackled on k3d's side, so let's see what nice solution we can come up with for the next release :+1:
Hi, I found this issue while investigating why when K3s containers created via K3d would not maintain the CoreDNS NodeHosts ConfigMap on host machine reboot. Could it be the same issue?
From my testing the issues comes down to the K3s starting by themselves without K3d orchestraction and it rewrites CoreDNS ConfigMap to only contain k3d-test-server-0 for my 1-server cluster.
and i think i've this issue, too, in a different scenario: just a k3s container restart, or docker restart, triggers the configmap to be emptied... issue here: 1112
