virtual-kubelet icon indicating copy to clipboard operation
virtual-kubelet copied to clipboard

Published 20 hours ago • verified publisher icon virtual-kubelet

Multiple NodeAddresses of one NodeAddressType causes merge corruption

Open danopia opened this issue 4 years ago • 6 comments

Environment summary

Not too relevant Provider (e.g. ACI, AWS Fargate): Bare Metal

Version (e.g. 0.1, 0.2-beta): v1.3.0

K8s Master Info (e.g. AKS, ACS, Bare Metal, EKS): GKE

Install Method (e.g. Helm Chart, ): manual

Issue Details

I have VK nodes with dual stack networking and noticed that it was surprisingly tricky to add both IP addresses to nodestatus with the same NodeAddressType value.

If I added both items in successive updates, the mergekey apparently results in one replacing the other. I had to introduce both ExternalIP entries at the same time and it works initially, but changing either address later will replace the first one.

It seems like this patch would need to be adapted over to node.go to fix the problem I'm seeing: https://github.com/kubernetes/kubernetes/pull/79391

I could try that myself, but wanted to submit this issue first to see if anyone else has dealt with this and has other advice.

Repo Steps

Example nodestatus updater that changes the second address after startup (IPv6 renumber):

func (np *MyNodeProvider) NotifyNodeStatus(ctx context.Context, f func(*corev1.Node)) {
	go func() {

		ticker := time.NewTicker(1 * time.Minute)
		firstRunC := make(chan struct{})
		close(firstRunC)

		np.nodeStatus.Addresses = []corev1.NodeAddress{
			{Type: corev1.NodeExternalIP, Address: "8.8.8.8"},
			{Type: corev1.NodeExternalIP, Address: "1::1"},
		}

		for {
			select {

			case <-ticker.C:
				log.Println("NodeIdentity: Performing periodic status refresh")
				np.nodeStatus.Addresses[1].Address = "2::1"

			case <-firstRunC:
				log.Println("NodeIdentity: Reporting initial NodeStatus")
			}
			firstRunC = nil

			// actually report
			newNode := &corev1.Node{
				ObjectMeta: metav1.ObjectMeta{
					Annotations: np.node.Annotations,
					Labels:      np.node.Labels,
				},
			}
			np.nodeStatus.DeepCopyInto(&newNode.Status)
			f(newNode)
			log.Println("NodeIdentity: Node status updated!")
		}
	}()
}

After startup:

$ kubectl get node -o yaml | grep -A1 address
  addresses:
  - address: 8.8.8.8
    type: ExternalIP
  - address: 1::1
    type: ExternalIP

Soon after:

$ kubectl get node -o yaml | grep -A1 address
  addresses:
  - address: 2::1
    type: ExternalIP
  - address: 1::1
    type: ExternalIP

Notice how the IPv4 is just gone now 😄 It's still in the virtual-kubelet.io/last-applied-node-status annotation though, which has an accurate picture of what's going on.

danopia avatar Nov 28 '20 22:11 danopia

Can you turn on auditing on our cluster and give us a dump of the bootstrap?

sargun avatar Nov 28 '20 22:11 sargun

It seems like GKE's operation logging isn't showing node updates, so I'm only seeing VK perform lease operations on startup: Screenshot 2020-11-29 at 00 29 23

Seems like I'll have to stand up a different kubernetes to capture apiserver-side logs of the operations at play

danopia avatar Nov 28 '20 23:11 danopia

How did you build your VK? Can you pass klog options / set klog debug to 100?

sargun avatar Nov 29 '20 00:11 sargun

I have a pretty straightforward VK I've written so I was able to add the klog flags and pass -v 100 which is what I assume you refer to. I had to chop the bootstrap up pretty carefully because of the startup request storm haha

Relevant requests of bootstrap 
I1129 01:35:49.198957    6073 round_trippers.go:423] curl -k -v -XGET  -H "Authorization: Bearer [REDACTED]" -H "Accept: application/json, */*" -H "User-Agent: kube-pet-node/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://[REDACTED]/api/v1/nodes/pet-penguin'
I1129 01:35:49.707591    6073 round_trippers.go:443] GET https://[REDACTED]/api/v1/nodes/pet-penguin 404 Not Found in 508 milliseconds
I1129 01:35:49.708559    6073 round_trippers.go:449] Response Headers:
I1129 01:35:49.709103    6073 round_trippers.go:452]     Content-Type: application/json
I1129 01:35:49.709540    6073 round_trippers.go:452]     Content-Length: 192
I1129 01:35:49.710065    6073 round_trippers.go:452]     Date: Sun, 29 Nov 2020 00:35:49 GMT
I1129 01:35:49.710499    6073 round_trippers.go:452]     Audit-Id: dc36682d-de9b-4072-9e59-870480a83523
I1129 01:35:49.711021    6073 round_trippers.go:452]     Cache-Control: no-cache, private
I1129 01:35:49.711534    6073 request.go:1068] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"nodes \"pet-penguin\" not found","reason":"NotFound","details":{"name":"pet-penguin","kind":"nodes"},"code":404}
...
I1129 01:35:49.719441    6073 request.go:1068] Request Body: {"kind":"Node","apiVersion":"v1","metadata":{"name":"pet-penguin","creationTimestamp":null,"labels":{"kubernetes.io/arch":"amd64","kubernetes.io/hostname":"penguin","kubernetes.io/os":"linux","kubernetes.io/role":"pet","lifetime":"persistent","purpose":"pet","type":"virtual-kubelet"},"annotations":{"kubernetes.io/node.class":"kube-pet"}},"spec":{"providerID":"gce://kube-pet-node/bare-metal/pet-penguin","taints":[{"key":"kubernetes.io/pet-node","value":"pet-penguin","effect":"NoSchedule"}]},"status":{"daemonEndpoints":{"kubeletEndpoint":{"Port":0}},"nodeInfo":{"machineID":"","systemUUID":"","bootID":"","kernelVersion":"","osImage":"","containerRuntimeVersion":"","kubeletVersion":"","kubeProxyVersion":"","operatingSystem":"","architecture":""}}}
I1129 01:35:49.728511    6073 round_trippers.go:423] curl -k -v -XPOST  -H "Content-Type: application/json" -H "User-Agent: kube-pet-node/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Accept: application/json, */*" -H "Authorization: Bearer [REDACTED]" 'https://[REDACTED]/api/v1/nodes'
I1129 01:35:49.913762    6073 round_trippers.go:443] POST https://[REDACTED]/api/v1/nodes 201 Created in 185 milliseconds
I1129 01:35:49.913812    6073 round_trippers.go:449] Response Headers:
I1129 01:35:49.913824    6073 round_trippers.go:452]     Content-Length: 947
I1129 01:35:49.913833    6073 round_trippers.go:452]     Date: Sun, 29 Nov 2020 00:35:50 GMT
I1129 01:35:49.913841    6073 round_trippers.go:452]     Audit-Id: ecd09a3a-d6ae-4f1c-85fa-08716a026dcd
I1129 01:35:49.913849    6073 round_trippers.go:452]     Cache-Control: no-cache, private
I1129 01:35:49.913856    6073 round_trippers.go:452]     Content-Type: application/json
I1129 01:35:49.913899    6073 request.go:1068] Response Body: {"kind":"Node","apiVersion":"v1","metadata":{"name":"pet-penguin","selfLink":"/api/v1/nodes/pet-penguin","uid":"ec42794b-16f5-471e-821f-be2b38724098","resourceVersion":"206249047","creationTimestamp":"2020-11-29T00:35:50Z","labels":{"kubernetes.io/arch":"amd64","kubernetes.io/hostname":"penguin","kubernetes.io/os":"linux","kubernetes.io/role":"pet","lifetime":"persistent","purpose":"pet","type":"virtual-kubelet"},"annotations":{"kubernetes.io/node.class":"kube-pet"}},"spec":{"providerID":"gce://kube-pet-node/bare-metal/pet-penguin","taints":[{"key":"kubernetes.io/pet-node","value":"pet-penguin","effect":"NoSchedule"},{"key":"node.kubernetes.io/not-ready","effect":"NoSchedule"}]},"status":{"daemonEndpoints":{"kubeletEndpoint":{"Port":0}},"nodeInfo":{"machineID":"","systemUUID":"","bootID":"","kernelVersion":"","osImage":"","containerRuntimeVersion":"","kubeletVersion":"","kubeProxyVersion":"","operatingSystem":"","architecture":""}}}
...
2020/11/29 01:35:50 K8S Event: Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"pet-penguin", UID:"pet-penguin", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'Starting' Starting kube-pet-node.
...
I1129 01:35:50.432079    6073 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kube-pet-node/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer [REDACTED]" 'https://[REDACTED]/api/v1/nodes/pet-penguin'
I1129 01:35:50.634242    6073 round_trippers.go:443] GET https://[REDACTED]/api/v1/nodes/pet-penguin 200 OK in 202 milliseconds
I1129 01:35:50.634252    6073 round_trippers.go:449] Response Headers:
I1129 01:35:50.634259    6073 round_trippers.go:452]     Audit-Id: 774fb334-e70c-428f-bffc-552758e66b61
I1129 01:35:50.634266    6073 round_trippers.go:452]     Cache-Control: no-cache, private
I1129 01:35:50.634273    6073 round_trippers.go:452]     Content-Type: application/json
I1129 01:35:50.634280    6073 round_trippers.go:452]     Content-Length: 921
I1129 01:35:50.634287    6073 round_trippers.go:452]     Date: Sun, 29 Nov 2020 00:35:50 GMT
I1129 01:35:50.634323    6073 request.go:1068] Response Body: {"kind":"Node","apiVersion":"v1","metadata":{"name":"pet-penguin","selfLink":"/api/v1/nodes/pet-penguin","uid":"ec42794b-16f5-471e-821f-be2b38724098","resourceVersion":"206249049","creationTimestamp":"2020-11-29T00:35:50Z","labels":{"kubernetes.io/arch":"amd64","kubernetes.io/hostname":"penguin","kubernetes.io/os":"linux","kubernetes.io/role":"pet","lifetime":"persistent","purpose":"pet","type":"virtual-kubelet"},"annotations":{"kubernetes.io/node.class":"kube-pet","node.alpha.kubernetes.io/ttl":"0"}},"spec":{"providerID":"gce://kube-pet-node/bare-metal/pet-penguin","taints":[{"key":"kubernetes.io/pet-node","value":"pet-penguin","effect":"NoSchedule"}]},"status":{"daemonEndpoints":{"kubeletEndpoint":{"Port":0}},"nodeInfo":{"machineID":"","systemUUID":"","bootID":"","kernelVersion":"","osImage":"","containerRuntimeVersion":"","kubeletVersion":"","kubeProxyVersion":"","operatingSystem":"","architecture":""}}}
...
I1129 01:35:50.641533    6073 request.go:1068] Request Body: {"metadata":{"annotations":{"virtual-kubelet.io/last-applied-node-status":"{\"capacity\":{\"cpu\":\"2\",\"ephemeral-storage\":\"10Gi\",\"hugepages-2Mi\":\"0\",\"memory\":\"14902712Ki\",\"pods\":\"0\"},\"allocatable\":{\"cpu\":\"1\",\"ephemeral-storage\":\"1Gi\",\"hugepages-2Mi\":\"0\",\"memory\":\"7451356Ki\",\"pods\":\"0\"},\"conditions\":[{\"type\":\"Ready\",\"status\":\"True\",\"lastHeartbeatTime\":\"2020-11-29T00:35:50Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"KubeletReady\",\"message\":\"Hello World\"},{\"type\":\"MemoryPressure\",\"status\":\"False\",\"lastHeartbeatTime\":\"2020-11-29T00:35:50Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"OK\",\"message\":\"Hello World\"},{\"type\":\"DiskPressure\",\"status\":\"False\",\"lastHeartbeatTime\":\"2020-11-29T00:35:50Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"OK\",\"message\":\"Hello World\"},{\"type\":\"PIDPressure\",\"status\":\"False\",\"lastHeartbeatTime\":\"2020-11-29T00:35:50Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"OK\",\"message\":\"Hello World\"},{\"type\":\"NetworkUnavailable\",\"status\":\"False\",\"lastHeartbeatTime\":\"2020-11-29T00:35:50Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"OK\",\"message\":\"Hello World\"}],\"addresses\":[{\"type\":\"Hostname\",\"address\":\"pet-penguin\"},{\"type\":\"InternalIP\",\"address\":\"10.1.1.1\"},{\"type\":\"InternalDNS\",\"address\":\"pet-penguin.local\"},{\"type\":\"ExternalIP\",\"address\":\"8.8.8.8\"},{\"type\":\"ExternalIP\",\"address\":\"1::1\"}],\"daemonEndpoints\":{\"kubeletEndpoint\":{\"Port\":10250}},\"nodeInfo\":{\"machineID\":\"[redacted]\",\"systemUUID\":\"\",\"bootID\":\"dc714fe0-8557-4163-9b25-7bb3e74cea66\",\"kernelVersion\":\"5.4.58-07649-ge120df5deade\",\"osImage\":\"Debian GNU/Linux 10 (buster)\",\"containerRuntimeVersion\":\"podman://2.1.1\",\"kubeletVersion\":\"kube-pet/development\",\"kubeProxyVersion\":\"nftables-pet/development\",\"operatingSystem\":\"linux\",\"architecture\":\"amd64\"}}","virtual-kubelet.io/last-applied-object-meta":"{\"name\":\"pet-penguin\",\"uid\":\"ec42794b-16f5-471e-821f-be2b38724098\",\"creationTimestamp\":null,\"labels\":{\"kubernetes.io/arch\":\"amd64\",\"kubernetes.io/hostname\":\"penguin\",\"kubernetes.io/os\":\"linux\",\"kubernetes.io/role\":\"pet\",\"lifetime\":\"persistent\",\"purpose\":\"pet\",\"type\":\"virtual-kubelet\"},\"annotations\":{\"kubernetes.io/node.class\":\"kube-pet\"}}"},"creationTimestamp":null},"status":{"addresses":[{"address":"pet-penguin","type":"Hostname"},{"address":"10.1.1.1","type":"InternalIP"},{"address":"pet-penguin.local","type":"InternalDNS"},{"address":"8.8.8.8","type":"ExternalIP"},{"address":"1::1","type":"ExternalIP"}],"allocatable":{"cpu":"1","ephemeral-storage":"1Gi","hugepages-2Mi":"0","memory":"7451356Ki","pods":"0"},"capacity":{"cpu":"2","ephemeral-storage":"10Gi","hugepages-2Mi":"0","memory":"14902712Ki","pods":"0"},"conditions":[{"lastHeartbeatTime":"2020-11-29T00:35:50Z","lastTransitionTime":"2020-11-29T00:35:47Z","message":"Hello World","reason":"KubeletReady","status":"True","type":"Ready"},{"lastHeartbeatTime":"2020-11-29T00:35:50Z","lastTransitionTime":"2020-11-29T00:35:47Z","message":"Hello World","reason":"OK","status":"False","type":"MemoryPressure"},{"lastHeartbeatTime":"2020-11-29T00:35:50Z","lastTransitionTime":"2020-11-29T00:35:47Z","message":"Hello World","reason":"OK","status":"False","type":"DiskPressure"},{"lastHeartbeatTime":"2020-11-29T00:35:50Z","lastTransitionTime":"2020-11-29T00:35:47Z","message":"Hello World","reason":"OK","status":"False","type":"PIDPressure"},{"lastHeartbeatTime":"2020-11-29T00:35:50Z","lastTransitionTime":"2020-11-29T00:35:47Z","message":"Hello World","reason":"OK","status":"False","type":"NetworkUnavailable"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"architecture":"amd64","bootID":"dc714fe0-8557-4163-9b25-7bb3e74cea66","containerRuntimeVersion":"podman://2.1.1","kernelVersion":"5.4.58-07649-ge120df5deade","kubeProxyVersion":"nftables-pet/development","kubeletVersion":"kube-pet/development","machineID":"[redacted]","operatingSystem":"linux","osImage":"Debian GNU/Linux 10 (buster)"}}}
I1129 01:35:50.641995    6073 round_trippers.go:423] curl -k -v -XPATCH  -H "Accept: application/json, */*" -H "Content-Type: application/strategic-merge-patch+json" -H "User-Agent: kube-pet-node/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer [REDACTED]" 'https://[REDACTED]/api/v1/nodes/pet-penguin/status'
I1129 01:35:50.908986    6073 round_trippers.go:443] PATCH https://[REDACTED]/api/v1/nodes/pet-penguin/status 200 OK in 266 milliseconds
I1129 01:35:50.909028    6073 round_trippers.go:449] Response Headers:
I1129 01:35:50.909044    6073 round_trippers.go:452]     Cache-Control: no-cache, private
I1129 01:35:50.909053    6073 round_trippers.go:452]     Content-Type: application/json
I1129 01:35:50.909096    6073 round_trippers.go:452]     Date: Sun, 29 Nov 2020 00:35:50 GMT
I1129 01:35:50.909106    6073 round_trippers.go:452]     Audit-Id: 4ea2c963-cc5e-4299-bfcc-34485145815c
I1129 01:35:50.909606    6073 request.go:1068] Response Body: {"kind":"Node","apiVersion":"v1","metadata":{"name":"pet-penguin","selfLink":"/api/v1/nodes/pet-penguin/status","uid":"ec42794b-16f5-471e-821f-be2b38724098","resourceVersion":"206249056","creationTimestamp":"2020-11-29T00:35:50Z","labels":{"kubernetes.io/arch":"amd64","kubernetes.io/hostname":"penguin","kubernetes.io/os":"linux","kubernetes.io/role":"pet","lifetime":"persistent","purpose":"pet","type":"virtual-kubelet"},"annotations":{"kubernetes.io/node.class":"kube-pet","node.alpha.kubernetes.io/ttl":"0","virtual-kubelet.io/last-applied-node-status":"{\"capacity\":{\"cpu\":\"2\",\"ephemeral-storage\":\"10Gi\",\"hugepages-2Mi\":\"0\",\"memory\":\"14902712Ki\",\"pods\":\"0\"},\"allocatable\":{\"cpu\":\"1\",\"ephemeral-storage\":\"1Gi\",\"hugepages-2Mi\":\"0\",\"memory\":\"7451356Ki\",\"pods\":\"0\"},\"conditions\":[{\"type\":\"Ready\",\"status\":\"True\",\"lastHeartbeatTime\":\"2020-11-29T00:35:50Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"KubeletReady\",\"message\":\"Hello World\"},{\"type\":\"MemoryPressure\",\"status\":\"False\",\"lastHeartbeatTime\":\"2020-11-29T00:35:50Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"OK\",\"message\":\"Hello World\"},{\"type\":\"DiskPressure\",\"status\":\"False\",\"lastHeartbeatTime\":\"2020-11-29T00:35:50Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"OK\",\"message\":\"Hello World\"},{\"type\":\"PIDPressure\",\"status\":\"False\",\"lastHeartbeatTime\":\"2020-11-29T00:35:50Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"OK\",\"message\":\"Hello World\"},{\"type\":\"NetworkUnavailable\",\"status\":\"False\",\"lastHeartbeatTime\":\"2020-11-29T00:35:50Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"OK\",\"message\":\"Hello World\"}],\"addresses\":[{\"type\":\"Hostname\",\"address\":\"pet-penguin\"},{\"type\":\"InternalIP\",\"address\":\"10.1.1.1\"},{\"type\":\"InternalDNS\",\"address\":\"pet-penguin.local\"},{\"type\":\"ExternalIP\",\"address\":\"8.8.8.8\"},{\"type\":\"ExternalIP\",\"address\":\"1::1\"}],\"daemonEndpoints\":{\"kubeletEndpoint\":{\"Port\":10250}},\"nodeInfo\":{\"machineID\":\"[redacted]\",\"systemUUID\":\"\",\"bootID\":\"dc714fe0-8557-4163-9b25-7bb3e74cea66\",\"kernelVersion\":\"5.4.58-07649-ge120df5deade\",\"osImage\":\"Debian GNU/Linux 10 (buster)\",\"containerRuntimeVersion\":\"podman://2.1.1\",\"kubeletVersion\":\"kube-pet/development\",\"kubeProxyVersion\":\"nftables-pet/development\",\"operatingSystem\":\"linux\",\"architecture\":\"amd64\"}}","virtual-kubelet.io/last-applied-object-meta":"{\"name\":\"pet-penguin\",\"uid\":\"ec42794b-16f5-471e-821f-be2b38724098\",\"creationTimestamp\":null,\"labels\":{\"kubernetes.io/arch\":\"amd64\",\"kubernetes.io/hostname\":\"penguin\",\"kubernetes.io/os\":\"linux\",\"kubernetes.io/role\":\"pet\",\"lifetime\":\"persistent\",\"purpose\":\"pet\",\"type\":\"virtual-kubelet\"},\"annotations\":{\"kubernetes.io/node.class\":\"kube-pet\"}}"}},"spec":{"providerID":"gce://kube-pet-node/bare-metal/pet-penguin","taints":[{"key":"kubernetes.io/pet-node","value":"pet-penguin","effect":"NoSchedule"}]},"status":{"capacity":{"cpu":"2","ephemeral-storage":"10Gi","hugepages-2Mi":"0","memory":"14902712Ki","pods":"0"},"allocatable":{"cpu":"1","ephemeral-storage":"1Gi","hugepages-2Mi":"0","memory":"7451356Ki","pods":"0"},"conditions":[{"type":"Ready","status":"True","lastHeartbeatTime":"2020-11-29T00:35:50Z","lastTransitionTime":"2020-11-29T00:35:47Z","reason":"KubeletReady","message":"Hello World"},{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2020-11-29T00:35:50Z","lastTransitionTime":"2020-11-29T00:35:47Z","reason":"OK","message":"Hello World"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2020-11-29T00:35:50Z","lastTransitionTime":"2020-11-29T00:35:47Z","reason":"OK","message":"Hello World"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2020-11-29T00:35:50Z","lastTransitionTime":"2020-11-29T00:35:47Z","reason":"OK","message":"Hello World"},{"type":"NetworkUnavailable","status":"False","lastHeartbeatTime":"2020-11-29T00:35:50Z","lastTransitionTime":"2020-11-29T00:35:47Z","reason":"OK","message":"Hello World"}],"addresses":[{"type":"Hostname","address":"pet-penguin"},{"type":"InternalIP","address":"10.1.1.1"},{"type":"InternalDNS","address":"pet-penguin.local"},{"type":"ExternalIP","address":"8.8.8.8"},{"type":"ExternalIP","address":"1::1"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"[redacted]","systemUUID":"","bootID":"dc714fe0-8557-4163-9b25-7bb3e74cea66","kernelVersion":"5.4.58-07649-ge120df5deade","osImage":"Debian GNU/Linux 10 (buster)","containerRuntimeVersion":"podman://2.1.1","kubeletVersion":"kube-pet/development","kubeProxyVersion":"nftables-pet/development","operatingSystem":"linux","architecture":"amd64"}}}
Relevant bits of a IP change 
2020/11/29 01:36:47 NodeIdentity: Discovered our IPv6 Address as 2::1
2020/11/29 01:36:47 NodeIdentity: Updated ExternalIP V6 in our status
2020/11/29 01:36:47 NodeIdentity: Node status updated!
...
I1129 01:36:47.732279    6073 round_trippers.go:423] curl -k -v -XGET  -H "User-Agent: kube-pet-node/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Accept: application/json, */*" -H "Authorization: Bearer [REDACTED]" 'https://[REDACTED]/api/v1/nodes/pet-penguin'
I1129 01:36:47.877044    6073 round_trippers.go:443] GET https://[REDACTED]/api/v1/nodes/pet-penguin 200 OK in 144 milliseconds
I1129 01:36:47.877079    6073 round_trippers.go:449] Response Headers:
I1129 01:36:47.877091    6073 round_trippers.go:452]     Audit-Id: e8b13631-0fbf-4838-814c-e940ee2d167b
I1129 01:36:47.877100    6073 round_trippers.go:452]     Cache-Control: no-cache, private
I1129 01:36:47.877112    6073 round_trippers.go:452]     Content-Type: application/json
I1129 01:36:47.877121    6073 round_trippers.go:452]     Date: Sun, 29 Nov 2020 00:36:48 GMT
I1129 01:36:47.877279    6073 request.go:1068] Response Body: {"kind":"Node","apiVersion":"v1","metadata":{"name":"pet-penguin","selfLink":"/api/v1/nodes/pet-penguin","uid":"ec42794b-16f5-471e-821f-be2b38724098","resourceVersion":"206249056","creationTimestamp":"2020-11-29T00:35:50Z","labels":{"kubernetes.io/arch":"amd64","kubernetes.io/hostname":"penguin","kubernetes.io/os":"linux","kubernetes.io/role":"pet","lifetime":"persistent","purpose":"pet","type":"virtual-kubelet"},"annotations":{"kubernetes.io/node.class":"kube-pet","node.alpha.kubernetes.io/ttl":"0","virtual-kubelet.io/last-applied-node-status":"{\"capacity\":{\"cpu\":\"2\",\"ephemeral-storage\":\"10Gi\",\"hugepages-2Mi\":\"0\",\"memory\":\"14902712Ki\",\"pods\":\"0\"},\"allocatable\":{\"cpu\":\"1\",\"ephemeral-storage\":\"1Gi\",\"hugepages-2Mi\":\"0\",\"memory\":\"7451356Ki\",\"pods\":\"0\"},\"conditions\":[{\"type\":\"Ready\",\"status\":\"True\",\"lastHeartbeatTime\":\"2020-11-29T00:35:50Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"KubeletReady\",\"message\":\"Hello World\"},{\"type\":\"MemoryPressure\",\"status\":\"False\",\"lastHeartbeatTime\":\"2020-11-29T00:35:50Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"OK\",\"message\":\"Hello World\"},{\"type\":\"DiskPressure\",\"status\":\"False\",\"lastHeartbeatTime\":\"2020-11-29T00:35:50Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"OK\",\"message\":\"Hello World\"},{\"type\":\"PIDPressure\",\"status\":\"False\",\"lastHeartbeatTime\":\"2020-11-29T00:35:50Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"OK\",\"message\":\"Hello World\"},{\"type\":\"NetworkUnavailable\",\"status\":\"False\",\"lastHeartbeatTime\":\"2020-11-29T00:35:50Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"OK\",\"message\":\"Hello World\"}],\"addresses\":[{\"type\":\"Hostname\",\"address\":\"pet-penguin\"},{\"type\":\"InternalIP\",\"address\":\"10.1.1.1\"},{\"type\":\"InternalDNS\",\"address\":\"pet-penguin.local\"},{\"type\":\"ExternalIP\",\"address\":\"8.8.8.8\"},{\"type\":\"ExternalIP\",\"address\":\"1::1\"}],\"daemonEndpoints\":{\"kubeletEndpoint\":{\"Port\":10250}},\"nodeInfo\":{\"machineID\":\"[redacted]\",\"systemUUID\":\"\",\"bootID\":\"dc714fe0-8557-4163-9b25-7bb3e74cea66\",\"kernelVersion\":\"5.4.58-07649-ge120df5deade\",\"osImage\":\"Debian GNU/Linux 10 (buster)\",\"containerRuntimeVersion\":\"podman://2.1.1\",\"kubeletVersion\":\"kube-pet/development\",\"kubeProxyVersion\":\"nftables-pet/development\",\"operatingSystem\":\"linux\",\"architecture\":\"amd64\"}}","virtual-kubelet.io/last-applied-object-meta":"{\"name\":\"pet-penguin\",\"uid\":\"ec42794b-16f5-471e-821f-be2b38724098\",\"creationTimestamp\":null,\"labels\":{\"kubernetes.io/arch\":\"amd64\",\"kubernetes.io/hostname\":\"penguin\",\"kubernetes.io/os\":\"linux\",\"kubernetes.io/role\":\"pet\",\"lifetime\":\"persistent\",\"purpose\":\"pet\",\"type\":\"virtual-kubelet\"},\"annotations\":{\"kubernetes.io/node.class\":\"kube-pet\"}}"}},"spec":{"providerID":"gce://kube-pet-node/bare-metal/pet-penguin","taints":[{"key":"kubernetes.io/pet-node","value":"pet-penguin","effect":"NoSchedule"}]},"status":{"capacity":{"cpu":"2","ephemeral-storage":"10Gi","hugepages-2Mi":"0","memory":"14902712Ki","pods":"0"},"allocatable":{"cpu":"1","ephemeral-storage":"1Gi","hugepages-2Mi":"0","memory":"7451356Ki","pods":"0"},"conditions":[{"type":"Ready","status":"True","lastHeartbeatTime":"2020-11-29T00:35:50Z","lastTransitionTime":"2020-11-29T00:35:47Z","reason":"KubeletReady","message":"Hello World"},{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2020-11-29T00:35:50Z","lastTransitionTime":"2020-11-29T00:35:47Z","reason":"OK","message":"Hello World"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2020-11-29T00:35:50Z","lastTransitionTime":"2020-11-29T00:35:47Z","reason":"OK","message":"Hello World"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2020-11-29T00:35:50Z","lastTransitionTime":"2020-11-29T00:35:47Z","reason":"OK","message":"Hello World"},{"type":"NetworkUnavailable","status":"False","lastHeartbeatTime":"2020-11-29T00:35:50Z","lastTransitionTime":"2020-11-29T00:35:47Z","reason":"OK","message":"Hello World"}],"addresses":[{"type":"Hostname","address":"pet-penguin"},{"type":"InternalIP","address":"10.1.1.1"},{"type":"InternalDNS","address":"pet-penguin.local"},{"type":"ExternalIP","address":"8.8.8.8"},{"type":"ExternalIP","address":"1::1"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"[redacted]","systemUUID":"","bootID":"dc714fe0-8557-4163-9b25-7bb3e74cea66","kernelVersion":"5.4.58-07649-ge120df5deade","osImage":"Debian GNU/Linux 10 (buster)","containerRuntimeVersion":"podman://2.1.1","kubeletVersion":"kube-pet/development","kubeProxyVersion":"nftables-pet/development","operatingSystem":"linux","architecture":"amd64"}}}
...
I1129 01:36:47.879355    6073 request.go:1068] Request Body: {"metadata":{"annotations":{"virtual-kubelet.io/last-applied-node-status":"{\"capacity\":{\"cpu\":\"2\",\"ephemeral-storage\":\"10Gi\",\"hugepages-2Mi\":\"0\",\"memory\":\"14902712Ki\",\"pods\":\"0\"},\"allocatable\":{\"cpu\":\"1\",\"ephemeral-storage\":\"1Gi\",\"hugepages-2Mi\":\"0\",\"memory\":\"7451356Ki\",\"pods\":\"0\"},\"conditions\":[{\"type\":\"Ready\",\"status\":\"True\",\"lastHeartbeatTime\":\"2020-11-29T00:36:47Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"KubeletReady\",\"message\":\"Hello World\"},{\"type\":\"MemoryPressure\",\"status\":\"False\",\"lastHeartbeatTime\":\"2020-11-29T00:36:47Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"OK\",\"message\":\"Hello World\"},{\"type\":\"DiskPressure\",\"status\":\"False\",\"lastHeartbeatTime\":\"2020-11-29T00:36:47Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"OK\",\"message\":\"Hello World\"},{\"type\":\"PIDPressure\",\"status\":\"False\",\"lastHeartbeatTime\":\"2020-11-29T00:36:47Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"OK\",\"message\":\"Hello World\"},{\"type\":\"NetworkUnavailable\",\"status\":\"False\",\"lastHeartbeatTime\":\"2020-11-29T00:36:47Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"OK\",\"message\":\"Hello World\"}],\"addresses\":[{\"type\":\"Hostname\",\"address\":\"pet-penguin\"},{\"type\":\"InternalIP\",\"address\":\"10.1.1.1\"},{\"type\":\"InternalDNS\",\"address\":\"pet-penguin.local\"},{\"type\":\"ExternalIP\",\"address\":\"8.8.8.8\"},{\"type\":\"ExternalIP\",\"address\":\"2::1\"}],\"daemonEndpoints\":{\"kubeletEndpoint\":{\"Port\":10250}},\"nodeInfo\":{\"machineID\":\"[redacted]\",\"systemUUID\":\"\",\"bootID\":\"dc714fe0-8557-4163-9b25-7bb3e74cea66\",\"kernelVersion\":\"5.4.58-07649-ge120df5deade\",\"osImage\":\"Debian GNU/Linux 10 (buster)\",\"containerRuntimeVersion\":\"podman://2.1.1\",\"kubeletVersion\":\"kube-pet/development\",\"kubeProxyVersion\":\"nftables-pet/development\",\"operatingSystem\":\"linux\",\"architecture\":\"amd64\"}}"},"creationTimestamp":null},"status":{"$setElementOrder/addresses":[{"type":"Hostname"},{"type":"InternalIP"},{"type":"InternalDNS"},{"type":"ExternalIP"},{"type":"ExternalIP"}],"$setElementOrder/conditions":[{"type":"Ready"},{"type":"MemoryPressure"},{"type":"DiskPressure"},{"type":"PIDPressure"},{"type":"NetworkUnavailable"}],"addresses":[{"address":"2::1","type":"ExternalIP"}],"conditions":[{"lastHeartbeatTime":"2020-11-29T00:36:47Z","type":"Ready"},{"lastHeartbeatTime":"2020-11-29T00:36:47Z","type":"MemoryPressure"},{"lastHeartbeatTime":"2020-11-29T00:36:47Z","type":"DiskPressure"},{"lastHeartbeatTime":"2020-11-29T00:36:47Z","type":"PIDPressure"},{"lastHeartbeatTime":"2020-11-29T00:36:47Z","type":"NetworkUnavailable"}]}}
I1129 01:36:47.879484    6073 round_trippers.go:423] curl -k -v -XPATCH  -H "Accept: application/json, */*" -H "Authorization: Bearer [REDACTED]" -H "User-Agent: kube-pet-node/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Content-Type: application/strategic-merge-patch+json" 'https://[REDACTED]/api/v1/nodes/pet-penguin/status'
I1129 01:36:48.097210    6073 round_trippers.go:443] PATCH https://[REDACTED]/api/v1/nodes/pet-penguin/status 200 OK in 217 milliseconds
I1129 01:36:48.097269    6073 round_trippers.go:449] Response Headers:
I1129 01:36:48.097282    6073 round_trippers.go:452]     Audit-Id: 630f0419-442b-484a-af76-2bfdaf81fa85
I1129 01:36:48.097293    6073 round_trippers.go:452]     Cache-Control: no-cache, private
I1129 01:36:48.097302    6073 round_trippers.go:452]     Content-Type: application/json
I1129 01:36:48.097312    6073 round_trippers.go:452]     Date: Sun, 29 Nov 2020 00:36:48 GMT
I1129 01:36:48.105122    6073 request.go:1068] Response Body: {"kind":"Node","apiVersion":"v1","metadata":{"name":"pet-penguin","selfLink":"/api/v1/nodes/pet-penguin/status","uid":"ec42794b-16f5-471e-821f-be2b38724098","resourceVersion":"206249503","creationTimestamp":"2020-11-29T00:35:50Z","labels":{"kubernetes.io/arch":"amd64","kubernetes.io/hostname":"penguin","kubernetes.io/os":"linux","kubernetes.io/role":"pet","lifetime":"persistent","purpose":"pet","type":"virtual-kubelet"},"annotations":{"kubernetes.io/node.class":"kube-pet","node.alpha.kubernetes.io/ttl":"0","virtual-kubelet.io/last-applied-node-status":"{\"capacity\":{\"cpu\":\"2\",\"ephemeral-storage\":\"10Gi\",\"hugepages-2Mi\":\"0\",\"memory\":\"14902712Ki\",\"pods\":\"0\"},\"allocatable\":{\"cpu\":\"1\",\"ephemeral-storage\":\"1Gi\",\"hugepages-2Mi\":\"0\",\"memory\":\"7451356Ki\",\"pods\":\"0\"},\"conditions\":[{\"type\":\"Ready\",\"status\":\"True\",\"lastHeartbeatTime\":\"2020-11-29T00:36:47Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"KubeletReady\",\"message\":\"Hello World\"},{\"type\":\"MemoryPressure\",\"status\":\"False\",\"lastHeartbeatTime\":\"2020-11-29T00:36:47Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"OK\",\"message\":\"Hello World\"},{\"type\":\"DiskPressure\",\"status\":\"False\",\"lastHeartbeatTime\":\"2020-11-29T00:36:47Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"OK\",\"message\":\"Hello World\"},{\"type\":\"PIDPressure\",\"status\":\"False\",\"lastHeartbeatTime\":\"2020-11-29T00:36:47Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"OK\",\"message\":\"Hello World\"},{\"type\":\"NetworkUnavailable\",\"status\":\"False\",\"lastHeartbeatTime\":\"2020-11-29T00:36:47Z\",\"lastTransitionTime\":\"2020-11-29T00:35:47Z\",\"reason\":\"OK\",\"message\":\"Hello World\"}],\"addresses\":[{\"type\":\"Hostname\",\"address\":\"pet-penguin\"},{\"type\":\"InternalIP\",\"address\":\"10.1.1.1\"},{\"type\":\"InternalDNS\",\"address\":\"pet-penguin.local\"},{\"type\":\"ExternalIP\",\"address\":\"8.8.8.8\"},{\"type\":\"ExternalIP\",\"address\":\"2::1\"}],\"daemonEndpoints\":{\"kubeletEndpoint\":{\"Port\":10250}},\"nodeInfo\":{\"machineID\":\"[redacted]\",\"systemUUID\":\"\",\"bootID\":\"dc714fe0-8557-4163-9b25-7bb3e74cea66\",\"kernelVersion\":\"5.4.58-07649-ge120df5deade\",\"osImage\":\"Debian GNU/Linux 10 (buster)\",\"containerRuntimeVersion\":\"podman://2.1.1\",\"kubeletVersion\":\"kube-pet/development\",\"kubeProxyVersion\":\"nftables-pet/development\",\"operatingSystem\":\"linux\",\"architecture\":\"amd64\"}}","virtual-kubelet.io/last-applied-object-meta":"{\"name\":\"pet-penguin\",\"uid\":\"ec42794b-16f5-471e-821f-be2b38724098\",\"creationTimestamp\":null,\"labels\":{\"kubernetes.io/arch\":\"amd64\",\"kubernetes.io/hostname\":\"penguin\",\"kubernetes.io/os\":\"linux\",\"kubernetes.io/role\":\"pet\",\"lifetime\":\"persistent\",\"purpose\":\"pet\",\"type\":\"virtual-kubelet\"},\"annotations\":{\"kubernetes.io/node.class\":\"kube-pet\"}}"}},"spec":{"providerID":"gce://kube-pet-node/bare-metal/pet-penguin","taints":[{"key":"kubernetes.io/pet-node","value":"pet-penguin","effect":"NoSchedule"}]},"status":{"capacity":{"cpu":"2","ephemeral-storage":"10Gi","hugepages-2Mi":"0","memory":"14902712Ki","pods":"0"},"allocatable":{"cpu":"1","ephemeral-storage":"1Gi","hugepages-2Mi":"0","memory":"7451356Ki","pods":"0"},"conditions":[{"type":"Ready","status":"True","lastHeartbeatTime":"2020-11-29T00:36:47Z","lastTransitionTime":"2020-11-29T00:35:47Z","reason":"KubeletReady","message":"Hello World"},{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2020-11-29T00:36:47Z","lastTransitionTime":"2020-11-29T00:35:47Z","reason":"OK","message":"Hello World"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2020-11-29T00:36:47Z","lastTransitionTime":"2020-11-29T00:35:47Z","reason":"OK","message":"Hello World"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2020-11-29T00:36:47Z","lastTransitionTime":"2020-11-29T00:35:47Z","reason":"OK","message":"Hello World"},{"type":"NetworkUnavailable","status":"False","lastHeartbeatTime":"2020-11-29T00:36:47Z","lastTransitionTime":"2020-11-29T00:35:47Z","reason":"OK","message":"Hello World"}],"addresses":[{"type":"Hostname","address":"pet-penguin"},{"type":"InternalIP","address":"10.1.1.1"},{"type":"InternalDNS","address":"pet-penguin.local"},{"type":"ExternalIP","address":"2::1"},{"type":"ExternalIP","address":"1::1"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"b0abca65d2495015e303a74b5eb933ab","systemUUID":"","bootID":"dc714fe0-8557-4163-9b25-7bb3e74cea66","kernelVersion":"5.4.58-07649-ge120df5deade","osImage":"Debian GNU/Linux 10 (buster)","containerRuntimeVersion":"podman://2.1.1","kubeletVersion":"kube-pet/development","kubeProxyVersion":"nftables-pet/development","operatingSystem":"linux","architecture":"amd64"}}}

Of note I think is that the startup / first NodeStatus patch is quite plain about the addresses, which successfully adds all of them:

  "status": {
    "addresses": [
      {
        "address": "pet-penguin",
        "type": "Hostname"
      },
      {
        "address": "10.1.1.1",
        "type": "InternalIP"
      },
      {
        "address": "pet-penguin.local",
        "type": "InternalDNS"
      },
      {
        "address": "8.8.8.8",
        "type": "ExternalIP"
      },
      {
        "address": "1::1",
        "type": "ExternalIP"
      }
    ],

... and then the followup NodeStatus patch (to update the 1::1 to 2::1) switches merge style:

  "status": {
    "$setElementOrder/addresses": [
      {
        "type": "Hostname"
      },
      {
        "type": "InternalIP"
      },
      {
        "type": "InternalDNS"
      },
      {
        "type": "ExternalIP"
      },
      {
        "type": "ExternalIP"
      }
    ],
    "addresses": [
      {
        "address": "2::1",
        "type": "ExternalIP"
      }
    ],

... and from there the address list is corrupted.

My understanding is that actual Kubelet works around this issue by patching the patch, as I linked in the original message, to get a patch more like this:

  "status": {
    "addresses": [
      {
        "address": "pet-penguin",
        "type": "Hostname"
      },
      {
        "address": "10.1.1.1",
        "type": "InternalIP"
      },
      {
        "address": "pet-penguin.local",
        "type": "InternalDNS"
      },
      {
        "address": "8.8.8.8",
        "type": "ExternalIP"
      },
      {
        "address": "2::1",
        "type": "ExternalIP"
      },
      {
        "$patch": "replace"
      }
    ],

danopia avatar Nov 29 '20 01:11 danopia

🤦 We need to implement this apparently: http://pr.k8s.io/79391

sargun avatar Dec 03 '20 22:12 sargun

Why hasn't this been fixed, @sargun? :-p

pires avatar Jan 29 '21 01:01 pires