NodePort services don't work with Cilium via Virtual IP

[reitermarkus]

Bug Report

Description

I deployed Cilium (without kube-proxy) according to the docs. After restarting all my pods for Cilium to take effect I noticed that NodePort services are not reachable anymore via the Talos VIP as they were before.

Am I supposed to deploy Cilium with kube-proxy for this to work? Not quite sure how Virtual IP is implemented and this is something that needs to be handled in Talos or if there is simply some additional configuration required.

Logs

N/A

Environment

• Talos version: [talosctl version --nodes <problematic nodes>]

Client:
	Tag:         v1.1.0
	SHA:         d55a1871
	Built:       
	Go version:  go1.18.3
	OS/Arch:     darwin/amd64
Server:
	NODE:        talos-cp-1
	Tag:         v1.1.0
	SHA:         d55a1871
	Built:       
	Go version:  go1.18.3
	OS/Arch:     linux/amd64
	Enabled:     

• Kubernetes version: [kubectl version --short]

Client Version: v1.24.2
Kustomize Version: v4.5.4
Server Version: v1.24.2

• Platform:

[frezbo]

did you first deploy with flannel and then switched to cilium?

[frezbo]

also vip is mostly for the kubernetes api server, and cilium might just drop traffic if it doesn;t see it coming from a node ip (just a hunch)

[smira]

as @frezbo said, Talos VIP is only for Kubernetes API server. What Talos does is makes sure that one healthy CP node announces the VIP, there is nothing more there.

[reitermarkus]

did you first deploy with flannel and then switched to cilium?

Yes, I switched from Flannel to Cilium.

[frezbo]

did you first deploy with flannel and then switched to cilium?

Yes, I switched from Flannel to Cilium.

If you're live switching CNI's make sure all endpoint objects have updated with the new address, would need a rolling restart if all pods

[reitermarkus]

Yes, I did that, every pod has been restarted.

[smira]

Talos VIP is only for Kubernetes API access, it's not supposed to work with NodePort services.