projectcalico
When eBPF is enabled, when ippool natOutgoing set to false connection to kubernetes api through serviceIP starts timing out
Noticed one different behavior when eBPF is enabled, when ippool natOutgoing set to false connection to kubernetes api through serviceIP starts timing out from pods wanting to use it, since there is no iptables rule sending service ip out to correct node.
How does this work in eBPF mode ?
I am forced to enable outgoing nat to solve this issue, is this a bug or expected behavior?
Originally posted by @ehsan310 in https://github.com/projectcalico/calico/issues/8812#issuecomment-2113624733
@ehsan310 Are your API servers on a different network,not reachable without MASQ or overlay?
since there is no iptables rule sending service ip out to correct node.
Destination IP is changed to whatever the service is translated to and then the packet is sent out. It must be routeable. It could be that kube-proxy does the MASQ on its own based on your cluster configuration, i.e. it knows that the traffic is leaving the pod network, and calico ebpf does not do it because it does not have that configuration - nat outgoing serves that purpose.
service ip is advertised via bgp and calico, there is no overlay network and I have a flat networking peered with ToR. also kube-proxy is disable and DS is removed , all old iptable rules are also removed. so it's only calico handling the traffic. Node to Node Mesh is also disabled.
that also make sense based on what you said , because we have separate mgmt and pod/service traffic network and k8s api is listening on mgmt network , so this is expected.
Sorry for putting this issue on aback burner, did you make any progress with it?
Sorry for putting this issue on aback burner, did you make any progress with it?
I think this is not a problem when using operator since it is possible to conifugure kube-api endpoints but on manifest installation it is not possible and with a installation where you have 2 network card on the cluster then this might be a problem since without nat enabled cluster can't communicate out to kube-api that translate to host network until BGP is established which is not available in during the initialization.
I think this is not a problem when using operator since it is possible to conifugure kube-api endpoints but on manifest installation it is not possible
Do you mean that kubernetes-services-endpoint config map? You can do that for manifest as well.
installation where you have 2 network card on the cluster then this might be a problem since without nat enabled cluster can't communicate out to kube-api
Do you mean SNAT? How did that work with iptables?
I think this is not a problem when using operator since it is possible to conifugure kube-api endpoints but on manifest installation it is not possible
Do you mean that
kubernetes-services-endpointconfig map? You can do that for manifest as well.installation where you have 2 network card on the cluster then this might be a problem since without nat enabled cluster can't communicate out to kube-api
Do you mean SNAT? How did that work with iptables?
oh really ? I thought that CM has only been used in tigera-operator pod !
the issue was when deploying calico via kubespray (which uses manifest) and disable kube-proxy then pods are unable to communicate to kube-api endpoint since there is no kube-proxy in place (no snat, or any iptables rule) then until calico-node pod starts nothing will work and the issue was calico-api-server and calico-kube-controllers were trying to connect to kube-api but was not able to reach it. in operator kubernetes-services-endpoint i pointed to master nic ip and port 6443 directly to make the connection. does that make sence ?
oh really ? I thought that CM has only been used in tigera-operator pod !
you can also put it straight into the manifest as env vars. That was the original way of doing it. Otherwise, as you are describing, our ebpf kube-proxy replacement would not be able to reach api server and implement the services. Bit of an chicken-egg problem. We are going to make it more automated with operator.
@ehsan310 I am closing this, if you still have issues, feel free to reopen.