apiserver-network-proxy

apiserver-network-proxy copied to clipboard

We have resources spanning 2 subnets A. a production subnet that has public IPs B. an experimental subnet that is not publicly addressable, and that sits behind a NAT

We have deployed a k8s cluster with its master node on A - following the guidance found in https://kubernetes.io/docs/tasks/extend-kubernetes/setup-konnectivity/

With our current setup it is possible for the nodes in B to join the cluster also we can spawn and delete pods and containers on the B side as well

However there are at least 2 functions that are problematic:

• kubectl logs pod-on-the-B-side if that pod runs on the B side (from either A or B)
• kubectl exec -ti pod-on-the-B-side -- /bin/bash if that pod runs on the B side (from either A or B)

in both cases after a timeout we get a message like this one

kubectl exec -ti the-pod -- /bin/bash
Error from server: error dialing backend: dial tcp 192.168.3.1:10250: i/o timeout

is this the expected behaviour, or else what can we have done wrong ?

I should have mentioned that 192.168.3.x is, of course, the NAT'ed private address on the the B side

Error from server: error dialing backend: dial tcp 192.168.3.1:10250: i/o timeout

From this error, it seems like the konnectivity agent on B is not able to connect to any of the nodes on B. Kubernetes' log functionality goes through kubelet which is 192.168.3.1:10250. Can you confirm that networking works between the pod and the node by running ping?

I have investigated this further, and found and fixed a couple glitches in our setup along the way (missing rbac configuration was preventing proper konnectivity deployment; port 8132 was being firewalled from B to A, keeping the agent from reaching the A side)

at this point what I observe is this

• I rebuilt the cluster from scratch, with one master node on the A side, and one worker node on the B side
• I can check the konnectivity agent gets actually deployed on the B node (it was not making it there yesterday)
• when I try to spawn one user pod (called kiada-fit01) on the B node, it gets stuck at the ContainerCreating stage
• and kubectl describe on this user pod reports this message

    Warning  FailedCreatePodSandBox  9m22s                
    kubelet  Failed to create pod sandbox: 
    rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kiada-blablabla): 
    error adding pod kube-system_kiada-fit01-pod to CNI network "cbr0": 
    failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 10.244.1.1/24
  

• another odd observation is that the IP address that the konnectivity-agent pod receives is in 10.85.0.2 which is totally out of the configured pod-cidr for which I have the usual podSubnet: 10.244.0.0/16
• on the B node, the cni0 interface uses an address in this range too

  ip add sh cni0 | grep cni0
  6: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
      inet 10.85.0.1/16 brd 10.85.255.255 scope global cni0
  

---

I have replayed all this a few times today already, and am not getting consistent results, but this error failed to set bridge addr: "cni0" already has an IP address different from... is happening a lot

I could use some help on possible angles to troubleshoot this… thanks in advance !

PS

all the nodes involved are set up using fedora35+CRI-O as the container runtime I have found mentions of the 10.85.0.0 block in the CRI-O repo here https://github.com/cri-o/cri-o/blob/main/contrib/cni/10-crio-bridge.conf

failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 10.244.1.1/24

This seems like an issue with your CNI setup. What CNI are you using?

this is with flannel

Is it possible you have two CNIs configured in your cluster by accident?

mmh, I guess if that was the case I would see the corresponding pods in kubectl get pods -A but here's what I have

k get pods -A
NAMESPACE     NAME                                          READY   STATUS        RESTARTS   AGE
kube-system   coredns-78fcd69978-jzths                      1/1     Running       0          60m
kube-system   coredns-78fcd69978-z96d4                      1/1     Running       0          60m
kube-system   etcd-sopnode-l1.inria.fr                      1/1     Running       15         60m
kube-system   kiada-fit01-pod                               0/1     Terminating   0          57m
kube-system   konnectivity-agent-fv4gw                      1/1     Running       0          59m
kube-system   konnectivity-server-sopnode-l1.inria.fr       1/1     Running       13         60m
kube-system   kube-apiserver-sopnode-l1.inria.fr            1/1     Running       12         60m
kube-system   kube-controller-manager-sopnode-l1.inria.fr   1/1     Running       15         60m
kube-system   kube-flannel-ds-gnzqz                         1/1     Running       0          60m
kube-system   kube-flannel-ds-xwnkk                         1/1     Running       0          60m
kube-system   kube-proxy-7zvj5                              1/1     Running       0          60m
kube-system   kube-proxy-jsz4l                              1/1     Running       0          60m
kube-system   kube-scheduler-sopnode-l1.inria.fr            1/1     Running       15         60m

so I would tend to answer no

So konnectivity doesn't actually overlap with CNI in anyway. So if you see FailedCreatePodSandBox errors on the kiada-fit01-pod pod, it seems unrelated to the network proxy and something to do with the CNI configuration of the cluster.

It seems like you do have pods that are working well though, is it possible that some nodes are misconfigured but others are not? Is kiada-fit01-pod on a different node than the other pods?

I figured out a piece of the puzzle what was happening yesterday was that the node on the B side was running a fedora image where /etc/cni/net.d/ was pre-populated with stuff from the crio rpm

# ls -l /etc/cni/net.d/
total 8
-rw-r--r--. 1 root root 438 Mar 15 15:11 100-crio-bridge.conf
-rw-r--r--. 1 root root  54 Mar 15 15:11 200-loopback.conf
# rpm -qf /etc/cni/net.d/*
cri-o-1.22.3-1.module_f35+14123+b0173ecb.x86_64
cri-o-1.22.3-1.module_f35+14123+b0173ecb.x86_64

and the 10.85.x.x thingy that I was seeing was coming from 100-crio-bridge.conf removing these files before joining the cluster helped sanitizing things further

---

I'm still experiencing the initial problem though, even if in a slightly different flavour

# kubectl logs pod-name
Error from server: Get "https://192.168.3.2:10250/containerLogs/kube-system/kiada-fit02-pod/kiada-fit02-cont": dial tcp 192.168.3.2:10250: i/o timeout

@andrewsykim in your initial answer you were saying

Can you confirm that networking works between the pod and the node by running ping?

not quite sure what you meant there ? pod meaning the konnectivity-agent pod on the B side, right ? and the node I take it would be the NAT'ed address 192.168.3.x ? also it looks like the konnectivity-agent image has a very low footprint and does not have /bin/bash nor /bin/ping, so how can I run this test ?

as you can see I am relatively new to kubernetes :-)

thanks for your insight in any case

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.