kubernetes-the-hard-way icon indicating copy to clipboard operation
kubernetes-the-hard-way copied to clipboard

Error from server: error dialing backend: dial tcp: lookup worker-1 on 127.0.0.53:53: server misbehaving

Open AlexZalyalowa opened this issue 4 years ago • 5 comments

Hello. I'm stuck at "Deploying the DNS Cluster Add-on" lesson. When I'm trying to execute a DNS lookup, I've got the error

Error from server: error dialing backend: dial tcp: lookup worker-1 on 127.0.0.53:53: server misbehaving

I've set up my cluster on hetzner cloud, not in google cloud, may be it's the reason of this issue, Routing between working is correct, I can ping worker-1's pods from worker-0.

AlexZalyalowa avatar Jan 12 '21 16:01 AlexZalyalowa

@AlexZalyalowa did you solve that?

khlystov avatar Jan 21 '21 17:01 khlystov

@AlexZalyalowa did you solve that?

Nope.

AlexZalyalowa avatar Jan 21 '21 18:01 AlexZalyalowa

your problem is as follows: your node hostnames need to be resolvable. maybe put all the master/worker nodes and their IPs into /etc/hosts on all nodes? I fixed this issue by pointing the nameserver in /etc/resolv.conf to my dns server and setting up entries of all my worker/master nodes in that dns server. Edit: that's the root cause. Solution can be specific to your cloud provider, I posted my solution for this problem on my local network.

thrubovc avatar Mar 07 '21 11:03 thrubovc

Running the following command on all controller nodes made it work for me:

cat <<EOF | sudo tee -a hosts
10.240.0.20 worker-0
10.240.0.21 worker-1
10.240.0.22 worker-2
EOF

baurmatt avatar May 07 '21 12:05 baurmatt

Add this to kube-apiserver arguments (in the systemd service): --kubelet-preferred-address-types=InternalIP,ExternalIP. Default is Hostname,InternalIP,ExternalIP. So with this setting, it will try to talk to kubelets using the InternalIP first instead.

PhilippeChepy avatar Dec 12 '21 18:12 PhilippeChepy