nodes NotReady after removing default firewall rules

[camaeel]

1. create k3s cluster (3 nodes, medium)
2. add firewall rule: civo firewall rule create FIREWALL_ID -c 'MY_PUBLIC_IP/32' -d ingress -s 6443 -e 6443 -p TCP -l k8s
3. remove all other firewall rules except rule k8s, and default ICMP rule (named: 'Ping/traceroute')
4. restart worker nodes from GUI

Result - worker nodes become NotReady after some time:

$ kubectl --kubeconfig=civo-civo1-kubeconfig get node
NAME               STATUS     ROLES    AGE   VERSION
kube-node-ff61     NotReady   <none>   33m   v1.18.6+k3s1
kube-node-f6ae     NotReady   <none>   33m   v1.18.6+k3s1
kube-master-98ea   Ready      master   34m   v1.18.6+k3s1

I would like to restrict access to the cluster only to my IP.