hcloud-cloud-controller-manager icon indicating copy to clipboard operation
hcloud-cloud-controller-manager copied to clipboard

Published 20 hours ago • verified publisher icon hetznercloud

Hetzner Cloud Control manager not connecting with Hetzner

Open abishekas opened this issue 3 weeks ago • 5 comments

TL;DR

Hi Everyone,

We were planning to move our production environment to the Hetzner cloud. So we provisioned a Kubernetes cluster (self-managed) setup in Hetzner servers for our project and for making connection establishment between Hetzner and our Kubernetes cluster we used the Hetzner cloud controller manager. By following the below document, we provisioned it.

https://community.hetzner.com/tutorials/install-kubernetes-cluster#:~:text=Now%20deploy%20the%20Hetzner%20Cloud%20controller%20manager%20into%20the%20cluster

Expected behavior

We deployed this during March 2024 and everything was working as expected till yesterday, but today when we create a new server in the Hcloud console and add it to the same cluster, the hcloud providerid and the region topology labels are not added for that server and we are utilizing the nginx ingress as Loadbalancer for this setup. when we apply the ingress-nginx it will automatically connect with the load balancer in the cloud but from today that connection is also not working.

Observed behavior

We tried to resolve this with logs from the Hetzner cloud controller manager but we couldn't see any errors in the logs. I'm sharing the log data below for reference. We also tried provisioning a new setup to see if that works, but we received the same issue. We verified the network connectivity to Hetzner Cloud from our server through API calls, and through PING requests, it works fine. We even created a new setup with another region, but the issue still persists.

We have planned our production migration for this weekend, so any quick help would be greatly appreciated. Thanks.

Minimal working example

No response

Log output

I0612 17:13:22.834269       1 route_controller.go:216] action for Node "postgresql-testing" with CIDR "10.244.2.0/24": "keep"
I0612 17:13:22.834282       1 route_controller.go:216] action for Node "hcloud-owrker" with CIDR "10.244.4.0/24": "keep"
I0612 17:13:52.838954       1 route_controller.go:216] action for Node "master" with CIDR "10.244.0.0/24": "keep"
I0612 17:13:52.839004       1 route_controller.go:216] action for Node "postgresql-testing" with CIDR "10.244.2.0/24": "keep"
I0612 17:13:52.839018       1 route_controller.go:216] action for Node "hcloud-owrker" with CIDR "10.244.4.0/24": "keep"
I0612 17:13:52.839030       1 route_controller.go:216] action for Node "jenkins-server" with CIDR "10.244.1.0/24": "keep"
I0612 17:13:56.277820       1 load_balancers.go:137] "ensure Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" service="ingress-nginx-controller" nodes=["jenkins-server","postgresql-testing","hcloud-owrker"]
I0612 17:13:56.277968       1 event.go:307] "Event occurred" object="ingress-nginx/ingress-nginx-controller" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
I0612 17:13:56.777461       1 load_balancer.go:820] "update service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=80 loadBalancerID=1798225
I0612 17:13:57.567504       1 load_balancer.go:820] "update service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=443 loadBalancerID=1798225
E0612 17:13:58.576626       1 controller.go:298] error processing service ingress-nginx/ingress-nginx-controller (retrying with exponential backoff): failed to ensure load balancer: hcloud/loadBalancers.EnsureLoadBalancer: hcops/LoadBalancerOps.ReconcileHCLBTargets: providerID does not have one of the the expected prefixes (hcloud://, hrobot://, hcloud://bm-): 
I0612 17:13:58.576714       1 event.go:307] "Event occurred" object="ingress-nginx/ingress-nginx-controller" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: hcloud/loadBalancers.EnsureLoadBalancer: hcops/LoadBalancerOps.ReconcileHCLBTargets: providerID does not have one of the the expected prefixes (hcloud://, hrobot://, hcloud://bm-): "
I0612 17:14:22.848432       1 route_controller.go:216] action for Node "master" with CIDR "10.244.0.0/24": "keep"
I0612 17:14:22.848502       1 route_controller.go:216] action for Node "postgresql-testing" with CIDR "10.244.2.0/24": "keep"
I0612 17:14:22.848524       1 route_controller.go:216] action for Node "hcloud-owrker" with CIDR "10.244.4.0/24": "keep"
I0612 17:14:22.848543       1 route_controller.go:216] action for Node "jenkins-server" with CIDR "10.244.1.0/24": "keep"
I0612 17:14:52.888746       1 route_controller.go:216] action for Node "jenkins-server" with CIDR "10.244.1.0/24": "keep"
I0612 17:14:52.888807       1 route_controller.go:216] action for Node "master" with CIDR "10.244.0.0/24": "keep"
I0612 17:14:52.888831       1 route_controller.go:216] action for Node "postgresql-testing" with CIDR "10.244.2.0/24": "keep"
I0612 17:14:52.888856       1 route_controller.go:216] action for Node "hcloud-owrker" with CIDR "10.244.4.0/24": "keep"
I0612 17:15:22.824760       1 route_controller.go:216] action for Node "hcloud-owrker" with CIDR "10.244.4.0/24": "keep"
I0612 17:15:22.824815       1 route_controller.go:216] action for Node "jenkins-server" with CIDR "10.244.1.0/24": "keep"
I0612 17:15:22.824830       1 route_controller.go:216] action for Node "master" with CIDR "10.244.0.0/24": "keep"
I0612 17:15:22.824843       1 route_controller.go:216] action for Node "postgresql-testing" with CIDR "10.244.2.0/24": "keep"
I0612 17:15:52.824370       1 route_controller.go:216] action for Node "master" with CIDR "10.244.0.0/24": "keep"
I0612 17:15:52.824415       1 route_controller.go:216] action for Node "postgresql-testing" with CIDR "10.244.2.0/24": "keep"
I0612 17:15:52.824433       1 route_controller.go:216] action for Node "hcloud-owrker" with CIDR "10.244.4.0/24": "keep"
I0612 17:15:52.824448       1 route_controller.go:216] action for Node "jenkins-server" with CIDR "10.244.1.0/24": "keep"
I0612 17:16:22.965547       1 route_controller.go:216] action for Node "jenkins-server" with CIDR "10.244.1.0/24": "keep"
I0612 17:16:22.965596       1 route_controller.go:216] action for Node "master" with CIDR "10.244.0.0/24": "keep"
I0612 17:16:22.965618       1 route_controller.go:216] action for Node "postgresql-testing" with CIDR "10.244.2.0/24": "keep"
I0612 17:16:22.965637       1 route_controller.go:216] action for Node "hcloud-owrker" with CIDR "10.244.4.0/24": "keep"
I0612 17:16:52.946044       1 route_controller.go:216] action for Node "hcloud-owrker" with CIDR "10.244.4.0/24": "keep"
I0612 17:16:52.946077       1 route_controller.go:216] action for Node "jenkins-server" with CIDR "10.244.1.0/24": "keep"
I0612 17:16:52.946092       1 route_controller.go:216] action for Node "master" with CIDR "10.244.0.0/24": "keep"
I0612 17:16:52.946105       1 route_controller.go:216] action for Node "postgresql-testing" with CIDR "10.244.2.0/24": "keep"
I0612 17:17:22.863320       1 route_controller.go:216] action for Node "hcloud-owrker" with CIDR "10.244.4.0/24": "keep"
I0612 17:17:22.863365       1 route_controller.go:216] action for Node "jenkins-server" with CIDR "10.244.1.0/24": "keep"
I0612 17:17:22.863382       1 route_controller.go:216] action for Node "master" with CIDR "10.244.0.0/24": "keep"
I0612 17:17:22.863402       1 route_controller.go:216] action for Node "postgresql-testing" with CIDR "10.244.2.0/24": "keep"

Additional information

No response

abishekas avatar Jun 12 '24 17:06 abishekas