Capmox 0.5.0 unable to access proxmox-credentials secret

[mkamsikad2]

trafficstars

What steps did you take and what happened: I upgraded from Capmox 0.4.0 to 0.5.0

clusterctl upgrade apply --core cluster-api:v1.7.2 --bootstrap kubeadm:v1.7.2 --control-plane kubeadm:v1.7.2 --ipam in-cluster:v0.1.0 --infrastructure proxmox:v0.5.0

Following the upgrade the Capmox controller logs the following errors:

E0619 13:51:44.068023 1 controller.go:329] "Reconciler error" err="error getting infra provider cluster or control plane object: No credentials found, ProxmoxCluster missing credentialsRef" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/capi-management-v2-control-plane-qjwxf" namespace="default" name="capi-management-v2-control-plane-qjwxf" reconcileID="2ae9413d-cb8e-4425-9860-e870d84adb5a"

I then added created the required secret and added the secret in ProxmoxCluster.spec.credentialsRef.name

apiVersion: v1
stringData:
  secret: ${PROXMOX_SECRET}
  token: ${PROXMOX_TOKEN}
  url: ${PROXMOX_URL}
kind: Secret
metadata:
  name: "${CLUSTER_NAME}-proxmox-credentials"
  labels:
    platform.ionos.com/secret-type: "proxmox-credentials"

The capmox provider cannot then find the secret and is unable to communicate with proxmox:

E0619 13:51:43.567318       1 controller.go:329] "Reconciler error" err="error getting infra provider cluster or control plane object: Unable to initialize ProxmoxClient: failed to get credentials secret: Secret \"mk1-busi-cl-proxmox-credentials\" not found" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/mk1-busi-cl-worker-pzvch" namespace="default" name="mk1-busi-cl-worker-pzvch" reconcileID="de7a6641-3c01-443a-8c5b-02f2cd62797a"
	failed to create scope: Secret "mk1-busi-cl-proxmox-credentials" not found
[14:02:29] k8sadmin@image-k get secret mk1-busi-cl-proxmox-credentials
NAME                              TYPE     DATA   AGE
mk1-busi-cl-proxmox-credentials   Opaque   3      26m

E0619 13:57:18.765794       1 controller.go:329] "Reconciler error" err=<
	failed to create scope: Secret "mk1-busi-cl-proxmox-credentials" not found
	failed to get credentials secret
	github.com/ionos-cloud/cluster-api-provider-proxmox/pkg/scope.(*ClusterScope).setupProxmoxClient
		/workspace/pkg/scope/cluster.go:142
	github.com/ionos-cloud/cluster-api-provider-proxmox/pkg/scope.NewClusterScope
		/workspace/pkg/scope/cluster.go:119
	github.com/ionos-cloud/cluster-api-provider-proxmox/internal/controller.(*ProxmoxClusterReconciler).Reconcile
		/workspace/internal/controller/proxmoxcluster_controller.go:107
	sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
		/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119
	sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
		/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316
	sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
		/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
	sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
		/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
	runtime.goexit
		/usr/local/go/src/runtime/asm_amd64.s:1650
	Unable to initialize ProxmoxClient
	github.com/ionos-cloud/cluster-api-provider-proxmox/pkg/scope.NewClusterScope
		/workspace/pkg/scope/cluster.go:121
	github.com/ionos-cloud/cluster-api-provider-proxmox/internal/controller.(*ProxmoxClusterReconciler).Reconcile
		/workspace/internal/controller/proxmoxcluster_controller.go:107
	sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
		/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119
	sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
		/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316
	sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
		/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
	sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
		/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
	runtime.goexit
		/usr/local/go/src/runtime/asm_amd64.s:1650
 > controller="proxmoxcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxCluster" ProxmoxCluster="default/mk1-busi-cl" namespace="default" name="mk1-busi-cl" reconcileID="19414580-a689-45ae-8798-42eb43157644"
[14:07:42] k8sadmin@image-k get secret mk1-busi-cl-proxmox-credentials
NAME                              TYPE     DATA   AGE
mk1-busi-cl-proxmox-credentials   Opaque   3      31m

What did you expect to happen: I would have expected to upgrade capmox and then have to apply the required secret and add the required value into ProxmoxCluster.spec.credentialsRef.name I then would expect the capmox controller to communicate with proxmox.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api-provider-proxmox version: 0.5.0
• Kubernetes version: (use kubectl version): 1.28.8
• OS (e.g. from /etc/os-release): ubuntu 24.04

[mcbenjemaa]

@mkamsikad2 thanks for reporting this, While I tested this before I will go and give it another try to check if upgrading broke the ProxmoxClient.

[mkamsikad2]

@mcbenjemaa I have upgraded to Capmox 0.5.1 and have now configured ProxmoxCluster.spec.credentialsRef.namespace

I can now build clusters and perform alpha rollouts. One thing that is still an issue is that all upgraded clusters are in a Failed state.

cluster.cluster.x-k8s.io/capi-management-v2                  Failed        66d     
cluster.cluster.x-k8s.io/dev1-uk-busi-cl                     Failed        66d     
cluster.cluster.x-k8s.io/mk1-busi-cl                         Provisioned   4m32s   
cluster.cluster.x-k8s.io/test1-uk-busi-cl                    Failed        60d     

NAME                                                                CLUSTER              READY   ENDPOINT
proxmoxcluster.infrastructure.cluster.x-k8s.io/capi-management-v2   capi-management-v2   true    {"host":"10.20.1.220","port":6443}
proxmoxcluster.infrastructure.cluster.x-k8s.io/dev1-uk-busi-cl      dev1-uk-busi-cl      true    {"host":"10.10.1.10","port":6443}
proxmoxcluster.infrastructure.cluster.x-k8s.io/mk1-busi-cl          mk1-busi-cl          true    {"host":"10.10.1.150","port":6443}
proxmoxcluster.infrastructure.cluster.x-k8s.io/test1-uk-busi-cl     test1-uk-busi-cl     true    {"host":"10.10.1.30","port":6443}

I'm not sure how to bring the cluster out of a failed state.

[mcbenjemaa]

i will check this soon

[cfredericksen]

@mcbenjemaa I am also having this same issue. Thanks @mkamsikad2 for posting.

[mcbenjemaa]

I couldn't reproduce this, I upgraded from v0.4 to v0.5.1 and after that, i triggered an upgrade.

[mkamsikad2]

@mcbenjemaa It issue was cause by me upgrading from 0.4 to 0.5.0. Upgrading from 0.4 to 0.5.1 works. The only outstanding issue is the clusters which are in a failed state. There is no issue with those clusters and they function ok. Unsure how to reconcile the cluster state. Its not a major issue its just cosmetic

[mcbenjemaa]

for me, the clusters were not in failed state, i don't know what happened in your setup

[mcbenjemaa]

@mkamsikad2 this fix will be released in v0.6