Worker never starts

[rhjensen79]

/kind bug

What steps did you take and what happened: When i apply the cappx-test.yaml then 2 vm's get created and resized. The master starts, and seam to run all the way thru. The worker, never starts, so the deployment never finishes.

When i do a describe of the cluster is says provisioned. So i'm guessing it thinks it ok.

What did you expect to happen: I expect a full cluster, with 1 master and 1 worker, being provisioned.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

I'm using latest 0.3.5

• Cluster-api-provider-proxmox version:
  clusterctl version: &version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.1", GitCommit:"ef04465b2ba76214eea570e27e8146c96412e32a", GitTreeState:"clean", BuildDate:"2024-04-23T17:05:53Z", GoVersion:"go1.21.9", Compiler:"gc", Platform:"darwin/arm64"}

• Proxmox VE version: pve-manager/8.2.2/9355359cd7afbae4

• Kubernetes version: (use kubectl version): Client Version: v1.30.0 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.2

Logs From cappx-controller-manager

I0426 09:08:46.032786       1 qemu.go:21] "Reconciling QEMU" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-wj9zl" namespace="default" name="cappx-test-wj9zl" reconcileID=d07991f6-3318-4eb0-b6e7-e689bee83fe8
I0426 09:08:46.032794       1 qemu.go:58] "getting qemu from vmid" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-wj9zl" namespace="default" name="cappx-test-wj9zl" reconcileID=d07991f6-3318-4eb0-b6e7-e689bee83fe8
I0426 09:08:46.032883       1 scheduler.go:173] "Start Running Scheduler" Name="qemu-scheduler"
I0426 09:08:46.032912       1 scheduler.go:196] "getting next qemu from scheduling queue" Name="qemu-scheduler"
E0426 09:08:49.143170       1 qemu.go:26] "failed to get qemu" err="401 - 401 permission denied - invalid PVE ticket - " controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-wj9zl" namespace="default" name="cappx-test-wj9zl" reconcileID=d07991f6-3318-4eb0-b6e7-e689bee83fe8
E0426 09:08:49.143232       1 reconcile.go:27] "failed to create/get instance" err="401 - 401 permission denied - invalid PVE ticket - " controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-wj9zl" namespace="default" name="cappx-test-wj9zl" reconcileID=d07991f6-3318-4eb0-b6e7-e689bee83fe8
E0426 09:08:49.143251       1 proxmoxmachine_controller.go:160] "Reconcile error" err="401 - 401 permission denied - invalid PVE ticket - " controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-wj9zl" namespace="default" name="cappx-test-wj9zl" reconcileID=d07991f6-3318-4eb0-b6e7-e689bee83fe8
E0426 09:08:49.144324       1 controller.go:324] "Reconciler error" err="401 - 401 permission denied - invalid PVE ticket - " controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-wj9zl" namespace="default" name="cappx-test-wj9zl" reconcileID=d07991f6-3318-4eb0-b6e7-e689bee83fe8
I0426 09:09:36.364952       1 scheduler.go:175] "Stop Running Scheduler" Name="qemu-scheduler"
I0426 09:09:46.032184       1 scheduler.go:175] "Stop Running Scheduler" Name="qemu-scheduler"
I0426 09:14:07.077886       1 proxmoxmachine_controller.go:144] "Reconciling ProxmoxMachine" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-md-0-fwh9p-49pl4" namespace="default" name="cappx-test-md-0-fwh9p-49pl4" reconcileID=3bc683b1-d9c9-4036-801f-4d9e79a5d3c3
I0426 09:14:10.107725       1 reconcile.go:24] "Reconciling instance" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-md-0-fwh9p-49pl4" namespace="default" name="cappx-test-md-0-fwh9p-49pl4" reconcileID=3bc683b1-d9c9-4036-801f-4d9e79a5d3c3
I0426 09:14:10.108082       1 reconcile.go:105] "instance does not have providerID yet" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-md-0-fwh9p-49pl4" namespace="default" name="cappx-test-md-0-fwh9p-49pl4" reconcileID=3bc683b1-d9c9-4036-801f-4d9e79a5d3c3
I0426 09:14:10.108105       1 reconcile.go:89] "instance wasn't found. new instance will be created" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-md-0-fwh9p-49pl4" namespace="default" name="cappx-test-md-0-fwh9p-49pl4" reconcileID=3bc683b1-d9c9-4036-801f-4d9e79a5d3c3
I0426 09:14:10.108147       1 qemu.go:21] "Reconciling QEMU" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-md-0-fwh9p-49pl4" namespace="default" name="cappx-test-md-0-fwh9p-49pl4" reconcileID=3bc683b1-d9c9-4036-801f-4d9e79a5d3c3
I0426 09:14:10.108184       1 qemu.go:58] "getting qemu from vmid" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-md-0-fwh9p-49pl4" namespace="default" name="cappx-test-md-0-fwh9p-49pl4" reconcileID=3bc683b1-d9c9-4036-801f-4d9e79a5d3c3
I0426 09:14:10.108240       1 scheduler.go:173] "Start Running Scheduler" Name="qemu-scheduler"
I0426 09:14:10.108362       1 scheduler.go:196] "getting next qemu from scheduling queue" Name="qemu-scheduler"
E0426 09:14:13.132285       1 qemu.go:26] "failed to get qemu" err="401 - 401 permission denied - invalid PVE ticket - " controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-md-0-fwh9p-49pl4" namespace="default" name="cappx-test-md-0-fwh9p-49pl4" reconcileID=3bc683b1-d9c9-4036-801f-4d9e79a5d3c3
E0426 09:14:13.132332       1 reconcile.go:27] "failed to create/get instance" err="401 - 401 permission denied - invalid PVE ticket - " controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-md-0-fwh9p-49pl4" namespace="default" name="cappx-test-md-0-fwh9p-49pl4" reconcileID=3bc683b1-d9c9-4036-801f-4d9e79a5d3c3
E0426 09:14:13.132348       1 proxmoxmachine_controller.go:160] "Reconcile error" err="401 - 401 permission denied - invalid PVE ticket - " controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-md-0-fwh9p-49pl4" namespace="default" name="cappx-test-md-0-fwh9p-49pl4" reconcileID=3bc683b1-d9c9-4036-801f-4d9e79a5d3c3
E0426 09:14:13.134523       1 controller.go:324] "Reconciler error" err="401 - 401 permission denied - invalid PVE ticket - " controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-md-0-fwh9p-49pl4" namespace="default" name="cappx-test-md-0-fwh9p-49pl4" reconcileID=3bc683b1-d9c9-4036-801f-4d9e79a5d3c3
I0426 09:14:16.832442       1 proxmoxmachine_controller.go:144] "Reconciling ProxmoxMachine" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-wj9zl" namespace="default" name="cappx-test-wj9zl" reconcileID=8022b2d7-4f4e-41bc-9085-33c4b233bcd3
I0426 09:14:19.891301       1 reconcile.go:24] "Reconciling instance" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-wj9zl" namespace="default" name="cappx-test-wj9zl" reconcileID=8022b2d7-4f4e-41bc-9085-33c4b233bcd3
I0426 09:14:19.891383       1 reconcile.go:105] "instance does not have providerID yet" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-wj9zl" namespace="default" name="cappx-test-wj9zl" reconcileID=8022b2d7-4f4e-41bc-9085-33c4b233bcd3
I0426 09:14:19.891412       1 reconcile.go:89] "instance wasn't found. new instance will be created" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-wj9zl" namespace="default" name="cappx-test-wj9zl" reconcileID=8022b2d7-4f4e-41bc-9085-33c4b233bcd3
I0426 09:14:19.891435       1 qemu.go:21] "Reconciling QEMU" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-wj9zl" namespace="default" name="cappx-test-wj9zl" reconcileID=8022b2d7-4f4e-41bc-9085-33c4b233bcd3
I0426 09:14:19.891470       1 qemu.go:58] "getting qemu from vmid" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-wj9zl" namespace="default" name="cappx-test-wj9zl" reconcileID=8022b2d7-4f4e-41bc-9085-33c4b233bcd3
I0426 09:14:19.891862       1 scheduler.go:173] "Start Running Scheduler" Name="qemu-scheduler"
I0426 09:14:19.891984       1 scheduler.go:196] "getting next qemu from scheduling queue" Name="qemu-scheduler"
E0426 09:14:22.915171       1 qemu.go:26] "failed to get qemu" err="401 - 401 permission denied - invalid PVE ticket - " controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-wj9zl" namespace="default" name="cappx-test-wj9zl" reconcileID=8022b2d7-4f4e-41bc-9085-33c4b233bcd3
E0426 09:14:22.915244       1 reconcile.go:27] "failed to create/get instance" err="401 - 401 permission denied - invalid PVE ticket - " controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-wj9zl" namespace="default" name="cappx-test-wj9zl" reconcileID=8022b2d7-4f4e-41bc-9085-33c4b233bcd3
E0426 09:14:22.915348       1 proxmoxmachine_controller.go:160] "Reconcile error" err="401 - 401 permission denied - invalid PVE ticket - " controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-wj9zl" namespace="default" name="cappx-test-wj9zl" reconcileID=8022b2d7-4f4e-41bc-9085-33c4b233bcd3
E0426 09:14:22.916430       1 controller.go:324] "Reconciler error" err="401 - 401 permission denied - invalid PVE ticket - " controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-wj9zl" namespace="default" name="cappx-test-wj9zl" reconcileID=8022b2d7-4f4e-41bc-9085-33c4b233bcd3
I0426 09:15:10.107757       1 scheduler.go:175] "Stop Running Scheduler" Name="qemu-scheduler"
I0426 09:15:19.894443       1 scheduler.go:175] "Stop Running Scheduler" Name="qemu-scheduler"
I0426 09:25:08.500440       1 proxmoxmachine_controller.go:144] "Reconciling ProxmoxMachine" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-md-0-fwh9p-49pl4" namespace="default" name="cappx-test-md-0-fwh9p-49pl4" reconcileID=0a4c7924-da5e-4521-9275-837c686c795a
I0426 09:25:11.531027       1 reconcile.go:24] "Reconciling instance" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-md-0-fwh9p-49pl4" namespace="default" name="cappx-test-md-0-fwh9p-49pl4" reconcileID=0a4c7924-da5e-4521-9275-837c686c795a
I0426 09:25:11.531104       1 reconcile.go:105] "instance does not have providerID yet" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-md-0-fwh9p-49pl4" namespace="default" name="cappx-test-md-0-fwh9p-49pl4" reconcileID=0a4c7924-da5e-4521-9275-837c686c795a
I0426 09:25:11.531130       1 reconcile.go:89] "instance wasn't found. new instance will be created" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-md-0-fwh9p-49pl4" namespace="default" name="cappx-test-md-0-fwh9p-49pl4" reconcileID=0a4c7924-da5e-4521-9275-837c686c795a
I0426 09:25:11.531167       1 qemu.go:21] "Reconciling QEMU" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-md-0-fwh9p-49pl4" namespace="default" name="cappx-test-md-0-fwh9p-49pl4" reconcileID=0a4c7924-da5e-4521-9275-837c686c795a
I0426 09:25:11.531230       1 qemu.go:58] "getting qemu from vmid" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/cappx-test-md-0-fwh9p-49pl4" namespace="default" name="cappx-test-md-0-fwh9p-49pl4" reconcileID=0a4c7924-da5e-4521-9275-837c686c795a
I0426 09:25:11.531454       1 scheduler.go:173] "Start Running Scheduler" Name="qemu-scheduler"
I0426 09:25:11.531528       1 scheduler.go:196] "getting next qemu from scheduling queue" Name="qemu-scheduler"