Reprovisioning node with rancher fails with "Error creating machine: Error in driver during machine creation: Multiple resources with that name."

[meier-rene]

Hi, I'm using this plugin with rancher to create a kubernetes cluster. The creation of a new cluster works without a problem. But if I delete a node from a running cluster, rancher will redeploy a new one with the same name. I suspect that there is somewhere a race condition issue. The old node is not finally deletete in opennebula. Its in SHUTDOWN or EPILOG state and the new node is already created. Below you find the log from rancher. The failing node is called etcd2. The final error is: Error creating machine: Error in driver during machine creation: Multiple resources with that name.

2019/10/16 08:06:44 [INFO] Removing node etcd2 done 2019/10/16 08:06:44 [ERROR] NodeController c-4v52t/m-pbs69 [node-controller] failed with : nodes.management.cattle.io "m-pbs69" not found 2019/10/16 08:06:45 [INFO] Creating jail for c-4v52t 2019/10/16 08:06:45 [INFO] Provisioning node etcd2 2019/10/16 08:06:45 [INFO] [node-controller-docker-machine] Creating CA: /management-state/node/nodes/etcd2/certs/ca.pem 2019/10/16 08:06:46 [INFO] [node-controller-docker-machine] Creating client certificate: /management-state/node/nodes/etcd2/certs/cert.pem 2019/10/16 08:06:47 [INFO] [node-controller-docker-machine] Running pre-create checks... 2019/10/16 08:06:47 [INFO] [node-controller-docker-machine] Creating machine... 2019/10/16 08:06:47 [INFO] [node-controller-docker-machine] (etcd2) Creating SSH key... 2019/10/16 08:06:47 [ERROR] NodeController etcd2 recovered from panic "runtime error: invalid memory address or nil pointer dereference". (err=) Call stack: goroutine 4027 [running]: github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/runtime.RecoverFromPanic(0xc009b4bdb0) /go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:158 +0xb5 panic(0x2d21360, 0x5d48e80) /usr/local/go/src/runtime/panic.go:522 +0x1b5 github.com/rancher/rancher/pkg/controllers/user/monitoring.(*clusterMonitoringEnabledHandler).syncWindowsNode(0xc001c26940, 0xc00e20daa0, 0x5, 0x0, 0xc000a0c000, 0x7f46c70d3648, 0x0, 0x0) /go/src/github.com/rancher/rancher/pkg/controllers/user/monitoring/cluster_monitoring_enabled_handler.go:58 +0xe7 github.com/rancher/rancher/vendor/github.com/rancher/types/apis/core/v1.(*nodeController).AddHandler.func1(0xc00e20daa0, 0x5, 0x0, 0x0, 0xc009b4bd58, 0x3, 0x3, 0x0) /go/src/github.com/rancher/rancher/vendor/github.com/rancher/types/apis/core/v1/zz_generated_node_controller.go:149 +0xe0 github.com/rancher/rancher/vendor/github.com/rancher/norman/controller.(*genericController).syncHandler(0xc000c43260, 0x2aecfc0, 0xc00d3bf730, 0x0, 0x0) /go/src/github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:349 +0x35d github.com/rancher/rancher/vendor/github.com/rancher/norman/controller.(*genericController).processNextWorkItem(0xc000c43260, 0xc00598e800) /go/src/github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:258 +0xd3 github.com/rancher/rancher/vendor/github.com/rancher/norman/controller.(*genericController).runWorker(0xc000c43260) /go/src/github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:246 +0x2b github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc004e69480) /go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x54 github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc004e69480, 0x3b9aca00, 0x0, 0x1, 0xc0019a8c00) /go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8 github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc004e69480, 0x3b9aca00, 0xc0019a8c00) /go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d created by github.com/rancher/rancher/vendor/github.com/rancher/norman/controller.(*genericController).run /go/src/github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:238 +0xb7

2019/10/16 08:06:47 [INFO] [node-controller-docker-machine] (etcd2) Starting VM... 2019/10/16 08:06:49 [INFO] [node-controller-docker-machine] The default lines below are for a sh/bash shell, you can specify the shell you're using, with the --shell flag. 2019/10/16 08:06:49 [INFO] [node-controller-docker-machine] 2019/10/16 08:06:50 [INFO] Generating and uploading node config etcd2 2019/10/16 08:06:50 [ERROR] NodeController c-4v52t/m-p9xbc [node-controller] failed with : Error creating machine: Error in driver during machine creation: Multiple resources with that name.