vsphere-csi-driver icon indicating copy to clipboard operation
vsphere-csi-driver copied to clipboard

Published 20 hours ago • verified publisher icon kubernetes-sigs

"failed to set control flag keepAfterDeleteVm" after CSI migration is enabled

Open jsafrane opened this issue 1 year ago • 28 comments

/kind bug

What happened: After I enabled CSI migration in a 1.25 cluster, I got some Pods pending with unattachable volumes. The CSI driver fails every ControllerPublish with failed to set control flag keepAfterDeleteVm for volumeID "bf019d24-72e2-4ba0-9b72-10df23a69156" with err: ServerFaultCode: The operation is not allowed in the current state.

What you expected to happen: Volumes are attached just fine.

How to reproduce it (as minimally and precisely as possible):

  1. Create 50 PVCs + Deployments that use them on a cluster without CSI migration.
  2. Enable CSI migration
    1. Enable it in kube-apiserver, then KCM and scheduler.
    2. Enable it in the CSI driver ConfigMap and restart the driver Pods.
    3. Drain nodes one by one and enable it in kubelet + restart kubelet.
  3. Check the Deployments from step 1.

Once out of 8 attempts I got failed to set control flag keepAfterDeleteVm for ~5 of the Deployments. Most of the Deployments were fine. The error repeats for at least 30 minutes for each ControllerPublish retry of each affected volume, then I gave up and finished my tests.

Anything else we need to know?:

The Deployment Pods were running just fine before CSI migration was enabled. There is no pending volume resize or anything that could change state of a volume, at least not in Kubernetes. I don't have the cluster any longer, so I cannot check how the volumes look like in vCenter. What volume "state" would prevent a volume from being marked by keepAfterDeleteVm? And how could Kubernetes / the CSI driver / anything else put the volume to such a state?

Environment:

  • csi-vsphere version: 2.4.1
  • vsphere-cloud-controller-manager version: ?
  • Kubernetes version: v1.25.4
  • vSphere version: 7.0.3
  • OS (e.g. from /etc/os-release): Red Hat Enterprise Linux CoreOS 412.86.202301100600-0
  • Kernel (e.g. uname -a): 4.18.0-372.40.1.el8_6.x86_64
  • Install tools: OpenShift 4.12

jsafrane avatar Jan 17 '23 16:01 jsafrane

CSI driver logs:

2023-01-10T16:16:38.891319829Z 2023-01-10T16:16:38.891Z DEBUG   migration/migration.go:214      VolumeID: "e582b9ad-5574-4086-b201-773b5a2f677d" found from the cache for VolumePath: "[WorkloadDatastore] 706cfa61-7a2b-d2dd-d60e-06b6d880c5b7/jsafrane-vk95d-dynamic-pvc-f659ef81-d56c-4373-b18e-7a49309c159c.vmdk"   {"Trac
2023-01-10T16:16:38.894946321Z 2023-01-10T16:16:38.894Z INFO    migration/migration.go:254      Set keepAfterDeleteVm control flag using Vslm APIs      {"TraceId": "d8ee18c7-4fc8-48de-a900-96d5e0231f28"}
2023-01-10T16:16:39.091082831Z 2023-01-10T16:16:39.091Z INFO    vanilla/controller.go:1026      ControllerPublishVolume: called with args {VolumeId:[WorkloadDatastore] 706cfa61-7a2b-d2dd-d60e-06b6d880c5b7/jsafrane-vk95d-dynamic-pvc-b9a4dd3b-a5fd-403e-8bc9-ec8117c5f75f.vmdk NodeId:jsafrane-vk95d-worker-bdm8z VolumeCap
2023-01-10T16:16:39.091140000Z 2023-01-10T16:16:39.091Z DEBUG   migration/migration.go:214      VolumeID: "bf019d24-72e2-4ba0-9b72-10df23a69156" found from the cache for VolumePath: "[WorkloadDatastore] 706cfa61-7a2b-d2dd-d60e-06b6d880c5b7/jsafrane-vk95d-dynamic-pvc-b9a4dd3b-a5fd-403e-8bc9-ec8117c5f75f.vmdk"   {"Trac
2023-01-10T16:16:39.095525997Z 2023-01-10T16:16:39.095Z INFO    migration/migration.go:254      Set keepAfterDeleteVm control flag using Vslm APIs      {"TraceId": "d4896889-af56-4397-bacc-064ca2894602"}
2023-01-10T16:16:39.166971232Z 2023-01-10T16:16:39.166Z ERROR   volume/manager.go:2284  failed to set control flag keepAfterDeleteVm  for volumeID "bf019d24-72e2-4ba0-9b72-10df23a69156" with err: ServerFaultCode: The operation is not allowed in the current state. {"TraceId": "d4896889-af56-4397-bacc-064ca2894602"}
2023-01-10T16:16:39.166971232Z sigs.k8s.io/vsphere-csi-driver/v2/pkg/common/cns-lib/volume.(*defaultManager).ProtectVolumeFromVMDeletion
2023-01-10T16:16:39.166971232Z  /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/common/cns-lib/volume/manager.go:2284
2023-01-10T16:16:39.166971232Z sigs.k8s.io/vsphere-csi-driver/v2/pkg/apis/migration.(*volumeMigration).ProtectVolumeFromVMDeletion
2023-01-10T16:16:39.166971232Z  /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/apis/migration/migration.go:255
2023-01-10T16:16:39.166971232Z sigs.k8s.io/vsphere-csi-driver/v2/pkg/csi/service/vanilla.(*controller).ControllerPublishVolume.func1
2023-01-10T16:16:39.166971232Z  /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/csi/service/vanilla/controller.go:1104
2023-01-10T16:16:39.166971232Z sigs.k8s.io/vsphere-csi-driver/v2/pkg/csi/service/vanilla.(*controller).ControllerPublishVolume
2023-01-10T16:16:39.166971232Z  /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/csi/service/vanilla/controller.go:1135
2023-01-10T16:16:39.166971232Z github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerPublishVolume_Handler
2023-01-10T16:16:39.166971232Z  /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5705
2023-01-10T16:16:39.166971232Z google.golang.org/grpc.(*Server).processUnaryRPC
2023-01-10T16:16:39.166971232Z  /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:1297
2023-01-10T16:16:39.166971232Z google.golang.org/grpc.(*Server).handleStream
2023-01-10T16:16:39.166971232Z  /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:1626
2023-01-10T16:16:39.166971232Z google.golang.org/grpc.(*Server).serveStreams.func1.2
2023-01-10T16:16:39.166971232Z  /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:941
2023-01-10T16:16:39.167038115Z 2023-01-10T16:16:39.166Z ERROR   migration/migration.go:257      failed to protect migrated volume from vm deletion. Volume ID: "bf019d24-72e2-4ba0-9b72-10df23a69156", err: ServerFaultCode: The operation is not allowed in the current state. {"TraceId": "d4896889-af56-4397-bacc-064ca2894
2023-01-10T16:16:39.167038115Z sigs.k8s.io/vsphere-csi-driver/v2/pkg/apis/migration.(*volumeMigration).ProtectVolumeFromVMDeletion
2023-01-10T16:16:39.167038115Z  /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/apis/migration/migration.go:257
2023-01-10T16:16:39.167038115Z sigs.k8s.io/vsphere-csi-driver/v2/pkg/csi/service/vanilla.(*controller).ControllerPublishVolume.func1
2023-01-10T16:16:39.167038115Z  /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/csi/service/vanilla/controller.go:1104
2023-01-10T16:16:39.167038115Z sigs.k8s.io/vsphere-csi-driver/v2/pkg/csi/service/vanilla.(*controller).ControllerPublishVolume
2023-01-10T16:16:39.167038115Z  /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/csi/service/vanilla/controller.go:1135
2023-01-10T16:16:39.167038115Z github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerPublishVolume_Handler
2023-01-10T16:16:39.167038115Z  /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5705
2023-01-10T16:16:39.167038115Z google.golang.org/grpc.(*Server).processUnaryRPC
2023-01-10T16:16:39.167038115Z  /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:1297
2023-01-10T16:16:39.167038115Z google.golang.org/grpc.(*Server).handleStream
2023-01-10T16:16:39.167038115Z  /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:1626
2023-01-10T16:16:39.167038115Z google.golang.org/grpc.(*Server).serveStreams.func1.2
2023-01-10T16:16:39.167038115Z  /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:941
2023-01-10T16:16:39.167038115Z 2023-01-10T16:16:39.166Z ERROR   vanilla/controller.go:1106      failed to set keepAfterDeleteVm control flag for VolumeID "bf019d24-72e2-4ba0-9b72-10df23a69156"        {"TraceId": "d4896889-af56-4397-bacc-064ca2894602"}
2023-01-10T16:16:39.167038115Z sigs.k8s.io/vsphere-csi-driver/v2/pkg/csi/service/vanilla.(*controller).ControllerPublishVolume.func1
2023-01-10T16:16:39.167038115Z  /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/csi/service/vanilla/controller.go:1106
2023-01-10T16:16:39.167038115Z sigs.k8s.io/vsphere-csi-driver/v2/pkg/csi/service/vanilla.(*controller).ControllerPublishVolume
2023-01-10T16:16:39.167038115Z  /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/csi/service/vanilla/controller.go:1135
2023-01-10T16:16:39.167038115Z github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerPublishVolume_Handler
2023-01-10T16:16:39.167038115Z  /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5705
2023-01-10T16:16:39.167038115Z google.golang.org/grpc.(*Server).processUnaryRPC
2023-01-10T16:16:39.167038115Z  /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:1297
2023-01-10T16:16:39.167038115Z google.golang.org/grpc.(*Server).handleStream
2023-01-10T16:16:39.167038115Z  /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:1626
2023-01-10T16:16:39.167038115Z google.golang.org/grpc.(*Server).serveStreams.func1.2
2023-01-10T16:16:39.167038115Z  /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:941
2023-01-10T16:16:39.167038115Z 2023-01-10T16:16:39.166Z DEBUG   vanilla/controller.go:1136      controllerPublishVolumeInternal: returns fault "csi.fault.Internal" for volume "bf019d24-72e2-4ba0-9b72-10df23a69156"   {"TraceId": "d4896889-af56-4397-bacc-064ca2894602"}
2023-01-10T16:16:39.177353940Z 2023-01-10T16:16:39.177Z INFO    volume/manager.go:2287  Successfully set keepAfterDeleteVm control flag for volumeID: "e582b9ad-5574-4086-b201-773b5a2f677d"    {"TraceId": "d8ee18c7-4fc8-48de-a900-96d5e0231f28"}
2023-01-10T16:16:39.177353940Z 2023-01-10T16:16:39.177Z INFO    migration/migration.go:260      Migrated Volume with ID: "e582b9ad-5574-4086-b201-773b5a2f677d" is protected from VM deletion   {"TraceId": "d8ee18c7-4fc8-48de-a900-96d5e0231f28"}

I intentionally left successful keepAfterDeleteVm seting for different volumes just before and after the failure for volume bf019d24-72e2-4ba0-9b72-10df23a69156.

jsafrane avatar Jan 17 '23 16:01 jsafrane

I have been able to reproduce this issue. I am not sure if this is because of number of attached volumes on a node. I have vcenter logs and support bundle if anyone at VMWare wants to debug this further.

gnufied avatar Jan 20 '23 20:01 gnufied

Have you checked draining node is detaching volume from the Node VM using vSphere in-tree plugin? We have observed in the past that the in-tree vSphere plugin does not detach the volume from the Node during drain and that results in CSI attach volume call failing to "set keepAfterDeleteVm control flag for VolumeID".

https://docs.vmware.com/en/VMware-vSphere-Container-Storage-Plug-in/2.7/rn/vmware-vsphere-container-storage-plugin-27-release-notes/index.html

This flag is important for migrated volume. Without this flag, if Node VM is deleted while migrated volumes are attached to it, Volumes will also get deleted along with the VM.

divyenpatel avatar Jan 20 '23 20:01 divyenpatel

/assign @divyenpatel

lipingxue avatar Jan 24 '23 23:01 lipingxue

We were able to reproduce this bug in a cluster on a node which has 30 volumes attached to it. The CSI driver supports 59 volumes to a node, so this number is well below that limit.

The workaround mentioned in - https://docs.vmware.com/en/VMware-vSphere-Container-Storage-Plug-in/3.0/rn/vmware-vsphere-container-storage-plugin-30-release-notes/index.html doesn't seem feasible. When we are migrating a larger cluster.

gnufied avatar Mar 23 '23 13:03 gnufied

@gnufied if volumes are still attached to Node VM even after drain, it is not a migration issue. It is an in-tree driver issue. Do we know why the in-tree plugin is not detaching volume from the node upon drain?

Can you check logs of kube-controller after Node drain to see if we have any issue in the in-tree plugin?

divyenpatel avatar Mar 23 '23 18:03 divyenpatel

@divyenpatel I have not seen any detach failures in the KCM. We will try and reproduce it and see what is going on. If it is really a detach failure or some kind of sticky bit set on the volume.

gnufied avatar Mar 23 '23 19:03 gnufied

@gnufied

another workaround we can try is to patch CnsVSphereVolumeMigration CR and set ProtectVolumeFromVMDelete in the CnsVSphereVolumeMigrationSpec as true. Refer to https://github.com/kubernetes-sigs/vsphere-csi-driver/blob/master/pkg/apis/migration/v1alpha1/types.go#L41 This will skip calling API to set this control flag, and if the volume is attached to the intended Node VM, we will be able to proceed, but if volume is attached to diff node vm, then pod will remain stuck with ResourceVolumeInUse fault.

divyenpatel avatar Mar 23 '23 19:03 divyenpatel

@gnufied even if there is a timeout during controller publish, CSI might get a call again to re-attach the same volume to the same node, and if the prior call has failed to update the CR after a successful control flag set, we do not proceed to attach the volume to the node. we update the CR first and then attach the volume to the Node. if during attach we observe volume is already attached to the intended node, we mark attach as a success. So with this workflow, it is very important to have volume detached from node VM at the time of setting the control flag to protect volume from VM deletion.

I see bug is reported for csi-vsphere version: 2.4.1 but we added this protection in 2.4.2 - https://github.com/kubernetes-sigs/vsphere-csi-driver/releases/tag/v2.4.2

also we have fixed some issues regarding attach/detach for Node API server object deletion, Node VM deletion etc in newer releases of the driver.

Have you tried migration using the latest v3.0.0 release to see if the issue is getting fixed?

divyenpatel avatar Mar 24 '23 18:03 divyenpatel

We ran the migration with 3.0.0-rc1, we haven't again tested with 3.0 final version but we managed to reproduce with 3.0.0-rc1.

Unfortunately, I haven't had any luck after that one run to reproduce the bug. But I do have KCM logs and Support Bundle.

Even if detach failed on original node, KCM should keep retrying detach because attach statuses are held in node.Status.Attached and it gets repopulated on KCM restart.

Do we know what is causing the failure in the first place? Happy to jump in a call if that helps.

gnufied avatar Mar 24 '23 18:03 gnufied

Do we know what is causing the failure in the first place? Happy to jump in a call if that helps.

Based on the workflow we are speculating there might be an issue in the in-tree plugin doing false detach. Unless we look into logs and support bundles we don't know where the issue is. Can you file an SR and provide us support bundle?

divyenpatel avatar Mar 24 '23 18:03 divyenpatel

got logs from @gnufied

Volume Registration as FCD was successful as part of the Migration. Volume ID: cb96305a-352a-4f3a-b22f-824e658f31d4 for PV - pvc-72ec9236-1da4-46d8-b3b0-ad5a44815b7a having Volume Path [vsanDatastore] d644b863-84ba-f34c-3d6a-3cfdfe98c6b8/jcallen3-ndk95-dynamic-pvc-72ec9236-1da4-46d8-b3b0-ad5a44815b7a.vmdk

2023-03-22T20:37:53.187112444Z {"level":"info","time":"2023-03-22T20:37:53.187050892Z","caller":"migration/migration.go:519","msg":"Registering volume: \"[vsanDatastore] d644b863-84ba-f34c-3d6a-3cfdfe98c6b8/jcallen3-ndk95-dynamic-pvc-72ec9236-1da4-46d8-b3b0-ad5a44815b7a.vmdk\" using backingDiskURLPath :\"https://vc/folder/d644b863-84ba-f34c-3d6a-3cfdfe98c6b8/jcallen3-ndk95-dynamic-pvc-72ec9236-1da4-46d8-b3b0-ad5a44815b7a.vmdk?dcPath=IBMCdatacenter&dsName=vsanDatastore\""}
2023-03-22T20:37:55.899766138Z {"level":"info","time":"2023-03-22T20:37:55.899673539Z","caller":"migration/migration.go:532","msg":"Successfully registered volume \"[vsanDatastore] d644b863-84ba-f34c-3d6a-3cfdfe98c6b8/jcallen3-ndk95-dynamic-pvc-72ec9236-1da4-46d8-b3b0-ad5a44815b7a.vmdk\" as container volume with ID: \"cb96305a-352a-4f3a-b22f-824e658f31d4\""}
2023-03-22T20:37:55.899866937Z {"level":"info","time":"2023-03-22T20:37:55.899727553Z","caller":"migration/migration.go:223","msg":"Successfully registered volumeSpec: &{[vsanDatastore] d644b863-84ba-f34c-3d6a-3cfdfe98c6b8/jcallen3-ndk95-dynamic-pvc-72ec9236-1da4-46d8-b3b0-ad5a44815b7a.vmdk } with CNS. VolumeID: cb96305a-352a-4f3a-b22f-824e658f31d4"}
2023-03-22T20:37:55.900150294Z {"level":"info","time":"2023-03-22T20:37:55.900081763Z","caller":"migration/migration.go:371","msg":"creating CR for cnsVSphereVolumeMigration: &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:cb96305a-352a-4f3a-b22f-824e658f31d4 GenerateName: Namespace: SelfLink: UID: ResourceVersion: Generation:0 CreationTimestamp:0001-01-01 00:00:00 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[] Finalizers:[] ManagedFields:[]} Spec:{VolumePath:[vsanDatastore] d644b863-84ba-f34c-3d6a-3cfdfe98c6b8/jcallen3-ndk95-dynamic-pvc-72ec9236-1da4-46d8-b3b0-ad5a44815b7a.vmdk VolumeID:cb96305a-352a-4f3a-b22f-824e658f31d4 ProtectVolumeFromVMDelete:false}}"}
2023-03-22T20:37:56.193874956Z {"level":"info","time":"2023-03-22T20:37:56.193634408Z","caller":"migration/migration.go:381","msg":"Successfully created CR for cnsVSphereVolumeMigration: &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:cb96305a-352a-4f3a-b22f-824e658f31d4 GenerateName: Namespace: SelfLink: UID:fee9e743-5574-4075-8bc6-5afe67d8fdaa ResourceVersion:288689 Generation:1 CreationTimestamp:2023-03-22 20:37:55 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[] Finalizers:[] ManagedFields:[{Manager:vmware-vsphere-csi-driver-syncer Operation:Update APIVersion:cns.vmware.com/v1alpha1 Time:2023-03-22 20:37:55 +0000 UTC FieldsType:FieldsV1 FieldsV1:{\"f:spec\":{\".\":{},\"f:protectvolumefromvmdelete\":{},\"f:volumeid\":{},\"f:volumepath\":{}}} Subresource:}]} Spec:{VolumePath:[vsanDatastore] d644b863-84ba-f34c-3d6a-3cfdfe98c6b8/jcallen3-ndk95-dynamic-pvc-72ec9236-1da4-46d8-b3b0-ad5a44815b7a.vmdk VolumeID:cb96305a-352a-4f3a-b22f-824e658f31d4 ProtectVolumeFromVMDelete:false}}"}

According to kubecontroller log, volume was last attached to the node jcallen3-ndk95-worker-wmv6j

2023-03-22T20:33:02.627450414+00:00 stderr F I0322 20:33:02.627423       1 actual_state_of_world.go:357] Volume "kubernetes.io/vsphere-volume/[vsanDatastore] d644b863-84ba-f34c-3d6a-3cfdfe98c6b8/jcallen3-ndk95-dynamic-pvc-72ec9236-1da4-46d8-b3b0-ad5a44815b7a.vmdk" is already added to attachedVolume list to node "jcallen3-ndk95-worker-wmv6j", update device path "/dev/disk/by-id/wwn-0x6000c2968912f14759e7ebe23eace2b4"

Volume is getting attached to the new Node - jcallen3-ndk95-worker-64h6q and kube-controller is not alerting for MultiAttachErrorReported, which means at that time volume is not attached to prior node jcallen3-ndk95-worker-wmv6j.

2023-03-22T20:43:31.946364207+00:00 stderr F I0322 20:43:31.946281       1 reconciler.go:346] "attacherDetacher.AttachVolume started" volume={VolumeToAttach:{MultiAttachErrorReported:false VolumeName:kubernetes.io/csi/csi.vsphere.vmware.com^[vsanDatastore] d644b863-84ba-f34c-3d6a-3cfdfe98c6b8/jcallen3-ndk95-dynamic-pvc-72ec9236-1da4-46d8-b3b0-ad5a44815b7a.vmdk VolumeSpec:0xc01174a3f0 NodeName:jcallen3-ndk95-worker-64h6q ScheduledPods:[&Pod{ObjectMeta:{sandbox-77b5dd8995-ppxwr sandbox-77b5dd8995- emacs82  dcfa7a63-2721-41a2-b523-d7758afe7dff 297539 0 2023-03-22 20:43:31 +0000 UTC <nil> <nil> map[pod-template-hash:77b5dd8995 run:sandbox] map[openshift.io/scc:restricted-v2 seccomp.security.alpha.kubernetes.io/pod:runtime/default] [{apps/v1 ReplicaSet sandbox-77b5dd8995 4143bec9-faf6-4792-98ef-21e674f7c099 0xc010d0b737 0xc010d0b738}] [] [{kube-controller-manager Update v1 2023-03-22 20:43:31 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:pod-template-hash":{},"f:run":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"4143bec9-faf6-4792-98ef-21e674f7c099\"}":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"sandbox\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:securityContext":{".":{},"f:allowPrivilegeEscalation":{},"f:capabilities":{".":{},"f:drop":{}},"f:runAsNonRoot":{},"f:seccompProfile":{".":{},"f:type":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/mnt/test\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"vol\"}":{".":{},"f:name":{},"f:persistentVolumeClaim":{".":{},"f:claimName":{}}}}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:vol,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:myclaim,ReadOnly:false,},RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:nil,StorageOS:nil,CSI:nil,Ephemeral:nil,},},Volume{Name:kube-api-access-9n8vz,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:openshift-service-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:service-ca.crt,Path:service-ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:sandbox,Image:gcr.io/google_containers/busybox,Command:[/bin/sh -c while true; do date; echo `date` >>/mnt/test/date; sleep 5; done],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:vol,ReadOnly:false,MountPath:/mnt/test,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-9n8vz,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[],Drop:[ALL],},Privileged:nil,SELinuxOptions:nil,RunAsUser:*1001510000,RunAsNonRoot:*true,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:*false,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:&SeccompProfile{Type:RuntimeDefault,LocalhostProfile:nil,},},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:jcallen3-ndk95-worker-64h6q,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:&SELinuxOptions{User:,Role:,Type:,Level:s0:c39,c14,},RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:*1001510000,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:&SeccompProfile{Type:RuntimeDefault,LocalhostProfile:nil,},},ImagePullSecrets:[]LocalObjectReference{LocalObjectReference{Name:default-dockercfg-lpcbd,},},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,HostUsers:nil,SchedulingGates:[]PodSchedulingGate{},ResourceClaims:[]PodResourceClaim{},},Status:PodStatus{Phase:Pending,Conditions:[]PodCondition{PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2023-03-22 20:43:31 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:,PodIP:,StartTime:<nil>,ContainerStatuses:[]ContainerStatus{},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{},EphemeralContainerStatuses:[]ContainerStatus{},},}]}}

CSI Volume attachment shows volume is not attached to the intended node - jcallen3-ndk95-worker-64h6q

---
apiVersion: storage.k8s.io/v1
kind: VolumeAttachment
metadata:
  annotations:
    csi.alpha.kubernetes.io/node-id: jcallen3-ndk95-worker-64h6q
  creationTimestamp: "2023-03-22T20:43:31Z"
  finalizers:
  - external-attacher/csi-vsphere-vmware-com
  managedFields:
  - apiVersion: storage.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:attacher: {}
        f:nodeName: {}
        f:source:
          f:persistentVolumeName: {}
    manager: kube-controller-manager
    operation: Update
    time: "2023-03-22T20:43:31Z"
  - apiVersion: storage.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:csi.alpha.kubernetes.io/node-id: {}
        f:finalizers:
          .: {}
          v:"external-attacher/csi-vsphere-vmware-com": {}
    manager: csi-attacher
    operation: Update
    time: "2023-03-22T20:43:35Z"
  - apiVersion: storage.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:attachError:
          .: {}
          f:message: {}
          f:time: {}
    manager: csi-attacher
    operation: Update
    subresource: status
    time: "2023-03-22T23:02:30Z"
  name: csi-af7f48a80894e283b946abf022a3b0fc6255c879e309223f0051171788741fe6
  resourceVersion: "455747"
  uid: 4fc44ff2-b6d1-462e-9ef5-6f326059217b
spec:
  attacher: csi.vsphere.vmware.com
  nodeName: jcallen3-ndk95-worker-64h6q
  source:
    persistentVolumeName: pvc-72ec9236-1da4-46d8-b3b0-ad5a44815b7a
status:
  attachError:
    message: 'rpc error: code = Internal desc = failed to set keepAfterDeleteVm control
      flag for VolumeID "cb96305a-352a-4f3a-b22f-824e658f31d4"'
    time: "2023-03-22T23:02:30Z"
  attached: false

CSI ControllerPublishVolume is failing to set control flag due to "CNS: Failed to retrieve datastore for vol cb96305a-352a-4f3a-b22f-824e658f31d4"

Note: This is different issue then what was originally reported. Originally it was reported for "failed to set control flag keepAfterDeleteVm for volumeID "bf019d24-72e2-4ba0-9b72-10df23a69156" with err: ServerFaultCode: The operation is not allowed in the current state." which means volume is attached to Vm while we are calling set control flag API.

2023-03-22T20:43:35.221964575Z {"level":"info","time":"2023-03-22T20:43:35.221832144Z","caller":"vanilla/controller.go:2025","msg":"ControllerPublishVolume: called with args {VolumeId:[vsanDatastore] d644b863-84ba-f34c-3d6a-3cfdfe98c6b8/jcallen3-ndk95-dynamic-pvc-72ec9236-1da4-46d8-b3b0-ad5a44815b7a.vmdk NodeId:jcallen3-ndk95-worker-64h6q VolumeCapability:mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER >  Readonly:false Secrets:map[] VolumeContext:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"33771182-5016-4d35-acfb-448ee370e1d0"}
2023-03-22T20:43:35.227474497Z {"level":"info","time":"2023-03-22T20:43:35.227364963Z","caller":"migration/migration.go:254","msg":"Set keepAfterDeleteVm control flag using Vslm APIs","TraceId":"33771182-5016-4d35-acfb-448ee370e1d0"}
2023-03-22T20:43:35.248307185Z {"level":"error","time":"2023-03-22T20:43:35.248138086Z","caller":"volume/manager.go:2580","msg":"failed to set control flag keepAfterDeleteVm  for volumeID \"cb96305a-352a-4f3a-b22f-824e658f31d4\" with err: ServerFaultCode: CNS: Failed to retrieve datastore for vol cb96305a-352a-4f3a-b22f-824e658f31d4.","TraceId":"33771182-5016-4d35-acfb-448ee370e1d0","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/common/cns-lib/volume.(*defaultManager).ProtectVolumeFromVMDeletion\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/common/cns-lib/volume/manager.go:2580\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/apis/migration.(*volumeMigration).ProtectVolumeFromVMDeletion\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/apis/migration/migration.go:255\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).ControllerPublishVolume.func1\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/csi/service/vanilla/controller.go:2108\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).ControllerPublishVolume\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/csi/service/vanilla/controller.go:2147\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerPublishVolume_Handler\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5707\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:1283\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:1620\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:922"}
2023-03-22T20:43:35.248356360Z {"level":"error","time":"2023-03-22T20:43:35.248274729Z","caller":"migration/migration.go:257","msg":"failed to protect migrated volume from vm deletion. Volume ID: \"cb96305a-352a-4f3a-b22f-824e658f31d4\", err: ServerFaultCode: CNS: Failed to retrieve datastore for vol cb96305a-352a-4f3a-b22f-824e658f31d4.","TraceId":"33771182-5016-4d35-acfb-448ee370e1d0","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/apis/migration.(*volumeMigration).ProtectVolumeFromVMDeletion\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/apis/migration/migration.go:257\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).ControllerPublishVolume.func1\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/csi/service/vanilla/controller.go:2108\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).ControllerPublishVolume\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/csi/service/vanilla/controller.go:2147\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerPublishVolume_Handler\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5707\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:1283\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:1620\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:922"}
2023-03-22T20:43:35.248446399Z {"level":"error","time":"2023-03-22T20:43:35.248352137Z","caller":"vanilla/controller.go:2110","msg":"failed to set keepAfterDeleteVm control flag for VolumeID \"cb96305a-352a-4f3a-b22f-824e658f31d4\"","TraceId":"33771182-5016-4d35-acfb-448ee370e1d0","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).ControllerPublishVolume.func1\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/csi/service/vanilla/controller.go:2110\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).ControllerPublishVolume\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/csi/service/vanilla/controller.go:2147\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerPublishVolume_Handler\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5707\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:1283\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:1620\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:922"}
2023-03-22T20:43:35.248503041Z {"level":"error","time":"2023-03-22T20:43:35.248427108Z","caller":"vanilla/controller.go:2150","msg":"Operation failed, reporting failure status to Prometheus. Operation Type: \"attach-volume\", Volume Type: \"block\", Fault Type: \"csi.fault.Internal\"","TraceId":"33771182-5016-4d35-acfb-448ee370e1d0","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).ControllerPublishVolume\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/csi/service/vanilla/controller.go:2150\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerPublishVolume_Handler\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5707\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:1283\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:1620\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/src/github.com/kubernetes-sigs/vsphere-csi-driver/vendor/google.golang.org/grpc/server.go:922"}

I am following up with vSphere FCD team to debug why FCD is lost?

divyenpatel avatar Mar 24 '23 20:03 divyenpatel

We have update from FCD team.

As part of the migration process, we can reduce the likelihood of encountering this issue by waiting for all in-tree PVs to be registered as FCD disks before invoking any VM reconfigure APIs. Once all PVs have been registered, we can perform a Node drain operation to detach all registered FCDs from the Node VM.

We can upgrade the control plane node with Migration enabled for the kube-controller with vSphere CSI driver pre-installed. and check cnsVSphereVolumeMigration CR is created with all PVs which need to be migrated with VolumeID mapped to vmdk path in cnsVSphereVolumeMigration CR.

After that, we can perform the node drain. This can help reduce the possibility of this issue of having FCD not found in the CNS DB while performing attach operation.

divyenpatel avatar Mar 30 '23 19:03 divyenpatel

The FCD team also needs a verbose log to continue debug why FCD is lost.

divyenpatel avatar Mar 30 '23 19:03 divyenpatel

The issue we observed based on the logs and resolution is published here - https://kb.vmware.com/s/article/91752

divyenpatel avatar Apr 20 '23 16:04 divyenpatel

Our CI tests start have the same failures often now. I checked the kb article, but there is not Resolution yet. The workaround is to Restart vsan-health service. But this will not help for the test failures. Will the real fix be available any time soon?

jingxu97 avatar Jun 20 '23 06:06 jingxu97

Re https://github.com/kubernetes-sigs/vsphere-csi-driver/issues/2165#issuecomment-1490788517, the problem is CSI migration is normally performed during cluster upgrade, when the node drain happened. So we cannot make sure FCD is registered before node drain starts.

@divyenpatel @gnufied @jsafrane

jingxu97 avatar Jun 20 '23 06:06 jingxu97

Yeah same for us. Migration is enabled automatically on upgrade and process is usually same for all cloudproviders. It is not feasible to add hooks for one cloudprovider to wait and talk to vCenter before starting to drain the nodes.

gnufied avatar Jul 18 '23 18:07 gnufied

@gnufied if both vCenter and ESXi on vSphere 8.0u1, then we will not hit into this issue while setting control flag.

divyenpatel avatar Jul 31 '23 23:07 divyenpatel

@divyenpatel does this require any changes in CSI driver or it is automatic?

Also does 8.0u1 has fixes for both the issue we discussed above?

  1. failed to set control flag keepAfterDeleteVm because of inconsistent state.
  2. failed to set control flag because volume was not found.

gnufied avatar Aug 02 '23 14:08 gnufied

7.0p07 and 8.0u1 has the fix for allow setting control flag even when volume is attached to some VM, so we will not hit into error while setting the control flag even when disk is attached to some vm.

I will get back to you after checking with team regarding second issue - volume is lost from CNS.

divyenpatel avatar Aug 14 '23 18:08 divyenpatel

Second issue is not present in 7.0 but only there in 8.0 and 8.0u1. in 8.0u2 we have fixed the second issue regarding CNS volume not found in the memory. Ref - https://kb.vmware.com/s/article/91752

divyenpatel avatar Aug 22 '23 20:08 divyenpatel

7.0p07 and 8.0u2 (GA in Aug 31st) is recommended for in-tree vSphere volume to CSI migration.

divyenpatel avatar Aug 22 '23 20:08 divyenpatel

@divyenpatel Is there a manual workaround without updating to the vsphere new versions?

jingxu97 avatar Oct 04 '23 17:10 jingxu97

@divyenpatel I searched up on 7.0p07, it looks like mapping to build 21424296 https://packages.vmware.com/tools/versions But the build was published 2023/03/30 according to https://kb.vmware.com/s/article/2143832.

Could you please help double check this is the right version? I think the fix should be merged after Aug?

jingxu97 avatar Oct 23 '23 23:10 jingxu97

7.0p07 maps to Build: 21477706 https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-vcenter-server-70u3l-release-notes.html

divyenpatel avatar Oct 24 '23 13:10 divyenpatel

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Jan 31 '24 00:01 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle rotten
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot avatar Mar 01 '24 01:03 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-triage-robot avatar Mar 31 '24 02:03 k8s-triage-robot

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot avatar Mar 31 '24 02:03 k8s-ci-robot