Volume delete failed with static provisioning when persistentVolumeReclaimPolicy is Delete

[aparker-cityworks]

/kind bug

What happened? Persistent Volume deletion failed with error:

I1204 23:24:58.375475       1 controller.go:1502] delete "tenant-shared-efs-pv": started
E1204 23:24:58.376974       1 controller.go:1512] delete "tenant-shared-efs-pv": volume deletion failed: rpc error: code = NotFound desc = Failed to find access point for volume: fs-04653df1fc5fe9f42:/__shared-data
W1204 23:24:58.377321       1 controller.go:989] Retrying syncing volume "tenant-shared-efs-pv", failure 0
I1204 23:24:58.377688       1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolume", Namespace:"", Name:"tenant-shared-efs-pv", UID:"9ea5799b-f6fe-4084-aa22-26a985f6465d", APIVersion:"v1", ResourceVersion:"6113468", FieldPath:""}): type: 'Warning' reason: 'VolumeFailedDelete' rpc error: code = NotFound desc = Failed to find access point for volume: fs-04653df1fc5fe9f42:/__shared-data
E1204 23:24:58.377733       1 controller.go:1007] error syncing volume "tenant-shared-efs-pv": rpc error: code = NotFound desc = Failed to find access point for volume: fs-04653df1fc5fe9f42:/__shared-data

What you expected to happen? The PV should have been deleted successfully with the persistentVolumeReclaimPolicy set to Delete.

How to reproduce it (as minimally and precisely as possible)? Create PV and PVC similar to the static provisioning/volume_path example, but set the persistentVolumeReclaimPolicy on the PV to Delete. When the PVC is deleted the controller will run the delete volume function and will fail with the above message.

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com

---
# PV with persistentVolumeReclaimPolicy: Delete
apiVersion: v1
kind: PersistentVolume
metadata:
    name: tenant-shared-efs-pv
spec:
    capacity:
        storage: 5Gi
    volumeMode: Filesystem
    accessModes:
        - ReadWriteMany
    storageClassName: efs-sc
    persistentVolumeReclaimPolicy: Delete
    csi:
        driver: efs.csi.aws.com
        volumeHandle: fs-04653df1fc5fe9f42:/__shared-data

---
# PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
    name: tenant-shared-efs-claim
spec:
    accessModes:
        - ReadWriteMany
    storageClassName: efs-sc
    volumeName: tenant-shared-efs-pv
    resources:
        requests:
            storage: 5Gi

Environment

• Kubernetes version (use kubectl version): Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.3-eks-4f4795d

• Driver version: 1.7.1

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten