PVC with blocked delete results in silent hang of Kustomize

[mikee]

Version: 0.19.1 Settings: force: true prune: false

Ran into a curious issue where a particular overlay was never finishing reconciliation. After testing and patching the controller to add some debug I found the following issue.

1. Create a PVC using an overlay of a particular size.
2. Create a pod that uses that PVC
3. Manually increase the size of the PVC
4. Reconcile the overlay again.

What happens is that k8s throws back a nasty error saying rightly that it can't shrink a PVC: The PersistentVolumeClaim "pvc-test" is invalid: spec.resources.requests.storage: Forbidden: field can not be less than previous value

However - the controller doesn't report this back. (Here we get into guess work). What it looks like happens is

1. the controller tries to force the change by deleting the PVC.
2. the PVC delete is blocked as it's in use
3. the controller just hangs in the wait phase and this kustomize layer will never complete or fail. As as result there is no way to tell what happened without extensive debugging.