Periodically Cleanup Orphaned Shadow Secrets

[jaireddjawed]

Description

When an HCPVaultSecretsApp is deleted, the handleDeletion() method is called to remove the app's shadow secrets from k8s. However, if handleDeletion() fails to remove the secrets for some reason, the orphaned shadow secrets remain in k8s indefinitely because we don't have a mechanism that attempts to remove these shadow secrets again later.

This PR addresses this issue by creating a goroutine that periodically checks for deleted HVS apps and removes the app's shadow secrets.

Local Testing

• Commented the initial deletion attempt of a HCPVaultSecretsApp in Reconcile() This is to mock the scenario where the first handleDeletion() call fails (mentioned here).

• Created an HCPVaultSecretsApp that includes a dynamic secret (yaml config below)

apiVersion: v1
data:
  clientID: redacted
  clientSecret: redacted
kind: Secret
metadata:
  name: vso-demo-sp
  namespace: default
type: Opaque

---
apiVersion: secrets.hashicorp.com/v1beta1
kind: HCPAuth
metadata:
  name: default
  namespace: vault-secrets-operator-system
spec:
  organizationID: redacted
  projectID: redacted
  servicePrincipal:
    secretRef: vso-demo-sp

---
apiVersion: secrets.hashicorp.com/v1beta1
kind: HCPVaultSecretsApp
metadata:
  name: web-application
  namespace: default
spec:
  appName: sample-app
  destination:
    create: true
    labels:
      hvs: "true"
    name: web-application
  refreshAfter: 15m

• Confirmed that secret exists in k8s (kubectl get secrets -o yaml)
• Deleted HCPVaultSecretsApp from k8s (kubectl delete hcpvaultsecretsapps.secrets.hashicorp.com web-application)
• Confirmed that the secret no longer exists in k8s (kubectl get secrets -o yaml)

Jira Ticket

https://hashicorp.atlassian.net/browse/VAULT-31820

[benashz]

@jaireddjawed let's hold off on bringing this change into the v0.10.0 release. I have moved it to the v0.10.1 milestone.