Resource deletion race condition

[jabbrwcky]

Hi,

I noticed that when using Pulumi to tear down an operator-managed instance and the namespace it is deployed to, the Redis stateful set is frequently deleted before the Redis CR is deleted, because the namespace deletion triggers the stateful set deletion, which happens fast for small (CI) instances.\

Apparently too fast for the operator, which fails to delete the CR, causing the namespace to hang.

Currently using redis-operator version: v0.11.0

Possible solutions (IMO):

• Add finalizers to managed resources (stateful sets/pvc) - the owner reference does not prevent deletion
• Gracefully handle the cases when controlled resources are deleted (e.g. recreate them if the CR is not set to delete and delete them or handle already deleted resources)

The second solution would be closer to the behaviour that kubernetes exposes for its own controllers (e.g. Replicasets/Pods), but might take more care and work than the more heavy-handed first approach

(I might find time later to implement a fix after familiarizing myself with the codebase)