Deleting installations should uninstall and keep the Kubernetes behavior consistent with resource garbage collection

[troy0820]

What design is being proposed?

Currently when you are deleting an installation resource, the custom resource will be deleted and the installation will be uninstalled. The behavior from before would uninstall the installation but leave around the custom resource. In our discussions it was said that we want to keep this behavior consistent with how Kubernetes does resources. The allowance of deleting a namespace where it would delete all the resources (uninstall and remove custom resources) should not impede the operations that porter operator needs to do to clean up the installations present on the cluster.

Additional Context

Propose that when deleting resources, we remove the installation and its components but allow the state to be saved in the backend e.g. mongodb, etc for reinstallation, etc. The operator right now lives in a namespace where that can happen but also can be deleted to ruin this implemenation.

Risks/Concerns

1. The risk is that the grpc server outputs will reflect the installation as maybe installed and/or not reflect the current state of the cluster.
2. Kubernetes doesn't like the way we are doing this and demands we follow the constraints of the cluster

What other approaches were considered?

1. Not removing the resource, however that will lead to a situation where the finalizer will never be removed and not allow the namespace to be deleted.
2. Removing the installation and uninstalling but not saving the state of the installation in mongo.

Implementation details

TBD - Gathering notes from the community meeting and discussing with the maintainers

Checklist

• [ ] This proposal has remained open for at least one week, to allow time for community feedback.

[schristoff]

Per discussion: Find other projects to figure out what behavior we should mimic - either requiring an uninstall -> delete, or on delete do we uninstall

[troy0820]

Continued discussion happened around this domain and was said to enable deleting in the respect of how cross-plane does this. They have a deletion policy, that marks the crd with a policy (annotation) where the default delete allows to remove all things.

For us this would include all installation outputs, and resources that the operator generated. We can start with the default behavior we expect when deleting k8s resources of remove everything and allow that to trigger the deletion. The removal of the installation (not the custom resource) is within the path of what we expect to do when the policy is delete.

TLDR;

1. Enable deletion policy with default and orphan (words we can change later, annotations on the installation)
2. Discuss orphan to be used as a thing to either delete the custom resource and leave the install running or remove the installation of things and keep the custom resource in a state to audit what was installed.