[investigate] Clean up owned resources in Finalizer

[yitsushi]

Investigation about what resources:

1. we want to delete, but it's not deleted now, so we can fix it
2. we don't want to delete, and it's not deleted now, so we can verify
3. we don't want to delete, and it's deleted not, so we can fix it (i don't think we have such resource now)

All resources should be documented if they are deleted with the Terraform resource or not and why.

---

When a Terraform object is deleted, ConfigMaps and Secrets are not deleted, while they have OwnerReference to the deleted Terraform objects.

It's possibly a bug as we should clean up after ourselves. If someone uses that leftover garbage (as the gc should clean it up), that feels like they use something they shouldn't.

1. It's an issue as we leave garbage behind
2. It's an issue if someone creates a new Terraform object with the same ref, should we keep the existing one (with all history) and update OwnerReference? Should we nuke and create one?

Possible resolution:

• Delete all objects we created.
• If a ConfigMap / Secret was there before we did nothing (user already created it), don't set owner reference (as we are not the owner, we did not create it)

[yiannistri]

@squaremo what is your opinion on this?

[squaremo]

When a Terraform object is deleted, ConfigMaps and Secrets are not deleted, while they have OwnerReference to the deleted Terraform objects.

The Kubernetes API should take care of removing dependent objects. Any idea why it's not?

[yitsushi]

Now I wonder which controller handles deletion of ConfigMaps Can't find anything obvious in https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/ Maybe it doesn't have a controller at all as it's just a bunch of key/values without runtime behaviour so it just gets stored in etcd

by @yiannistri (on Slack)

I don't know, but it's worth a try to dig deeper in that hole, for example k8s IRC/Matrix/Slack channels.

[squaremo]

I'm unclear on whether we have direct evidence that ConfigMap object with ownerships don't get garbage-collected -- can we do an experiment?

[lasomethingsomething]

This is related: https://github.com/weaveworks/tf-controller/issues/538

[lasomethingsomething]

Worth considering this: https://github.com/weaveworks/tf-controller/issues/538#issuecomment-1448204835

[yitsushi]

I'm unclear on whether we have direct evidence that ConfigMap object with ownerships don't get garbage-collected -- can we do an experiment?

I can confirm, secrets get deleted when OwnerReference is set, it gets deleted even when I have my own finalizer and it does not delete the secret: https://github.com/yitsushi/controller-ownref-experiment

So somewhere we set something that prevents secrets and configmaps to be deleted.

---

we set OwnerReference on plan secret and configmap:

• https://github.com/weaveworks/tf-controller/blob/main/runner/server_save_tfplan.go#L123
• https://github.com/weaveworks/tf-controller/blob/main/runner/server_save_tfplan.go#L181

[yitsushi]

Descriptions needs some loving touch. Last time we talked about it, we said we turn this issue into an investigation about what resources:

1. we want to delete, but it's not deleted now, so we can fix it
2. we don't want to delete, and it's not deleted now, so we can verify
3. we don't want to delete, and it's deleted not, so we can fix it (i don't think we have such resource now)

All resources should be documented if they are deleted with the Terraform resource or not and why.