TF deletion hanged when envFrom are also deleted and destroyResourcesOnDeletion is true

[davidblaisonneau-orange]

First, thanks for this great controller, it is a pleasure to work with it.

We have an issue when deleting TF linked to a ConfigMap, and when the terraform model requires a variable.

Here is the TF:

---
apiVersion: infra.contrib.fluxcd.io/v1alpha2
kind: Terraform
metadata:
  name: hello-alice
  namespace: flux-system
spec:
  interval: 1m
  approvePlan: auto
  path: ./
  destroyResourcesOnDeletion: true
  sourceRef:
    kind: GitRepository
    name: tf-hello
  varsFrom:
    - kind: ConfigMap
      name: hello-config-alice

At deletion, from FluxCD, we destroy TF + depending CM, but TF controller is looping on the deletion process because the runner cannot access the configmap anymore.

{"level":"error",
 "ts":"2023-06-28T13:43:41.993Z",
 "logger":"runner.terraform",
 "msg":"unable to get object key",
 "instance-id":"2b46e6cb-96bf-4df3-8758-48c94481dd2f",
 "objectKey":{
    "name":"hello-config-alice",
    "namespace":"flux-system"},
 "configmap":"",
 "error":"configmaps \"hello-config-alice\" not found"
}

I have push a demo project here for more details.

Test versions are:

flux: v2.0.0-rc.5
helm-controller: v0.34.1
kustomize-controller: v1.0.0-rc.4
notification-controller: v1.0.0-rc.4
source-controller: v1.0.0-rc.5
tf-controller: v0.15.0-rc.4

Can you confirm this behavior ?

Is there any way to avoid it - without removing the destroyResourcesOnDeletion ?

Maybe adding a finalizer on ConfigMap or Secrets per runner can be an option, or having a cache for tfvars ?

Thanks a lot for your help.

========

User Story

As a Terraform Controller user, I'd like Terraform Controller to reliably handle deletions when both Terraform Custom Resource and linked ConfigMaps are deleted, so that I do not run into issues during the deletion process.

Acceptance Criteria

• [ ] Terraform Controller completes the deletion process when both a linked ConfigMap and the Terraform Custom Resource are deleted.
• [ ] The Terraform Controller does not loop on the deletion process if it cannot access a linked ConfigMap.
• [ ] Terraform BLOB caching should be considered as a possible solution to solve this issue and avoid finalizers on ConfigMaps or Secrets.
• [ ] We’ve added an e2e test to measure success of the implementation.
• [ ] We’ve created related user documentation.

Terraform BLOB caching should be considered as a possible solution to solve this issue and avoid finalizers on ConfigMaps or Secrets.

[chanwit]

Hi @davidblaisonneau-orange thank you for reporting this. We're aware of this kind of behavior.

Adding finalizers to ConfigMaps or Secrets which are not managed by the controller would cause other problems. A generalized solution to the resource deletion problems would be caching the generated BLOBs and use them only for the destruction phase.

[davidblaisonneau-orange]

Thanks @chanwit for this quick answer.

For my understanding what would be the issue with finalizers ?

[chanwit]

It's from the discussion with the Flux team.

They suggested that adding finalizers to those objects would lead to some unforeseen behaviors, for example, it would block the uninstallation process of Flux, etc.

After the discussion with them, we haven't decided to use finalizers for this problem yet.

However, terraform BLOB caching might help us solve this whole set of problems.