cannot delete resource with terraform plan failed when Source not found

[marcobarducci]

i noticed that there is no way to delete a resource with a plan failed. with tfctl you've an output that confirms that the resource is deleted but you can see it again with tfctl get. you cannot even delete the crd. seems that it cannot find the state and goes into a loop.

{"level":"info","ts":"2022-11-11T21:05:16.219Z","msg":"show runner pod state: ","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"uwe2-test-ek s","namespace":"flux-system"},"namespace":"flux-system","name":"uwe2-test-eks","reconcileID":"ca8fdf73-c7fe-4b9f-b2ac-aff01ad146c4","reconciliation-loop-id":"eb0c8bc6-ccf8-4c5b-bfc8-d4782aaeb75c","start-time":"2 022-11-11T21:05:16.219Z","name":"uwe2-test-eks","state":"not-found"}

========

User Story

As a Terraform Controller user, I'd like to be able to delete a Terraform resource when the plan fails due to an unresolvable source or variable, so that I can clean up resources that cannot be applied correctly.

Acceptance Criteria

• [ ] Can successfully delete a Terraform resource when the plan fails due to a missing or unresolvable source.
• [ ] Can successfully delete a Terraform resource when the plan fails due to an unresolved variable.
• [ ] Properly tear down and clean up all resources associated with the Terraform resource upon deletion, including GitRepositories created by branch planner.
• [ ] Testing added.
• [ ] We’ve created related user documentation.

[chanwit]

Thank you @marcobarducci for reporting this.

Could you describe this problem in more details please? Would be great to have steps to reproduce the problem.

[marcobarducci]

when you deploy a resource that will fail (for example Source 'OCIRepository/aws-package-v4.33.0' not found) or if it fails the plan because cannot find some variable, you cannot delete it. if you run tfctl delete the ctl will output "resource deleted" but the resource still remain.

[chanwit]

I see. Thanks for the input. I'll take a look at it shortly.

[navpreet-securitas]

similar problem happens when working with branch planner. since branch planner creates new flux - GitRepository whenever a PR is created and then creates terraform object. it happened once for me that when a PR is merged, seems it triggered deletion of GitRepository before deleting terraform object. Then terraform object deletion is struck due to this main issue.

abc-test-dev-service-monitoring-54   False     Source 'GitRepository/flux-system/abckconfig-service-monitoring-54' not found   45m

error which i get in logs :

2023-08-24 15:57:14 | {"level":"info","ts":"2023-08-24T13:57:14.009Z","logger":"polling-server","msg":"deleted Terraform abc-test-dev-service-monitoring-54 in the namespace flux-system","version":"","sha":""}
2023-08-24 15:57:13 | {"level":"error","ts":"2023-08-24T13:57:13.991Z","logger":"polling-server","msg":"unable to delete Source for Terraform abc-test-dev-service-monitoring-54 in the namespace flux-system","version":"","sha":"","error":"unable to get Source for Terraform abc-test-dev-service-monitoring-54 in the namespace flux-system: unable to get Source: gitrepositories.source.toolkit.fluxcd.io \"abckconfig-service-monitoring-54\" not found"}
2023-08-24 15:57:13 | {"level":"info","ts":"2023-08-24T13:57:13.985Z","logger":"polling-server","msg":"the PR either does not exist or has been closed, deleting corresponding Terraform object...","version":"","sha":"","terraform":"abc-test-dev","namespace":"flux-system","source":"abckconfig","PR ID":"54"}

@chanwit : another question related to branch planner. Shouldn't it ideally check if respective TF code is modified in PR , then only create terraform object. Like for me when we create a PR just to update a README.md for git repo( not managed by TF) , it still creates terraform object and continue to reconcile it

[chanwit]

similar problem happens when working with branch planner. since branch planner creates new flux - GitRepository whenever a PR is created and then creates terraform object. it happened once for me that when a PR is merged, seems it triggered deletion of GitRepository before deleting terraform object. Then terraform object deletion is struck due to this main issue.

@navpreet-securitas I spotted a bug there in the branch planner. Will take a look shortly.

@chanwit : another question related to branch planner. Shouldn't it ideally check if respective TF code is modified in PR , then only create terraform object. Like for me when we create a PR just to update a README.md for git repo( not managed by TF) , it still creates terraform object and continue to reconcile it

Yeah we're thinking about improving it towards this direction :+1:

[lasomethingsomething]

@chanwit @yitsushi Is https://github.com/weaveworks/tf-controller/issues/245 an aspect of this issue? If yes, let's close it and keep this one going.