[UX] Project ID collision handling

[matt-declarativecloud]

trafficstars

Deploying with Cloud Build

Issue

When Project creation fails with Error 409: Requested entity already exists, alreadyExists the randomized Project ID is written to .tfstate in "provider[\"registry.terraform.io/hashicorp/random\"]" and even though Project creation fails, the Project ID is recorded in .tfstate outputs

Workaround

To resolve the Project ID collision the .tfstate must be modified to remove the object created by the hashicorp/random provider

Enhancement ideas

• Project outputs should only be written when Project Resource creation succeeds
• It would be great if the hashicorp/random resource could be tainted in this scenario so subsequent apply operations generate new random Project IDs

[morgante]

Unfortunately there isn't any way for us to do this. It's core Terraform behavior.

[bharathkkb]

reopening to include some sample commands on how to taint and recover in the troubleshooting docs

[mark1000]

Trying to understand what needs to be documented. Is it the case that if the user gets the listed error message, they should delete the project name from .tfstate and try running terraform again?

[matt-declarativecloud]

IMO instruction should be given to temporarily

• connect to remote state
• taint the resource as described in 494

Tension is inherent between the goal of having a black box GitOps interface for customers that doesn't require Terraform expertise, and the likelihood that at some point an escape hatch will be required. For cases like this it will be useful to explain how to temporarily break out of the GitOps workflow and wire up Terraform to remote state for targeted, resource-level operations.

[M88YEH]

Any update on this? Whilst rehearsing the deployment of landing zones with the CFT - my team has experienced this issue multiple times. Ideally the modules building out projects, would be updated to use a larger range of "random Id's" but i realise there are many different modules which would need the same issue remediation.

Could all the layers instead have an extra random id inserted into the project name template (before the module's are called?)

for example: prj-c-dns-hub-c98c becomes prj-c-dns-hub-7bfe-c98c

This should then hopefully avoid the opportunity for the same "unique" project name to be generated twice, obviously there is a probability but a much lesser one than currently

[g-awmalik]

@bharathkkb - are you still planning to work on this?

[bharathkkb]

@daniel-cit once https://github.com/terraform-google-modules/terraform-google-project-factory/pull/742 lands lets update project factory everywhere to use new approach added in https://github.com/terraform-google-modules/terraform-google-project-factory/pull/735 which should reduce this from happening

[stevena-cloudsec]

@bharathkkb - Is this work still happening? Thanks!