1-org failed Step #1 - "tf plan validate all": ERROR: (gcloud.beta.terraform.vet)

[Atul7696]

TL;DR

Terraform plan fails at step 16 for 1-org. Have used below code in apply and plan yamls.

logsBucket: '${_LOGS_BUCKET_NAME}' options: logging: GCS_ONLY

Expected behavior

It should succeed and terraform plan should be successfully created for the branch not listed in the yamls/tfwrapper script.

Observed behavior

Below is the error received

Step #1 - "tf plan validate all": + parent_resource_id = "XXXXXX78673" Step #1 - "tf plan validate all": + parent_resource_type = "organization" Step #1 - "tf plan validate all": + restricted_net_hub_project_id = (known after apply) Step #1 - "tf plan validate all": + scc_notification_name = "scc-notify" Step #1 - "tf plan validate all": + scc_notifications_project_id = (known after apply) Step #1 - "tf plan validate all": Step #1 - "tf plan validate all": Warning: Value for undeclared variable Step #1 - "tf plan validate all": Step #1 - "tf plan validate all": The root module does not declare a variable named Step #1 - "tf plan validate all": "networks_step_terraform_service_account" but a value was found in file Step #1 - "tf plan validate all": "terraform.tfvars". To use this value, add a "variable" block to the Step #1 - "tf plan validate all": configuration. Step #1 - "tf plan validate all": Step #1 - "tf plan validate all": Using a variables file to set an undeclared variable is deprecated and will Step #1 - "tf plan validate all": become an error in a future release. If you wish to provide certain "global" Step #1 - "tf plan validate all": settings to all configurations in your organization, use TF_VAR_... Step #1 - "tf plan validate all": environment variables to set these instead. Step #1 - "tf plan validate all": Step #1 - "tf plan validate all": Step #1 - "tf plan validate all": ------------------------------------------------------------------------ Step #1 - "tf plan validate all": Step #1 - "tf plan validate all": This plan was saved to: /workspace/tmp_plan/envs-shared.tfplan Step #1 - "tf plan validate all": Step #1 - "tf plan validate all": To perform exactly these actions, run the following command to apply: Step #1 - "tf plan validate all": terraform apply "/workspace/tmp_plan/envs-shared.tfplan" Step #1 - "tf plan validate all": Step #1 - "tf plan validate all": *************** TERRAFORM VALIDATE ****************** Step #1 - "tf plan validate all": At environment: envs/shared Step #1 - "tf plan validate all": Using policy from: /workspace/policy-library Step #1 - "tf plan validate all": ***************************************************** Step #1 - "tf plan validate all": WARNING: This command is using service account impersonation. All API calls will be executed as [[email protected]]. Step #1 - "tf plan validate all": WARNING: This command is using service account impersonation. All API calls will be executed as [[email protected]]. Step #1 - "tf plan validate all": WARNING: This command is using service account impersonation. All API calls will be executed as [[email protected]]. Step #1 - "tf plan validate all": Cloning into '/workspace/policy-library'... Step #1 - "tf plan validate all": WARNING: This command is using service account impersonation. All API calls will be executed as [[email protected]]. Step #1 - "tf plan validate all": warning: remote HEAD refers to nonexistent ref, unable to checkout. Step #1 - "tf plan validate all": Step #1 - "tf plan validate all": Project [prj-b-cicd-e2e1] repository [gcp-policies] was cloned to [/workspace/policy-library]. Step #1 - "tf plan validate all": /workspace/envs/shared /workspace/envs/shared Step #1 - "tf plan validate all": Branch 'main' set up to track remote branch 'main' from 'origin'. Step #1 - "tf plan validate all": Switched to a new branch 'main' Step #1 - "tf plan validate all": /workspace/envs/shared Step #1 - "tf plan validate all": Pausing command execution: Step #1 - "tf plan validate all": Step #1 - "tf plan validate all": This command requires the terraform-tools component to be installed. Would you Step #1 - "tf plan validate all": like to install the terraform-tools component to continue command execution? Step #1 - "tf plan validate all": (Y/n)?
Step #1 - "tf plan validate all": ERROR: (gcloud.beta.terraform.vet) Step #1 - "tf plan validate all": You cannot perform this action because the Google Cloud CLI component manager Step #1 - "tf plan validate all": is disabled for this installation. You can run the following command Step #1 - "tf plan validate all": to achieve the same result for this installation: Step #1 - "tf plan validate all": Step #1 - "tf plan validate all": sudo apt-get install google-cloud-sdk-terraform-tools Step #1 - "tf plan validate all": Step #1 - "tf plan validate all": Finished Step #1 - "tf plan validate all" ERROR ERROR: build step 1 "us-east4-docker.pkg.dev/prj-b-cicd-/prj-tf-runners/terraform" failed: step exited with non-zero status: 33

Terraform Configuration

NA

Terraform Version

Version 0.13.7

Additional information

https://github.com/terraform-google-modules/terraform-example-foundation/issues/775 fyip @hadi-alnehlawi Same error experienced in the above thread.

Trying with org-terraform@prj-b-seed service account with hub and spoke enabled and also using network terraform sa in the terraform.tfvars.

[Atul7696]

commented line tf_validate "$env_path" "$env" "$policysource" "$component" Failed at gcs bucket

Step https://github.com/terraform-google-modules/terraform-example-foundation/pull/1 - "tf plan validate all": Step https://github.com/terraform-google-modules/terraform-example-foundation/pull/1 - "tf plan validate all": This plan was saved to: /workspace/tmp_plan/envs-shared.tfplan Step https://github.com/terraform-google-modules/terraform-example-foundation/pull/1 - "tf plan validate all": Step https://github.com/terraform-google-modules/terraform-example-foundation/pull/1 - "tf plan validate all": To perform exactly these actions, run the following command to apply: Step https://github.com/terraform-google-modules/terraform-example-foundation/pull/1 - "tf plan validate all": terraform apply "/workspace/tmp_plan/envs-shared.tfplan" Step https://github.com/terraform-google-modules/terraform-example-foundation/pull/1 - "tf plan validate all": Step https://github.com/terraform-google-modules/terraform-example-foundation/pull/1 - "tf plan validate all": .git/info doesn't match ^(development|non-production|production|shared)$; skipping Step https://github.com/terraform-google-modules/terraform-example-foundation/pull/1 - "tf plan validate all": .git/branches doesn't match ^(development|non-production|production|shared)$; skipping Step https://github.com/terraform-google-modules/terraform-example-foundation/pull/1 - "tf plan validate all": .git/refs doesn't match ^(development|non-production|production|shared)$; skipping Step https://github.com/terraform-google-modules/terraform-example-foundation/pull/1 - "tf plan validate all": .git/logs doesn't match ^(development|non-production|production|shared)$; skipping Step https://github.com/terraform-google-modules/terraform-example-foundation/pull/1 - "tf plan validate all": .git/hooks doesn't match ^(development|non-production|production|shared)$; skipping Step https://github.com/terraform-google-modules/terraform-example-foundation/pull/1 - "tf plan validate all": .git/objects doesn't match ^(development|non-production|production|shared)$; skipping Finished Step https://github.com/terraform-google-modules/terraform-example-foundation/pull/1 - "tf plan validate all" PUSH ERROR ERROR: bucket "gs://prj-cloudbuild-artifacts-a35a" does not exist

[Atul7696]

Able to pass the stage by doing below steps

1. COmmented below lines in terraform.tfvars function tf_validate()

gcloud beta terraform vet "${tf_env}.json" --policy-library="${policy_file_path}" --project="${project_id}" || exit 33

1. Providing cloud build ci cd project service account [email protected] as storage admin privileges on remote terraform state bucket.
2. In the terraform.tfvars used the organization_step_terrafpr,_service_account_email.
3. Added source repository writer permissions to terraform-org-sa account.
4. Added the variable block of networks_step_terraform_service_account in variables.tf as in terraform.tfvars enabled hub and spoke model and also added the reference for the network step sa.
5. Then triggered the build.
6. Fails at the network service account and refers to invalid service account.

Starting Step #4 - "tf apply" Step #4 - "tf apply": Already have image (with digest): us-east4-docker.pkg.dev/prj-b-cicd-e2e1/prj-tf-runners/terraform Step #4 - "tf apply": policies doesn't match production; skipping Step #4 - "tf apply": lib doesn't match production; skipping Step #4 - "tf apply": .git doesn't match production; skipping Step #4 - "tf apply": *************** TERRAFORM APPLY ******************* Step #4 - "tf apply": At environment: envs/shared Step #4 - "tf apply": *************************************************** Step #4 - "tf apply": [0m[1mgoogle_scc_notification_config.scc_notification_config: Creating...[0m[0m Step #4 - "tf apply": [0m[1mgoogle_project_iam_member.network_sa_base["roles/compute.instanceAdmin"]: Creating...[0m[0m Step #4 - "tf apply": [0m[1mgoogle_project_iam_member.network_sa_base["roles/resourcemanager.projectIamAdmin"]: Creating...[0m[0m Step #4 - "tf apply": [0m[1mgoogle_project_iam_member.network_sa_restricted["roles/resourcemanager.projectIamAdmin"]: Creating...[0m[0m Step #4 - "tf apply": [0m[1mgoogle_project_iam_member.network_sa_restricted["roles/compute.instanceAdmin"]: Creating...[0m[0m Step #4 - "tf apply": [0m[1mgoogle_project_iam_member.network_sa_restricted["roles/iam.serviceAccountAdmin"]: Creating...[0m[0m Step #4 - "tf apply": [0m[1mgoogle_project_iam_member.network_sa_base["roles/iam.serviceAccountAdmin"]: Creating...[0m[0m Step #4 - "tf apply": [0m[1mgoogle_project_iam_member.network_sa_base["roles/iam.serviceAccountUser"]: Creating...[0m[0m Step #4 - "tf apply": [0m[1mgoogle_project_iam_member.network_sa_restricted["roles/iam.serviceAccountUser"]: Creating...[0m[0m Step #4 - "tf apply": [31m Step #4 - "tf apply": [1m[31mError: [0m[0m[1mRequest Create IAM Members roles/iam.serviceAccountAdmin serviceAccount: for project "prj-c-base-net-hub-7d32" returned error: Batch request and retried single request "Create IAM Members roles/iam.serviceAccountAdmin serviceAccount: for project "prj-c-base-net-hub-7d32"" both failed. Final error: Error applying IAM policy for project "prj-c-base-net-hub-7d32": Error setting IAM policy for project "prj-c-base-net-hub-7d32": googleapi: Error 400: Invalid service account ()., badRequest[0m Step #4 - "tf apply": Step #4 - "tf apply": [0m on projects.tf line 243, in resource "google_project_iam_member" "network_sa_base": Step #4 - "tf apply": 243: resource "google_project_iam_member" "network_sa_base" [4m{[0m Step #4 - "tf apply": [0m Step #4 - "tf apply": [0m[0m Step #4 - "tf apply": [31m Step #4 - "tf apply": [1m[31mError: [0m[0m[1mRequest Create IAM Members roles/resourcemanager.projectIamAdmin serviceAccount: for project "prj-c-base-net-hub-7d32" returned error: Batch request and retried single request "Create IAM Members roles/resourcemanager.projectIamAdmin serviceAccount: for project "prj-c-base-net-hub-7d32"" both failed. Final error: Error applying IAM policy for project "prj-c-base-net-hub-7d32": Error setting IAM policy for project "prj-c-base-net-hub-7d32": googleapi: Error 400: Invalid service account ()., badRequest[0m Step #4 - "tf apply": Step #4 - "tf apply": [0m on projects.tf line 243, in resource "google_project_iam_member" "network_sa_base": Step #4 - "tf apply": 243: resource "google_project_iam_member" "network_sa_base" [4m{[0m Step #4 - "tf apply": [0m Step #4 - "tf apply": [0m[0m Step #4 - "tf apply": [31m Step #4 - "tf apply": [1m[31mError: [0m[0m[1mRequest Create IAM Members roles/iam.serviceAccountUser serviceAccount: for project "prj-c-base-net-hub-7d32" returned error: Batch request and retried single request "Create IAM Members roles/iam.serviceAccountUser serviceAccount: for project "prj-c-base-net-hub-7d32"" both failed. Final error: Error applying IAM policy for project "prj-c-base-net-hub-7d32": Error setting IAM policy for project "prj-c-base-net-hub-7d32": googleapi: Error 400: Invalid service account ()., badRequest[0m Step #4 - "tf apply": Step #4 - "tf apply": [0m on projects.tf line 243, in resource "google_project_iam_member" "network_sa_base": Step #4 - "tf apply": 243: resource "google_project_iam_member" "network_sa_base" [4m{[0m Step #4 - "tf apply": [0m Step #4 - "tf apply": [0m[0m Step #4 - "tf apply": [31m Step #4 - "tf apply": [1m[31mError: [0m[0m[1mRequest Create IAM Members roles/compute.instanceAdmin serviceAccount: for project "prj-c-base-net-hub-7d32" returned error: Batch request and retried single request "Create IAM Members roles/compute.instanceAdmin serviceAccount: for project "prj-c-base-net-hub-7d32"" both failed. Final error: Error applying IAM policy for project "prj-c-base-net-hub-7d32": Error setting IAM policy for project "prj-c-base-net-hub-7d32": googleapi: Error 400: Invalid service account ()., badRequest[0m Step #4 - "tf apply": Step #4 - "tf apply": [0m on projects.tf line 243, in resource "google_project_iam_member" "network_sa_base": Step #4 - "tf apply": 243: resource "google_project_iam_member" "network_sa_base" [4m{[0m Step #4 - "tf apply": [0m Step #4 - "tf apply": [0m[0m Step #4 - "tf apply": [31m Step #4 - "tf apply": [1m[31mError: [0m[0m[1mRequest Create IAM Members roles/compute.instanceAdmin serviceAccount: for project "prj-c-restricted-net-hub-3398" returned error: Batch request and retried single request "Create IAM Members roles/compute.instanceAdmin serviceAccount: for project "prj-c-restricted-net-hub-3398"" both failed. Final error: Error applying IAM policy for project "prj-c-restricted-net-hub-3398": Error setting IAM policy for project "prj-c-restricted-net-hub-3398": googleapi: Error 400: Invalid service account ()., badRequest[0m Step #4 - "tf apply": Step #4 - "tf apply": [0m on projects.tf line 289, in resource "google_project_iam_member" "network_sa_restricted": Step #4 - "tf apply": 289: resource "google_project_iam_member" "network_sa_restricted" [4m{[0m Step #4 - "tf apply": [0m Step #4 - "tf apply": [0m[0m Step #4 - "tf apply": [31m Step #4 - "tf apply": [1m[31mError: [0m[0m[1mRequest Create IAM Members roles/iam.serviceAccountAdmin serviceAccount: for project "prj-c-restricted-net-hub-3398" returned error: Batch request and retried single request "Create IAM Members roles/iam.serviceAccountAdmin serviceAccount: for project "prj-c-restricted-net-hub-3398"" both failed. Final error: Error applying IAM policy for project "prj-c-restricted-net-hub-3398": Error setting IAM policy for project "prj-c-restricted-net-hub-3398": googleapi: Error 400: Invalid service account ()., badRequest[0m Step #4 - "tf apply": Step #4 - "tf apply": [0m on projects.tf line 289, in resource "google_project_iam_member" "network_sa_restricted": Step #4 - "tf apply": 289: resource "google_project_iam_member" "network_sa_restricted" [4m{[0m Step #4 - "tf apply": [0m Step #4 - "tf apply": [0m[0m Step #4 - "tf apply": [31m Step #4 - "tf apply": [1m[31mError: [0m[0m[1mRequest Create IAM Members roles/resourcemanager.projectIamAdmin serviceAccount: for project "prj-c-restricted-net-hub-3398" returned error: Batch request and retried single request "Create IAM Members roles/resourcemanager.projectIamAdmin serviceAccount: for project "prj-c-restricted-net-hub-3398"" both failed. Final error: Error applying IAM policy for project "prj-c-restricted-net-hub-3398": Error setting IAM policy for project "prj-c-restricted-net-hub-3398": googleapi: Error 400: Invalid service account ()., badRequest[0m Step #4 - "tf apply": Step #4 - "tf apply": [0m on projects.tf line 289, in resource "google_project_iam_member" "network_sa_restricted": Step #4 - "tf apply": 289: resource "google_project_iam_member" "network_sa_restricted" [4m{[0m Step #4 - "tf apply": [0m Step #4 - "tf apply": [0m[0m Step #4 - "tf apply": [31m Step #4 - "tf apply": [1m[31mError: [0m[0m[1mRequest Create IAM Members roles/iam.serviceAccountUser serviceAccount: for project "prj-c-restricted-net-hub-3398" returned error: Batch request and retried single request "Create IAM Members roles/iam.serviceAccountUser serviceAccount: for project "prj-c-restricted-net-hub-3398"" both failed. Final error: Error applying IAM policy for project "prj-c-restricted-net-hub-3398": Error setting IAM policy for project "prj-c-restricted-net-hub-3398": googleapi: Error 400: Invalid service account ()., badRequest[0m Step #4 - "tf apply": Step #4 - "tf apply": [0m on projects.tf line 289, in resource "google_project_iam_member" "network_sa_restricted": Step #4 - "tf apply": 289: resource "google_project_iam_member" "network_sa_restricted" [4m{[0m Step #4 - "tf apply": [0m Step #4 - "tf apply": [0m[0m Step #4 - "tf apply": [31m Step #4 - "tf apply": [1m[31mError: [0m[0m[1mError creating NotificationConfig: googleapi: Error 400: Security Command Center Legacy has been permanently disabled as of June 7, 2021. Migrate to Security Command Center's Standard tier or Premium tier to maintain access to Security Command Center. See https://cloud.google.com/security-command-center/docs/quickstart-security-command-center for more info.[0m Step #4 - "tf apply": Step #4 - "tf apply": [0m on scc_notification.tf line 32, in resource "google_scc_notification_config" "scc_notification_config": Step #4 - "tf apply": 32: resource "google_scc_notification_config" "scc_notification_config" [4m{[0m Step #4 - "tf apply": [0m Step #4 - "tf apply": [0m[0m Finished Step #4 - "tf apply" ERROR ERROR: build step 4 "us-east4-docker.pkg.dev/prj-b-cicd-e2e1/prj-tf-runners/terraform" failed: step exited with non-zero status: 1

[Atul7696]

Added below in terraform.tfvars

networks_step_terraform_service_account_email = "[email protected]" networks_step_terraform_service_account = "[email protected]"

It fixed the network related service account errors.

For SCC, enabled the scc at org level. Then it was successful

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days