GoogleCloudPlatform
Add acceptance tests for provider configuration behaviour using `user_project_override`
This PR adds acceptance tests that demonstrate:
- how the provider behaves when provider configuration arguments come from different sources ( config vs ENVs)
- schema-level validation that's in place, e.g. handling of empty strings
- use cases: how does this argument impact the providers behaviour in plan/apply
I've tried to clean up this PR and the self-review comments and other comments should be useful during review.
Release Note Template for Downstream PRs (will be copied)
Hi there, I'm the Modular magician. I've detected the following information about your changes:
Diff report
Your PR generated some diffs in downstreams - here they are.
google provider: Diff ( 3 files changed, 662 insertions(+), 158 deletions(-))
google-beta provider: Diff ( 3 files changed, 662 insertions(+), 158 deletions(-))
Tests analytics
Total tests: 3993 Passed tests: 3585 Skipped tests: 407 Affected tests: 1
Click here to see the affected service packages
All service packages are affected
Action taken
Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
- TestAccComputeInstance_confidentialHyperDiskBootDisk
Get to know how VCR tests work
$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccComputeInstance_confidentialHyperDiskBootDisk[Debug log]
$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$
$\textcolor{green}{\textsf{All tests passed!}}$
View the build log or the debug log for each test
Hi there, I'm the Modular magician. I've detected the following information about your changes:
Diff report
Your PR generated some diffs in downstreams - here they are.
google provider: Diff ( 3 files changed, 663 insertions(+), 158 deletions(-))
google-beta provider: Diff ( 4 files changed, 788 insertions(+), 158 deletions(-))
Here are some builds running acceptance tests on commit 60b55a6 :
- SDK : https://hashicorp.teamcity.com/buildConfiguration/TerraformProviders_GoogleCloud_GOOGLE_BETA_MMUPSTREAMTESTS_GOOGLEBETA_PACKAGE_PROVIDER/233834?mode=branches&buildTab=overview&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandBuildChangesSection=true
- Plugin Framework : https://hashicorp.teamcity.com/buildConfiguration/TerraformProviders_GoogleCloud_GOOGLE_BETA_MMUPSTREAMTESTS_GOOGLEBETA_PACKAGE_FWPROVIDER/233833?hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandBuildChangesSection=true
I'm expecting a failure in the PF test due to the intentional change in 60b55a6 that means the setup function doesn't make project-2 in the acc test a Firebase-enabled project. Using the resource google_firebase_project is insufficient for some reason.
If I reverse the changes of 60b55a6 the test passes (and google_firebase_project is a no-op in that context). I'll push a commit to do that soon.
Outcome from the builds above:
-
Identified https://github.com/hashicorp/terraform-provider-google/issues/19468
--- FAIL: TestAccFwProvider_user_project_override/user_project_override_controls_which_project_is_used_for_quota_and_billing_purposes (158.20s)
---
framework_provider_user_project_override_test.go:277: Step 2/3 error: Error running apply: exit status 1
Error: Error creating Project: googleapi: Error 403: com.google.apps.framework.auth.IamPermissionDeniedException: Permission denied to enable service [firebase.googleapis.com] {
Details: [
IAM{policy: 'serviceusage_services-/resourcemanager_projectnumbers/\/000000ef6c400524/services/firebase.googleapis.com' resource: 'projectnumbers/1234567890/services/firebase.googleapis.com' permission: 'serviceusage.services.enable'} denied {auditlog: false, cloudaudit: true}
Permission denied to enable service [firebase.googleapis.com] {
IAM{policy: 'serviceusage_services-/resourcemanager_projectnumbers/\/000000ef6c400524/services/firebase.googleapis.com' resource: 'projectnumbers/1234567890/services/firebase.googleapis.com' permission: 'serviceusage.services.enable'} denied {auditlog: false, cloudaudit: true}
}
]
} (Neither system permission tenantmanagement.write granted nor fine-grained check com.google.api.tenant.auth.PublicAnnotations$DefaultTenantManagerAuthChecker passed): securityContext=ValidatedSecurityContext{gaiauser/0x1}
with google_firebase_project.default,
on terraform_plugin_test.tf line 13, in resource "google_firebase_project" "default":
13: resource "google_firebase_project" "default" {
Tests analytics
Total tests: 4032 Passed tests: 3621 Skipped tests: 409 Affected tests: 2
Click here to see the affected service packages
All service packages are affected
Action taken
Found 2 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
- TestAccComputeInstance_confidentialHyperDiskBootDisk
- TestAccFrameworkProviderBasePath_setInvalidBasePath
Get to know how VCR tests work
$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccComputeInstance_confidentialHyperDiskBootDisk[Debug log]
$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$
$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccFrameworkProviderBasePath_setInvalidBasePath[Error message] [Debug log]
$\textcolor{red}{\textsf{Errors occurred during RECORDING mode. Please fix them to complete your PR.}}$
View the build log or the debug log for each test
Hi there, I'm the Modular magician. I've detected the following information about your changes:
Diff report
Your PR generated some diffs in downstreams - here they are.
google provider: Diff ( 3 files changed, 663 insertions(+), 158 deletions(-))
google-beta provider: Diff ( 4 files changed, 788 insertions(+), 158 deletions(-))
Tests analytics
Total tests: 4034 Passed tests: 3623 Skipped tests: 409 Affected tests: 2
Click here to see the affected service packages
All service packages are affected
Action taken
Found 2 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
- TestAccComputeInstance_confidentialHyperDiskBootDisk
- TestAccFrameworkProviderBasePath_setInvalidBasePath
Get to know how VCR tests work
$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccComputeInstance_confidentialHyperDiskBootDisk[Debug log]
$\textcolor{red}{\textsf{Tests failed when rerunning REPLAYING mode:}}$
TestAccComputeInstance_confidentialHyperDiskBootDisk[Error message] [Debug log]
Tests failed due to non-determinism or randomness when the VCR replayed the response after the HTTP request was made.
Please fix these to complete your PR. If you believe these test failures to be incorrect or unrelated to your change, or if you have any questions, please raise the concern with your reviewer.
$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccFrameworkProviderBasePath_setInvalidBasePath[Error message] [Debug log]
$\textcolor{red}{\textsf{Errors occurred during RECORDING mode. Please fix them to complete your PR.}}$
View the build log or the debug log for each test
New TeamCity builds using the c2938dd commit:
SDK: https://hashicorp.teamcity.com/buildConfiguration/TerraformProviders_GoogleCloud_GOOGLE_BETA_MMUPSTREAMTESTS_GOOGLEBETA_PACKAGE_PROVIDER/233840?hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandBuildChangesSection=true
Framework: https://hashicorp.teamcity.com/buildConfiguration/TerraformProviders_GoogleCloud_GOOGLE_BETA_MMUPSTREAMTESTS_GOOGLEBETA_PACKAGE_FWPROVIDER/233841?hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandBuildChangesSection=true
In these tests the google_firebase_project resource is present but it's not needed - it recognises that the project is already a Firebase project and just pulls it into state
Got some weird errors in https://github.com/GoogleCloudPlatform/magic-modules/pull/11686#issuecomment-2349495742 - both look like timing issues. One is in the setup function (SDK), the other is reporting the API isn't enabled (PF) - not sure if setup problem or test failure.
Edit: The tests passed after re-running 🤷 - I'll add a sleep to help avoid "If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry" and continue investigating
Hi there, I'm the Modular magician. I've detected the following information about your changes:
Diff report
Your PR generated some diffs in downstreams - here they are.
google provider: Diff ( 3 files changed, 663 insertions(+), 158 deletions(-))
google-beta provider: Diff ( 4 files changed, 788 insertions(+), 158 deletions(-))
Hi there, I'm the Modular magician. I've detected the following information about your changes:
Diff report
Your PR generated some diffs in downstreams - here they are.
google provider: Diff ( 4 files changed, 664 insertions(+), 158 deletions(-))
google-beta provider: Diff ( 4 files changed, 790 insertions(+), 158 deletions(-))
Tests analytics
Total tests: 4041 Passed tests: 3631 Skipped tests: 410 Affected tests: 0
Click here to see the affected service packages
All service packages are affected
$\textcolor{green}{\textsf{All tests passed!}}$
View the build log
Tests analytics
Total tests: 4041 Passed tests: 3631 Skipped tests: 410 Affected tests: 0
Click here to see the affected service packages
All service packages are affected
$\textcolor{green}{\textsf{All tests passed!}}$
View the build log
Hi there, I'm the Modular magician. I've detected the following information about your changes:
Diff report
Your PR generated some diffs in downstreams - here they are.
google provider: Diff ( 3 files changed, 663 insertions(+), 158 deletions(-))
google-beta provider: Diff ( 4 files changed, 769 insertions(+), 158 deletions(-))
Hi there, I'm the Modular magician. I've detected the following information about your changes:
Diff report
Your PR generated some diffs in downstreams - here they are.
google provider: Diff ( 4 files changed, 665 insertions(+), 160 deletions(-))
google-beta provider: Diff ( 4 files changed, 771 insertions(+), 160 deletions(-))
Tests analytics
Total tests: 4086 Passed tests: 3677 Skipped tests: 409 Affected tests: 0
Click here to see the affected service packages
All service packages are affected
$\textcolor{green}{\textsf{All tests passed!}}$
View the build log
The latest version of the code provides the required permissions for the TF config to manage Firebase resources. I'm seeing some issues when making the bootstrapped service accounts specifically when running the test in TeamCity/against the VCR testing GCP project and that code failing prevents debug logs being saved as artifacts.
I've updated the code so that the returned errors are as informative as possible, as that error is the only feedback that's visible (debug logs are lost). I've run the test a number of times in TeamCity and it's currently passing consistently (SDK, PF). I need a test failure to happen to inform any further improvements
Hi there, I'm the Modular magician. I've detected the following information about your changes:
Diff report
Your PR generated some diffs in downstreams - here they are.
google provider: Diff ( 4 files changed, 665 insertions(+), 160 deletions(-))
google-beta provider: Diff ( 4 files changed, 771 insertions(+), 160 deletions(-))
Tests analytics
Total tests: 4086 Passed tests: 3677 Skipped tests: 409 Affected tests: 0
Click here to see the affected service packages
All service packages are affected
$\textcolor{green}{\textsf{All tests passed!}}$
View the build log
Hi there, I'm the Modular magician. I've detected the following information about your changes:
Diff report
Your PR generated some diffs in downstreams - here they are.
google provider: Diff ( 4 files changed, 678 insertions(+), 173 deletions(-))
google-beta provider: Diff ( 4 files changed, 784 insertions(+), 173 deletions(-))
provider_user_project_override_test.go:275: error when setting up projects and retrieving access token: error setting IAM policy for 'project-2' with project id tf-test-kx8ehnrbfe-2: googleapi: Error 400: Service account tf-bootstrap-sa-pubsub@tf-test-kx8ehnrbfe.iam.gserviceaccount.com does not exist., badRequest
This is where the tests appear to be failing intermittently. Possible race between provisioning and using the service account?
Edit: Hopefully this has been addressed by 72bd475
Tests analytics
Total tests: 4086 Passed tests: 3676 Skipped tests: 409 Affected tests: 1
Click here to see the affected service packages
All service packages are affected
Action taken
Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
- TestAccNetworkConnectivitySpoke_networkConnectivitySpokeVpnTunnelBasicExample
Get to know how VCR tests work
$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccNetworkConnectivitySpoke_networkConnectivitySpokeVpnTunnelBasicExample[Debug log]
$\textcolor{red}{\textsf{Tests failed when rerunning REPLAYING mode:}}$
TestAccNetworkConnectivitySpoke_networkConnectivitySpokeVpnTunnelBasicExample[Error message] [Debug log]
Tests failed due to non-determinism or randomness when the VCR replayed the response after the HTTP request was made.
Please fix these to complete your PR. If you believe these test failures to be incorrect or unrelated to your change, or if you have any questions, please raise the concern with your reviewer.
$\textcolor{green}{\textsf{All tests passed!}}$
View the build log or the debug log for each test
Tests analytics
Total tests: 4086 Passed tests: 3676 Skipped tests: 409 Affected tests: 1
Click here to see the affected service packages
All service packages are affected
Action taken
Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
- TestAccNetworkConnectivitySpoke_networkConnectivitySpokeVpnTunnelBasicExample
Get to know how VCR tests work
$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccNetworkConnectivitySpoke_networkConnectivitySpokeVpnTunnelBasicExample[Debug log]
$\textcolor{red}{\textsf{Tests failed when rerunning REPLAYING mode:}}$
TestAccNetworkConnectivitySpoke_networkConnectivitySpokeVpnTunnelBasicExample[Error message] [Debug log]
Tests failed due to non-determinism or randomness when the VCR replayed the response after the HTTP request was made.
Please fix these to complete your PR. If you believe these test failures to be incorrect or unrelated to your change, or if you have any questions, please raise the concern with your reviewer.
$\textcolor{green}{\textsf{All tests passed!}}$
View the build log or the debug log for each test
Hi there, I'm the Modular magician. I've detected the following information about your changes:
Diff report
Your PR generated some diffs in downstreams - here they are.
google provider: Diff ( 4 files changed, 681 insertions(+), 173 deletions(-))
google-beta provider: Diff ( 4 files changed, 787 insertions(+), 173 deletions(-))