magic-modules
magic-modules copied to clipboard
GoogleCloudPlatform
add `dataGovernanceType` field to `bigquery_routine` resource
Fixes https://github.com/hashicorp/terraform-provider-google/issues/17061
Release Note Template for Downstream PRs (will be copied)
bigquery: added `dataGovernanceType` to `google_bigquery_routine` resource
Tests analytics
Total tests: 107
Passed tests 99
Skipped tests: 8
Affected tests: 0
Click here to see the affected service packages
- bigquery
$\textcolor{green}{\textsf{All tests passed in REPLAYING mode.}}$ View the build log
Hi there, I'm the Modular magician. I've detected the following information about your changes:
Diff report
Your PR generated some diffs in downstreams - here they are.
Terraform GA: Diff ( 3 files changed, 97 insertions(+)) Terraform Beta: Diff ( 3 files changed, 97 insertions(+)) TF Conversion: Diff ( 1 file changed, 10 insertions(+)) TF OiCS: Diff ( 4 files changed, 113 insertions(+))
Tests analytics
Total tests: 108
Passed tests 99
Skipped tests: 8
Affected tests: 1
Click here to see the affected service packages
- bigquery
Action taken
Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccBigQueryRoutine_bigQueryRoutineDataGovernanceTypeExample
Get to know how VCR tests work
$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccBigQueryRoutine_bigQueryRoutineDataGovernanceTypeExample[Error message] [Debug log]
$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$ View the build log or the debug log for each test
Error: Error creating Routine: googleapi: Error 400: Unknown option: data_governance_type, invalidQuery
weird error since its stating that data_governance_type doesn't exist as a type but the docs shows it as a valid entry for OPTIONS
CREATE OR REPLACE FUNCTION SSN_Mask(ssn STRING) RETURNS STRING OPTIONS (data_governance_type="DATA_MASKING") AS ( SAFE.REGEXP_REPLACE(ssn, '[0-9]', 'X') # 123-45-6789 -> XXX-XX-XXXX );
I believe the feature is in the Preview launch stage (see "In Preview." at https://cloud.google.com/bigquery/docs/column-data-masking-intro#masking_options) and may need an explicit opt-in for the GCP project. @BBBmau were you able to get it to work with DDL in your project and what did you do to get on an allowlist?
The custom data masking feature was made generally available (GA) on 2/8: https://cloud.google.com/bigquery/docs/release-notes#February_08_2024. Let's rerun the tests to see if the API 400 error is no longer applicable.
The custom data masking feature was made generally available (GA) on 2/8: https://cloud.google.com/bigquery/docs/release-notes#February_08_2024. Let's rerun the tests to see if the API 400 error is no longer applicable.
@BBBmau This PR could use a merge from main. Do you still have some bandwidth to work on this?
Hi there, I'm the Modular magician. I've detected the following information about your changes:
Diff report
Your PR generated some diffs in downstreams - here they are.
Terraform GA: Diff ( 3 files changed, 97 insertions(+)) Terraform Beta: Diff ( 3 files changed, 97 insertions(+)) TF Conversion: Diff ( 1 file changed, 10 insertions(+)) TF OiCS: Diff ( 4 files changed, 113 insertions(+))
Hi there, I'm the Modular magician. I've detected the following information about your changes:
Diff report
Your PR generated some diffs in downstreams - here they are.
Terraform GA: Diff ( 3 files changed, 97 insertions(+)) Terraform Beta: Diff ( 3 files changed, 97 insertions(+)) TF Conversion: Diff ( 1 file changed, 10 insertions(+)) TF OiCS: Diff ( 4 files changed, 113 insertions(+))
Tests analytics
Total tests: 110
Passed tests: 101
Skipped tests: 8
Affected tests: 1
Click here to see the affected service packages
- bigquery
Action taken
Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccBigQueryRoutine_bigQueryRoutineDataGovernanceTypeExample
Get to know how VCR tests work
$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccBigQueryRoutine_bigQueryRoutineDataGovernanceTypeExample[Error message] [Debug log]
$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$ View the build log or the debug log for each test
Tests analytics
Total tests: 110
Passed tests: 101
Skipped tests: 8
Affected tests: 1
Click here to see the affected service packages
- bigquery
Action taken
Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccBigQueryRoutine_bigQueryRoutineDataGovernanceTypeExample
Get to know how VCR tests work
$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccBigQueryRoutine_bigQueryRoutineDataGovernanceTypeExample[Error message] [Debug log]
$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$ View the build log or the debug log for each test
Error: Error creating Routine: googleapi: Error 400: Project ID, dataset ID and routine ID cannot be missing or empty in call to routines.insert, invalid
with google_bigquery_routine.sproc,
on terraform_plugin_test.tf line 6, in resource "google_bigquery_routine" "sproc":
6: resource "google_bigquery_routine" "sproc" {
```
Error: Error creating Routine: googleapi: Error 400: Project ID, dataset ID and routine ID cannot be missing or empty in call to routines.insert, invalid with google_bigquery_routine.sproc, on terraform_plugin_test.tf line 6, in resource "google_bigquery_routine" "sproc": 6: resource "google_bigquery_routine" "sproc" { ```
From the debug logs:
---[ REQUEST ]---------------------------------------
POST /bigquery/v2/projects/<project-id>/datasets/tf_test_dataset_id0c7gks16bz/routines?alt=json HTTP/1.1
Host: bigquery.googleapis.com
User-Agent: Terraform/1.2.5 (+https://www.terraform.io) Terraform-Plugin-SDK/2.31.0 terraform-provider-google-beta/acc
Content-Length: 313
Content-Type: application/json
Accept-Encoding: gzip
{
"dataGovernanceType": "DATA_MASKING",
"definitionBody": "CREATE FUNCTION SSN_Mask(ssn STRING) RETURNS STRING AS (SAFE.REGEXP_REPLACE(ssn, '[0-9]', 'x'));",
"language": "SQL",
"routineReference": {
"datasetId": "tf_test_dataset_id0c7gks16bz",
"projectId": "<project-id>",
"routineId": ""
},
"routineType": "PROCEDURE"
}
routineId was empty.
I added a comment to the PR which should help resolve this.
Hi there, I'm the Modular magician. I've detected the following information about your changes:
Diff report
Your PR generated some diffs in downstreams - here they are.
Terraform GA: Diff ( 3 files changed, 97 insertions(+)) Terraform Beta: Diff ( 3 files changed, 97 insertions(+)) TF Conversion: Diff ( 1 file changed, 10 insertions(+)) TF OiCS: Diff ( 4 files changed, 113 insertions(+))
Tests analytics
Total tests: 110
Passed tests: 101
Skipped tests: 8
Affected tests: 1
Click here to see the affected service packages
- bigquery
Action taken
Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccBigQueryRoutine_bigqueryRoutineDataGovernanceTypeExample
Get to know how VCR tests work
$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccBigQueryRoutine_bigqueryRoutineDataGovernanceTypeExample[Error message] [Debug log]
$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$ View the build log or the debug log for each test
back to the original error
The code change looks good. I will circle back after reach out to the BQ Routines team on the NotFound error.
back to the original error
The code change looks good. I will circle back after reach out to the BQ Routines team on the NotFound error.
I got this guidance: "The data_masking_type is only supported for scalar functions. The request has a procedure."
Could you try with a config with routine_type = "SCALAR_FUNCTION" and a simple scalar function?
@wj-chen attempted several different functions but an unexpected keyword error would appear:
=== CONT TestAccBigQueryRoutine_bigqueryRoutineDataGovernanceTypeExample
vcr_utils.go:152: Step 1/2 error: Error running apply: exit status 1
Error: Error creating Routine: googleapi: Error 400: Syntax error: Unexpected keyword CREATE at [1:1], invalidQuery
with google_bigquery_routine.sproc,
on terraform_plugin_test.tf line 6, in resource "google_bigquery_routine" "sproc":
6: resource "google_bigquery_routine" "sproc" {
--- FAIL: TestAccBigQueryRoutine_bigqueryRoutineDataGovernanceTypeExample (7.76s)