Updating a BigQuery View with a Schema Description Drops the Descriptions

[Ryan-Lintern]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.13.5 registry.terraform.io/hashicorp/google v4.6.0

Affected Resource(s)

• google_bigquery_table

Terraform Configuration Files

provider "google" {
  version     = "4.6.0"
  credentials = <REDACTED>
  region      = <REDACTED>
  zone        = <REDACTED>
}

variable "test_views_list" {
  type = list(object({
    name          = string
    description   = string
    schema        = string
    template_file = string
    table         = string
    table_suffix  = string
  }))
  default = [
    {
      name          = "TestView"
      description   = "Test View 1"
      schema        = "PATH/TO/SCHEMA/FILE.json"
      template_file = "PATH/TO/SQL/FILE.sql"
      table         = "TestTable*"
      table_suffix  = "'1','2'"
    },
    {
      name          = "TestView1"
      description   = "Test View 2"
      schema        = "PATH/TO/SCHEMA/FILE.json"
      template_file = "PATH/TO/SQL/FILE.sql"
      table         = "TestTable*"
      table_suffix  = "'1'"
    },
    {
      name          = "TestView2"
      description   = "Test View 2"
      schema        = "PATH/TO/SCHEMA/FILE.json"
      template_file = "PATH/TO/SQL/FILE.sql"
      table         = "TestTable*"
      table_suffix  = "'2'"
    }
  ]
}

resource "google_bigquery_table" "test_views" {
  for_each = { for test_view in var.test_views_list :
    test_view.name => test_view
  }
  project     = 'test_project_2_name'
  dataset_id  = 'test_dataset_2_id'
  table_id    = each.value.name
  description = each.value.description
  labels      = {}
  schema      = file(each.value.schema)
  view {
    query = templatefile(
      each.value.template_file,
      {
        project      = 'test_project_name'
        dataset      = 'test_dataset_name'
        table        = each.value.table
        table_suffix = each.value.table_suffix
      }
    )
    use_legacy_sql = false
  }
  deletion_protection = false
}

Debug Output

N/A

Panic Output

N/A

Expected Behavior

Updating a BigQuery view containing schema descriptions, where the descriptions aren't being updated themselves, should apply those updates without dropping the descriptions

Actual Behavior

Updating the view results in the schema descriptions being dropped

Steps to Reproduce

1. Make change to resource containing view schema descriptions - for example, change the view description and leave the schema as is.
2. terraform apply

Important Factoids

N/A

References

[Ryan-Lintern]

This behaviour seems linked to the following issue and Pull Request:

• https://github.com/hashicorp/terraform-provider-google/issues/9096
• https://github.com/hashicorp/terraform-provider-google/pull/10819

When updating the BigQuery View it appears that, when a schema is provided, this should possibly follow the same logic as the BigQueryTableCreate logic - i.e. update, then re-apply schema.

[edwardmedia]

@Ryan-Lintern do you have a config and the steps that can repro the issue?

[Ryan-Lintern]

I've updated the initial message to include an example config. For reproducing this issue, I've also provided an example - all it takes is a change to the View resource, whether that's changing the description, or changing the SQL code that builds the view. One of the instances where we hit this issue was when running the apply on different systems (Windows vs Linux). In these instances, the view code was seen to be changing, even though we hadn't altered it, and as a result it applied a change to the view, which refreshed the view and dropped the schema descriptions.

[Ryan-Lintern]

Reading up on the previous issue(s) and reviewing the code, it appears like the issue is that when the view is being updated, it needs to submit the schema in a separate API call, similar to how it is now done in the BigQueryCreateTable function, likely due to the update refreshing the view.

It should also be noted that if all that has changed is the schema itself, then Terraform happily applies it, as it's the one Update call, providing the schema change

[edwardmedia]

@Ryan-Lintern can you share both FILE.json and FILE.sql?

      schema        = "PATH/TO/SCHEMA/FILE.json"
      template_file = "PATH/TO/SQL/FILE.sql"

[Ryan-Lintern]

@edwardmedia I've updated the above to also include an example of our schema and view code files. There will be varying degrees of complexity in the files (beyond just a standard select statement), but the same approach is used for how we store those files

[edwardmedia]

@Ryan-Lintern can't repro your issue. I have updated the view schema in a description, and I see the new description in the result. Can you specify what exact field(s) you updated in the steps? Also please share the debug log? I probably still not get what problem you refer to.

BTW, your code still isn't working. Can you remove all dynamic code and make the simple config that can easily repro the issue?

[Ryan-Lintern]

@edwardmedia the main instance of seeing this issue involved a change to another part of the view, whether that was alongside the schema change, or even when the schema was never changed. In one example, the view code was changed - it wasn't actually, it was just the differences between Windows and Linux, but it was enough for Terraform to want to reapply the view resource. You should be able to replicate it by making a change to the view code.

I will also attempt to replicate the issue my end tomorrow, and will share what I'm able to of the debug logs.

[edwardmedia]

@Ryan-Lintern wait for more info & debug logs

[edwardmedia]

@Ryan-Lintern should I leave this issue open?

[edwardmedia]

Closing this assuming it is no longer an issue. @Ryan-Lintern please reopen once you have more info & logs to share. Thanks

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.