google_compute_backend_bucket always passes TTLs to GCP

[d10i]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

$ terraform -v
Terraform v1.0.11
on linux_amd64
+ provider registry.terraform.io/alexkappa/auth0 v0.24.0
+ provider registry.terraform.io/hashicorp/google v4.1.0

Affected Resource(s)

• google_compute_backend_bucket

Terraform Configuration Files

resource "google_compute_backend_bucket" "default" {
  project = "..."

  name        = "..."
  bucket_name = "..."

  enable_cdn = true
  cdn_policy {
    cache_mode        = "USE_ORIGIN_HEADERS"
    negative_caching  = true
    serve_while_stale = 86400
  }
}

Debug Output

(partially redacted with [...])

---[ REQUEST ]---------------------------------------
POST /compute/v1/projects/shoji-staging/global/backendBuckets?alt=json HTTP/1.1
Host: compute.googleapis.com
User-Agent: Terraform/1.0.11 (+https://www.terraform.io) Terraform-Plugin-SDK/2.5.0 terraform-provider-google/4.1.0
Content-Length: 255
Content-Type: application/json
Accept-Encoding: gzip

{
 "bucketName": "...",
 "cdnPolicy": {
  "cacheMode": "USE_ORIGIN_HEADERS",
  "clientTtl": 0,
  "defaultTtl": 0,
  "maxTtl": 0,
  "negativeCaching": true,
  "serveWhileStale": 86400,
  "signedUrlCacheMaxAgeSec": 0
 },
 "enableCdn": true,
 "name": "..."
}

[...]

Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 400 Bad Request
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Tue, 23 Nov 2021 11:49:36 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
  "error": {
    "code": 400,
    "message": "Invalid value for field 'resource.cdnPolicy.defaultTtl': '0'. default_ttl cannot be specified with USE_ORIGIN_HEADERS cache_mode.",
    "errors": [
      {
        "message": "Invalid value for field 'resource.cdnPolicy.defaultTtl': '0'. default_ttl cannot be specified with USE_ORIGIN_HEADERS cache_mode.",
        "domain": "global",
        "reason": "invalid"
      }
    ]
  }
}

Expected Behavior

I expected Terraform to not specify the clientTtl, defaultTtl and maxTtl fields in the request to GCP, as they were not specified in the Terraform configuration.

Actual Behavior

Terraform passes the clientTtl, defaultTtl and maxTtl fields in the request to GCP, with their values all set to 0. Then fails with error: Error creating BackendBucket: googleapi: Error 400: Invalid value for field 'resource.cdnPolicy.defaultTtl': '0'. default_ttl cannot be specified with USE_ORIGIN_HEADERS cache_mode., invalid.

Steps to Reproduce

1. terraform apply

[d10i]

I'm working on a Pull Request to fix the issue.

[apottere]

I think this is similar to https://github.com/hashicorp/terraform-provider-google/issues/10560, and may be tangentially related to https://github.com/hashicorp/terraform-provider-google/issues/10332

The provider should send the default value (0) if it's explicitly set to 0, but omit it if it's not set.

[nat-henderson]

Unfortunately, Terraform cannot distinguish between a value explicitly set to 0 and a value that is unset. It seems like it can on the first apply, but on refresh, the state will appear "set", even to d.GetOkExists.

That means that making this change would break users who need to set those values to 0 explicitly. We'll need something more complicated, which only sends the value if the cache_mode is not set to USE_ORIGIN_HEADERS. This is doable, but more complex than the provided PR, unfortunately.

[apottere]

Ah, makes sense. Yeah, in that case it seems like updating the provider to only send certain values based on the cache_mode would be correct.

[nat-henderson]

@d10i - are you interested in giving it another shot? It's a bit tricky, just let us know either way and we can triage this appropriately. :)

[apottere]

Also, does that mean that https://github.com/GoogleCloudPlatform/magic-modules/pull/5456 inadvertently broke the fix for https://github.com/hashicorp/terraform-provider-google/issues/10332?

[nat-henderson]

Most likely, yeah - fixing that should also be part of this bug.

[d10i]

@ndmckinley I could give it a try. What behaviour do we expect when TTLs are specified when cache_mode is USE_ORIGIN_HEADERS? Ignore TTLs (i.e. don't pass them to GCP), error or something else?

[Sytten]

This is breaking our deployments too

[nat-henderson]

I'm not sure - I suppose the best outcome would be a validation error, but I'd accept either of your suggestions (and likely anything you came up with in "something else", as long as it works well for customers!).

[d10i]

Hey @ndmckinley, I spent some time on this and drafted a PR: https://github.com/GoogleCloudPlatform/magic-modules/pull/5524

However, tests are not passing locally:

$ make testacc TEST=./google TESTARGS='-run=TestAccComputeBackendBucket_withCdnPolicy'
==> Checking source code against gofmt...
==> Checking that code complies with gofmt requirements...
go generate  ./...
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google -v -run=TestAccComputeBackendBucket_withCdnPolicy -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google/version.ProviderVersion=acc"
=== RUN   TestAccComputeBackendBucket_withCdnPolicy
=== PAUSE TestAccComputeBackendBucket_withCdnPolicy
=== CONT  TestAccComputeBackendBucket_withCdnPolicy
    provider_test.go:270: Step 5/12 error: After applying this test step, the plan was not empty.
        stdout:


        Terraform used the selected providers to generate the following execution
        plan. Resource actions are indicated with the following symbols:
          ~ update in-place

        Terraform will perform the following actions:

          # google_compute_backend_bucket.foobar will be updated in-place
          ~ resource "google_compute_backend_bucket" "foobar" {
                id                      = "projects/shoji-core/global/backendBuckets/tf-test-2j94hxnt40"
                name                    = "tf-test-2j94hxnt40"
                # (6 unchanged attributes hidden)

              ~ cdn_policy {
                  ~ client_ttl                   = 3600 -> 0
                  ~ default_ttl                  = 3600 -> 0
                    # (5 unchanged attributes hidden)

                    # (1 unchanged block hidden)
                }
            }

        Plan: 0 to add, 1 to change, 0 to destroy.
--- FAIL: TestAccComputeBackendBucket_withCdnPolicy (64.92s)
FAIL
FAIL    github.com/hashicorp/terraform-provider-google/google   64.945s
FAIL
make: *** [GNUmakefile:14: testacc] Error 1

I'm not too sure using a custom encoder is the right way to go and I'm not sure why the test plan has TTL changes.

Could anybody help?

[atrauzzi]

This is blowing up load balancer deployments and breaks the ability to configure CDNs. Just making sure, but does this issue have the right priotity?

[wyardley]

We were able to work around at least one related issue by pinning back to 3.90.1, and seems like it broke as of v4.0.0, if that helps clarify anything.

[tedder]

I suspect it was caused by #10375.

[d10i]

Indeed @tedder. I had originally tried to fix this issue with this PR which undoes part of the PR you linked and does fix this specific issue. The argument for not doing that is that it would break things for existing users that didn't explicitly set TTLs to 0 (see message). IMO it does not make sense to send 0 by default when that breaks things every time cache_mode is not set to CACHE_ALL_STATIC and so it would make sense for things to break for existing users that didn't explicitly set TTLs to 0. But I'm not a decision-maker here, so just trying to help fix the problem in other ways :grin:

[wyardley]

Anyone have any updates / news here? We're still pinning back.

[apottere]

Same, I'm not sure how to go about helping to get this fixed but it's definitely still causing problems for us. We're pinning back to 3.90.1 but that's preventing us from using things like secret_environment_variables in our functions.

[apottere]

@nat-henderson if I'm reading correctly, does your comment here (https://github.com/GoogleCloudPlatform/magic-modules/pull/5524#issuecomment-1005177801) mean it's not possible to send certain TTLs based on the value of cache_mode? Or does it only break if cnd_policy is unset in the terraform? In that case, could TTLs be sent based on the assumed default value of cnd_policy is unset?

[nat-henderson]

@rileykarson - This should go back on the triage pile - but can you weigh in on the level of tolerance you have for breaking existing configs?

@apottere - The ultimate problem is that, in Terraform, "unset" is not a concept that persists across applies. The first apply will understand that a value is "unset", but after that, because the value is "set" by the Read subsequent to the first Create, it will not be possible to detect the "unset" state.

[rileykarson]

Anything that breaks user configs would have to wait for 5.0.0. We could use an opt-in field to do this safely- omit_zero_ttls or something- which would mean that TTLs could not be set to zero anymore when set.

[wyardley]

Best I can see this seems better in the latest 4.x? At least, I haven't tried destroying / recreating things, but the states that we had pinned back plan clean now. I had thought at first that #11791 might fix this, but looking at it a bit more closely, I think this is different?

[tedder]

for the record, it was planning clean with the version update, but once changes were made, the TTL error began again- so this issue isn't fixed.

running "terraform1.2.3 apply -input=false": exit status 1
zz.google_compute_backend_bucket.cdn_backend_bucket: Modifying... [id=projects/xx/global/backendBuckets/yy]
╷
│ Error: Error updating BackendBucket "projects/xx/global/backendBuckets/yy":
   googleapi: Error 400: Invalid value for field 'resource.cdnPolicy.defaultTtl': '0'.
   default_ttl cannot be specified with USE_ORIGIN_HEADERS cache_mode., invalid
│ 
│   with zz.google_compute_backend_bucket.cdn_backend_bucket,
│   on ../../zz/loadbalancer.tf line 16, in resource "google_compute_backend_bucket" "cdn_backend_bucket":
│   16: resource "google_compute_backend_bucket" "cdn_backend_bucket" {
│ 

We're not specifying defaultTtl. We're changing the custom_response_headers and otherwise have a fairly straightfoward resource block.

resource "google_compute_backend_bucket" "cdn_backend_bucket" {
  name        = "myname"
  bucket_name = google_storage_bucket.cdn_bucket.name
  enable_cdn  = true

  // typically security related custom headers
  custom_response_headers = [ ... ]

  cdn_policy {
    cache_mode = "USE_ORIGIN_HEADERS"
  }
}

So, pinning it back to 3.90:

terraform {
  required_version = ...
  required_providers {
    google = {
      source = "hashicorp/google"
      # version pin required for CDN/bucket with custom headers until this is fixed:
      # https://github.com/hashicorp/terraform-provider-google/issues/10622
      # tested (confirmed broken) through 4.26.0.
      version = "~> 3.90.1"
    }
  }
}

[longlho]

hello is there any workaround for this?

[victorbiga]

I don't feel convinced pinning to old version. Any progress on this issue ? Would be great if there would be some movement on this or at least workaround?

The value of default_ttl is 0 in current state file

"module": "module.deployment_my_service.module.backend_documentation",
"mode": "managed",
"type": "google_compute_backend_bucket",
"name": "default",
"provider": "provider[\"registry.terraform.io/hashicorp/google\"]",
"instances": [
  {
    "schema_version": 0,
    "attributes": {
      "bucket_name": "documentation-my-service",
      "cdn_policy": [
        {
          "bypass_cache_on_request_headers": [],
          "cache_key_policy": [],
          "cache_mode": "USE_ORIGIN_HEADERS",
          "client_ttl": 0,
          "default_ttl": 0,
          "max_ttl": 0,
          "negative_caching": false,
          "negative_caching_policy": [],
          "request_coalescing": false,
          "serve_while_stale": 0,
          "signed_url_cache_max_age_sec": 0
        }

Having this in tf file does not solve:

bucket_name = google_storage_bucket.default.name
  cdn_policy {
    cache_mode = USE_ORIGIN_HEADERS
    default_ttl = 0
  }
  enable_cdn = true
  name       = var.backend_name

[victorbiga]

Adding the request_coalescing = true fixed the issue:

  bucket_name = google_storage_bucket.default.name
  cdn_policy {
    cache_mode         = var.cache_mode
    request_coalescing = true
  }
  enable_cdn = true
  name       = var.backend_name

[wyardley]

For us, request_coalescing = true still does not fix the issue when modifying custom_response_headers.

│ Error: Error updating BackendBucket "projects/xxx/global/backendBuckets/xyz": googleapi: Error 400: Invalid value for field 'resource.cdnPolicy.defaultTtl': '0'. default_ttl cannot be specified with USE_ORIGIN_HEADERS cache_mode., invalid

The API call has

{
 "bucketName": "xyz",
 "cdnPolicy": {
  "cacheMode": "USE_ORIGIN_HEADERS",
  "clientTtl": 0,
  "defaultTtl": 0,
  "negativeCaching": false,
  "requestCoalescing": true,
  "serveWhileStale": 0,
  "signedUrlCacheMaxAgeSec": 0
 },
 "customResponseHeaders": [

despite default_ttl being listed as an optional param, and us not having it set in cdn_policy:

Actual code (same as comments above from 6/27, but setting request_coalescing = true as mentioned above. What it seems like is that everything will plan / apply fine if no changes are being made - maybe in the case above, truing up request_coalescing with the actual state seems to fix things.

resource "google_compute_backend_bucket" "cdn_backend_bucket" {
  name        = join("-", [local.bucket_name, "backend"])
  bucket_name = google_storage_bucket.cdn_bucket.name
  enable_cdn  = true

  // this is a long set of custom headers
  custom_response_headers = local.full_custom_headers

  cdn_policy {
    cache_mode         = "USE_ORIGIN_HEADERS"
    request_coalescing = true
  }
}

[iJebus]

request_coalescing = true does not make any difference in my case 🤷 .

[IchordeDionysos]

Is there any news on this bug? It's really blocking ...

[IchordeDionysos]

The same issue also appears when using a compute_backend_service ...

[1ron]

Any updates on this?