google_apikeys_key missing X-Goog-User-Project (thus doesn't work with ADC)

[MPV]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

1.1.5

Affected Resource(s)

• google_apikeys_key

Terraform Configuration Files

resource "google_apikeys_key" "key" {
  name         = "mykey"
  display_name = "mykey"
}

Debug Output

[...]
-----------[REQUEST]----------
GET /v2/projects/REDACTED/locations/global/keys/mykey?alt=json HTTP/1.1
Host: apikeys.googleapis.com
User-Agent: Terraform/1.1.5 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google/dev DeclarativeClientLib/0.0.1
Content-Type: application/json
Accept-Encoding: gzip


-------[END REQUEST]--------: timestamp=2022-06-10T14:51:22.198+0200
[...]
2022-06-10T14:51:22.975+0200 [INFO]  provider.terraform-provider-google_v4.21.0_x5: 2022/06/10 14:51:22 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2022-06-10T14:51:22.975+0200
2022-06-10T14:51:22.975+0200 [INFO]  provider.terraform-provider-google_v4.21.0_x5: 2022/06/10 14:51:22 [DEBUG][DCL INFO] [RequestID:]  Google API Response: (id dkbr2) 
-----------[RESPONSE]----------
HTTP/2.0 403 Forbidden
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Fri, 10 Jun 2022 12:51:22 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
  "error": {
    "code": 403,
    "message": "Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the apikeys.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/. If you are getting this error with curl or similar tools, you may need to specify 'X-Goog-User-Project' HTTP header for quota and billing purposes. For more information regarding 'X-Goog-User-Project' header, please check https://cloud.google.com/apis/docs/system-parameters.",
    "status": "PERMISSION_DENIED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "SERVICE_DISABLED",
        "domain": "googleapis.com",
        "metadata": {
          "service": "apikeys.googleapis.com",
          "consumer": "projects/REDACTED"
        }
      }
    ]
  }
}

-------[END RESPONSE]--------: timestamp=2022-06-10T14:51:22.975+0200
2022-06-10T14:51:22.975+0200 [INFO]  provider.terraform-provider-google_v4.21.0_x5: 2022/06/10 14:51:22 [DEBUG][DCL INFO] Error appears not to be retryable: googleapi: Error 403: Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the apikeys.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/. If you are getting this error with curl or similar tools, you may need to specify 'X-Goog-User-Project' HTTP header for quota and billing purposes. For more information regarding 'X-Goog-User-Project' header, please check https://cloud.google.com/apis/docs/system-parameters.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "googleapis.com",
    "metadata": {
      "consumer": "projects/REDACTED",
      "service": "apikeys.googleapis.com"
    },
    "reason": "SERVICE_DISABLED"
  }
]: timestamp=2022-06-10T14:51:22.975+0200
2022-06-10T14:51:22.976+0200 [ERROR] vertex "module.mykeymodule.google_apikeys_key.key" error: Error when reading or editing ApikeysKey "projects/REDACTED/locations/global/keys/mykey": googleapi: Error 403: Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the apikeys.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/. If you are getting this error with curl or similar tools, you may need to specify 'X-Goog-User-Project' HTTP header for quota and billing purposes. For more information regarding 'X-Goog-User-Project' header, please check https://cloud.google.com/apis/docs/system-parameters.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "googleapis.com",
    "metadata": {
      "consumer": "projects/REDACTED",
      "service": "apikeys.googleapis.com"
    },
    "reason": "SERVICE_DISABLED"
  }
]
[...]

Panic Output

Expected Behavior

Able to terraform plan google_apikeys_key when using ADC and not logged in as a GCP SA.

Actual Behavior

$ terraform plan
[...]
╷
│ Error: Error when reading or editing ApikeysKey "projects/REDACTED/locations/global/keys/mykey": googleapi: Error 403: Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the apikeys.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/. If you are getting this error with curl or similar tools, you may need to specify 'X-Goog-User-Project' HTTP header for quota and billing purposes. For more information regarding 'X-Goog-User-Project' header, please check https://cloud.google.com/apis/docs/system-parameters.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/REDACTED",
│       "service": "apikeys.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│
│   with module.mykeymodule.google_apikeys_key.key,
│   on ../../modules/gcp-api-key/main.tf line 1, in resource "google_apikeys_key" "key":
│    1: resource "google_apikeys_key" "key" {
│
╵

Steps to Reproduce

1. terraform plan

Important Factoids

Authenticating as a user, with ADC.

The equivalent works fine when either using a service account, or when configuring the provider for example like this:

provider "google" {
  project               = var.gcp_project
  region                = var.gcp_region
  billing_project       = var.gcp_project
  user_project_override = true
}

References

• #8959
• #6738
• #7477
  • https://github.com/hashicorp/terraform-provider-google/issues/7477#issuecomment-707303599

[MPV]

@slevenick maybe knows more? Since you added the functionality back in:

• https://github.com/hashicorp/terraform-provider-google/issues/8959
• https://github.com/hashicorp/terraform-provider-google/pull/11249
• https://github.com/GoogleCloudPlatform/magic-modules/pull/5637

[panjek26]

need help for this also. i am facing the same error

[melinath]

Thanks for reporting this bug! It looks like this is an issue with how the DCL makes its requests.

Here are my notes for future reference while looking into this bug:

• x-goog-user-project is set on the core Config object.
• google_apikeys_key is a DCL-based resource that reuses config.client, so it should have the same client as the core config, including any additional headers.
• However, the DCL uses loggingTransport to round-trip its requests - as opposed to the Terraform provider, which uses headerTransportLayer as client.Transport. Since the DCL doesn't call client.Get, it maybe sidesteps using client.Transport and doesn't set the correct headers? But I'm having trouble tracking how it's actually using the client.
• b/236042741

Question for folks experiencing this: What provider version are you using? It looks like the user agent is being reported as terraform-provider-google/dev instead of a version number. The DCL version seems to be reported as 0.0.1, which is very out of date.

[ghost]

Thanks for reporting this bug! It looks like this is an issue with how the DCL makes its requests.

Here are my notes for future reference while looking into this bug:

• x-goog-user-project is set on the core Config object.
• google_apikeys_key is a DCL-based resource that reuses config.client, so it should have the same client as the core config, including any additional headers.
• However, the DCL uses loggingTransport to round-trip its requests - as opposed to the Terraform provider, which uses headerTransportLayer as client.Transport. Since the DCL doesn't call client.Get, it maybe sidesteps using client.Transport and doesn't set the correct headers? But I'm having trouble tracking how it's actually using the client.
• b/236042741

Question for folks experiencing this: What provider version are you using? It looks like the user agent is being reported as terraform-provider-google/dev instead of a version number. The DCL version seems to be reported as 0.0.1, which is very out of date.

We are using provider version 4.24

[jaksky]

4.31.0 is also having same issue (terraform 1.0.10)

[adambozson]

Experiencing this on 4.34.0, Terraform v1.2.6

[ScottSuarez]

Could you share your full configurations ?

I deployed with the example configuration below. All rest requests had the the correct headers.

What I suspect is happening is that you are using multiple provider versions but don't have a provider block for each of them. Please ensure if your configuration uses 2 provider versions you have a provider block for both google and google-beta.

provider "google-local" {
  project               = "some-project"
  region                = "us-central1"
  billing_project       = "some-project"
  user_project_override = true
}


resource "google_apikeys_key" "primary" {
  provider = google-local
  name         = "tf-test-key-0"
  display_name = "sample-key"
  project      = google_project.basic.name

  restrictions {
    api_targets {
      service = "translate.googleapis.com"
      methods = ["GET*"]
    }

    browser_key_restrictions {
      allowed_referrers = [".*"]
    }
  }
}

resource "google_project" "basic" {
  provider = google-local
  project_id = "tf-test-app-xxxxxx"
  name       = "tf-test-app-xxxxxx"
  org_id     = "---"
}

[ScottSuarez]

Closing this issue as no response. I am going to assume this is a non-issue.

[LoicMahieu]

Hi @ScottSuarez

I also experiencing this issue. I don't use multiple provider versions.

Here is my TF:

provider "google" {
  project               = "redacted"
  region                = "europe-west1"
}

resource "google_project_service" "apikeys" {
  service            = "apikeys.googleapis.com"
  disable_on_destroy = false
}

resource "google_apikeys_key" "primary" {
  provider = google
  name    = "test-server"
  project = "redacted"
  restrictions {
    api_targets {
      service = "geocoding_backend"
    }
  }
  depends_on = [
    google_project_service.apikeys
  ]
}

And here is the error logs:

google_apikeys_key.primary: Creating...
╷
│ Error: Error creating Key: failed to create a diff: failed to retrieve Key resource: googleapi: Error 403: Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the apikeys.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/. If you are getting this error with curl or similar tools, you may need to specify 'X-Goog-User-Project' HTTP header for quota and billing purposes. For more information regarding 'X-Goog-User-Project' header, please check https://cloud.google.com/apis/docs/system-parameters.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/764086051850",
│       "service": "apikeys.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│
│   with google_apikeys_key.primary,
│   on main.tf line 11, in resource "google_apikeys_key" "primary":
│   11: resource "google_apikeys_key" "primary" {

• registry.terraform.io/hashicorp/google 4.39.0

[ScottSuarez]

Hi Loic, this is a specific, opt in feature. Try adding user_project_override=true to your provider level configuration

[LoicMahieu]

Hi @ScottSuarez Great thanks for your quick help !

It works fine when adding user_project_override=true in my previous example. (Not sure to understand the whole thing ;))

But... I get the same error when adding a browser_key_restrictions.allowed_referrers field! Here is a complete example:

provider "google" {
  project               = "redacted"
  region                = "europe-west1"
  user_project_override = true
}

resource "google_project_service" "apikeys" {
  service            = "apikeys.googleapis.com"
  disable_on_destroy = false
}

resource "google_apikeys_key" "primary" {
  name     = "test-server"
  project  = "redacted"
  restrictions {
    api_targets {
      service = "geocoding_backend"
    }
  }
  depends_on = [
    google_project_service.apikeys
  ]
}


resource "google_apikeys_key" "secondary" {
  name     = "test-secondary"
  project  = "redacted"
  restrictions {
    api_targets {
      service = "geocoding_backend"
    }

    browser_key_restrictions {
      allowed_referrers = [
        "some-referrer.com",
      ]
    }
  }
  depends_on = [
    google_project_service.apikeys
  ]
}

Thanks in advance for your knowledge ;)

[ScottSuarez]

interesting, are you saying when you update the field through a sequential apply you get the error?

[LoicMahieu]

• I destroy the state successfully but I keep the same project
• I apply and I got this error:

│ Error: Error creating Key: Resource already exists - apply blocked by lifecycle params: &apikeys.Key{Name:(*string)(0x14000db0390), DisplayName:(*string)(nil), KeyString:(*string)(0x14000e56310), Uid:(*string)(0x14000db0cc0), Restrictions:(*apikeys.KeyRestrictions)(0x14000cf0d00), Project:(*string)(0x14000db0410)}.

• I rename the keys with -2
• I apply and it works
• I add more api_targets to test-secondary, it become:

resource "google_apikeys_key" "secondary" {
  name     = "test-secondary-2"
  display_name = "test-secondary-2"
  project  = "rediracted"
  restrictions {
    api_targets {
      service = "directions_backend"
    }
    api_targets {
      service = "geocoding_backend"
    }
    api_targets {
      service = "maps_backend"
    }
    api_targets {
      service = "places_backend"
    }
    browser_key_restrictions {
      allowed_referrers = [
        "some-referrer.com",
      ]
    }
  }
  depends_on = [
    google_project_service.apikeys
  ]
}

• I got the error:

│ Error: Error updating Key: googleapi: Error 403: Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the apikeys.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/. If you are getting this error with curl or similar tools, you may need to specify 'X-Goog-User-Project' HTTP header for quota and billing purposes. For more information regarding 'X-Goog-User-Project' header, please check https://cloud.google.com/apis/docs/system-parameters.

• Then, I rollback the api_targets and re-apply the plan, same error.

• I destroy the plan with success

• Got the error due to the key, I renamed it to -3

• I apply the plan but with all api_targets this time: it works !

[ScottSuarez]

Ahh the error message they give back encompasses the fix. Looks like they want you to set a billing project.

Try doing that on your provider cofig

provider "google" {
  project               = "some-project"
  region                = "us-central1"
  billing_project       = "some-project"
  user_project_override = true
}

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.