WIP: Improve authentication instructions in provider documentation

[toadjaune]

The current instructions (using gcloud auth login) do not seem to work : the application seems unable to pick it, and leads to the following error :

Error: failed to load config: failed to create client: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

Instead, using gcloud auth application-default login uses a similar login process (open a browser window to allow the login, etc), but instead sets the resulting creds as ADC.

If the code in its current state is meant to be able to pick user credentials set with gcloud auth login, I was not able to understand how. In this case, I suppose the documentation would benefit on extra details on this setup.

[toadjaune]

After further testing, it seems authenticating with gcloud auth application-default login does not work either (although it fails later, with a different error message)

I see 2 options there :

• authentication with a user account doesn't work
• authentication with a user account works, but I was unable to configure it properly

Could someone who got it working at some point please provide an example configuration for this setup ? :)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[toadjaune]

@sioncojp as mentioned in the text above, the proposed change doesn't actually work (I initially thought it did though), I'll have to figure out how to make it work before updating the docs.

[toadjaune]

Can anyone who got the authentication working with a user account (not a service account) please provide an example configuration ? I was unable to get it working on my side.

@DeviaVir maybe ?

[DeviaVir]

@toadjaune what error message did you hit RE:

After further testing, it seems authenticating with gcloud auth application-default login does not work either (although it fails later, with a different error message)

[toadjaune]

Unfortunately I don't have my test setup anymore, I'll set it up again in the weeks to come, to provide detailed config examples and corresponding error messages.

[toadjaune]

Hi again, here is a minimal example :

terraform {
  backend "local" {
    path = "sso.tfstate"
  }
  required_providers {
    gsuite = {
      source  = "DeviaVir/gsuite"
      version = "0.1.58"
    }
  }
  required_version = ">= 0.13"
}

provider "gsuite" {
  oauth_scopes = [
    "https://www.googleapis.com/auth/admin.directory.user",
    "https://www.googleapis.com/auth/admin.directory.group",
  ]
}

resource "gsuite_user" "developer" {
  name = {
    family_name = "Sillevis"
    given_name  = "Chase"
  }
  password = "here_is_some_very_good_password"
  primary_email = "[email protected]"
}

According to the documentation, this should work. Instead, I get this :

$ gcloud auth login
# successful login as my personal admin account
$ rf -rf .terraform
$ terraform init

Initializing the backend...

Successfully configured the backend "local"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Reusing previous version of deviavir/gsuite from the dependency lock file
- Installing deviavir/gsuite v0.1.58...
- Installed deviavir/gsuite v0.1.58 (self-signed, key ID AD9970F98EB884FD)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
$ terraform plan

Error: failed to load config: failed to create client: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

  on sso.tf line 14, in provider "gsuite":
  14: provider "gsuite" {

I tried various other configurations (notably, with OAuth scopes), with the same result. This error tends to make me believe that the gsuite provider is not able to leverage the user auth configured by gcloud auth login.

Next try, I logged in using gcloud auth application-default login (and set up a quota account with gcloud config set billing/quota_project some_gcp_account, should it matter).

Now, the plan works as expected, but an apply fails :

$ terraform apply

Error: [ERROR] Error creating user: googleapi: Error 403: Request had insufficient authentication scopes.
More details:
Reason: insufficientPermissions, Message: Insufficient Permission

By setting TF_LOG=DEBUG, we can get some interesting information :

2021-02-26T19:37:44.602+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: ---[ REQUEST ]---------------------------------------
2021-02-26T19:37:44.602+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: POST /admin/directory/v1/users?alt=json&prettyPrint=false HTTP/1.1
2021-02-26T19:37:44.602+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Host: www.googleapis.com
2021-02-26T19:37:44.602+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: User-Agent: google-api-go-client/0.5 (linux amd64) Terraform/0.14.7
2021-02-26T19:37:44.602+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Content-Length: 287
2021-02-26T19:37:44.603+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Content-Type: application/json
2021-02-26T19:37:44.603+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: X-Goog-Api-Client: gl-go/1.15.6 gdcl/20200514
2021-02-26T19:37:44.603+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Accept-Encoding: gzip
2021-02-26T19:37:44.603+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: 
2021-02-26T19:37:44.603+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: {
2021-02-26T19:37:44.603+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:  "changePasswordAtNextLogin": true,
2021-02-26T19:37:44.603+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:  "externalIds": [],
2021-02-26T19:37:44.603+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:  "includeInGlobalAddressList": true,
2021-02-26T19:37:44.603+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:  "name": {
2021-02-26T19:37:44.603+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:   "familyName": "Sillevis",
2021-02-26T19:37:44.603+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:   "givenName": "Chase"
2021-02-26T19:37:44.603+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:  },
2021-02-26T19:37:44.603+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:  "orgUnitPath": "Tech",
2021-02-26T19:37:44.603+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:  "organizations": [],
2021-02-26T19:37:44.603+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:  "password": "here_is_some_very_good_password",
2021-02-26T19:37:44.603+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:  "primaryEmail": "[email protected]",
2021-02-26T19:37:44.603+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:  "sshPublicKeys": []
2021-02-26T19:37:44.603+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: }
2021-02-26T19:37:44.603+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: 
2021-02-26T19:37:44.603+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: -----------------------------------------------------
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: 2021/02/26 19:37:44 [DEBUG] Google API Response Details:
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: ---[ RESPONSE ]--------------------------------------
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: HTTP/2.0 403 Forbidden
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Cache-Control: private
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Content-Type: application/json; charset=UTF-8
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Date: Fri, 26 Feb 2021 18:37:44 GMT
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Server: ESF
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Vary: Origin
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Vary: X-Origin
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Vary: Referer
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Www-Authenticate: Bearer realm="https://accounts.google.com/", error="insufficient_scope", scope="https://www.googleapis.com/auth/admin.directory.user https://www.googleapis.com/auth/directory.user"
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: X-Content-Type-Options: nosniff
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: X-Frame-Options: SAMEORIGIN
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: X-Xss-Protection: 0
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: 
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: {
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:   "error": {
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:     "code": 403,
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:     "message": "Request had insufficient authentication scopes.",
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:     "errors": [
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:       {
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:         "message": "Insufficient Permission",
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:         "domain": "global",
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:         "reason": "insufficientPermissions"
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:       }
2021-02-26T19:37:44.718+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:     ],
2021-02-26T19:37:44.719+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:     "status": "PERMISSION_DENIED"
2021-02-26T19:37:44.719+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:   }
2021-02-26T19:37:44.719+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: }
2021-02-26T19:37:44.719+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: 
2021-02-26T19:37:44.719+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: -----------------------------------------------------

Interestingly, we get the required scopes for this request (scope="https://www.googleapis.com/auth/admin.directory.user https://www.googleapis.com/auth/directory.user"), but even retrying explicitly specifying those scopes, with a super-admin user, gives the same result.

I tried a few other operations (importing an existing user, for example) with the same result.

One last interesting log I had in debug mode : before creating the user, we check if it exists. Here is the corresponding log :

2021-02-26T19:37:44.356+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: ---[ REQUEST ]---------------------------------------
2021-02-26T19:37:44.356+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: GET /admin/directory/v1/users?alt=json&customer=my_customer&prettyPrint=false&query=email%3Asillevis.chase%40example.com HTTP/1.1
2021-02-26T19:37:44.356+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Host: www.googleapis.com
2021-02-26T19:37:44.356+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: User-Agent: google-api-go-client/0.5 (linux amd64) Terraform/0.14.7
2021-02-26T19:37:44.356+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: X-Goog-Api-Client: gl-go/1.15.6 gdcl/20200514
2021-02-26T19:37:44.356+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Accept-Encoding: gzip
2021-02-26T19:37:44.356+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: 
2021-02-26T19:37:44.356+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: 
2021-02-26T19:37:44.356+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: -----------------------------------------------------
2021-02-26T19:37:44.600+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: 2021/02/26 19:37:44 [DEBUG] Google API Response Details:
2021-02-26T19:37:44.600+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: ---[ RESPONSE ]--------------------------------------
2021-02-26T19:37:44.600+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: HTTP/2.0 403 Forbidden
2021-02-26T19:37:44.600+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
2021-02-26T19:37:44.600+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Cache-Control: private
2021-02-26T19:37:44.600+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Content-Type: application/json; charset=UTF-8
2021-02-26T19:37:44.600+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Date: Fri, 26 Feb 2021 18:37:44 GMT
2021-02-26T19:37:44.600+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Server: ESF
2021-02-26T19:37:44.600+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Vary: Origin
2021-02-26T19:37:44.600+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Vary: X-Origin
2021-02-26T19:37:44.600+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: Vary: Referer
2021-02-26T19:37:44.600+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: X-Content-Type-Options: nosniff
2021-02-26T19:37:44.600+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: X-Frame-Options: SAMEORIGIN
2021-02-26T19:37:44.600+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: X-Xss-Protection: 0
2021-02-26T19:37:44.600+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: 
2021-02-26T19:37:44.600+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: {
2021-02-26T19:37:44.600+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:   "error": {
2021-02-26T19:37:44.600+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:     "code": 403,
.terraform-provider-gsuite_v0.1.58:     "message": "Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the admin.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/.",
2021-02-26T19:37:44.600+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:     "errors": [
2021-02-26T19:37:44.600+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:       {
2021-02-26T19:37:44.601+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:         "message": "Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the admin.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/.",
2021-02-26T19:37:44.601+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:         "domain": "usageLimits",
2021-02-26T19:37:44.601+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:         "reason": "accessNotConfigured",
2021-02-26T19:37:44.601+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:         "extendedHelp": "https://console.developers.google.com"
2021-02-26T19:37:44.601+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:       }
2021-02-26T19:37:44.601+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:     ],
2021-02-26T19:37:44.601+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:     "status": "PERMISSION_DENIED"
2021-02-26T19:37:44.601+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58:   }

I don't know what to make of this. Is it even possible to make any modification through the API with a user ? Or do we have to use a service account ? Or maybe I just failed to configure my quota project ?

At this point I'm thinking of creating a user through the console, and capture the corresponding traffic.

[DeviaVir]

In the error you post above it says:

2021-02-26T19:37:44.601+0100 [DEBUG] plugin.terraform-provider-gsuite_v0.1.58: "message": "Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the admin.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/.",

I'm curious if configuring the billing/quota_project setting in gcloud would resolve this for you. I think you can check what your current project is set to using gcloud config get-value project, then if you'd check out billing, make sure it is set to a valid billing account.

Otherwise, using a service account (as suggested above) is probably easier, this is what I know other customers use successfully.

[toadjaune]

I tried various ways of setting the quota project, such as :

• gcloud config set project ...
• gcloud config set billing/quota_project ...
• gcloud auth application-default set-quota-project ...

None of them seem to change anything.

It would seem that this API is indeed not usable at all from a user account.

I guess I'll switch to using a service account, which is cleaner on the principle anyway. The only issue I have with this setup is that permissions given to terraform allow taking over the entire organization (by impersonating a super-admin and resetting their password), which I'd rather avoid. I'll check if it is indeed the case.

Now, regarding this issue. If we agree that using this provider with a user account is apparently impossible, how about reflecting it in the documentation ? I don't think any API change is required, since both credentials and impersonated_user_email can be set via environment variables, they'll remain optional.

[toadjaune]

Actually, we're gonna need to make changes bigger than that, as I have great news :

It's now possible to give admin permissions directly to a service account in GSuite \o/

I just started testing this new setup, but it seems to work just fine so far (letting oauth_scopes empty, and setting impersonated_user_email to the service account's email, as the provider will currently refuse to start if we provide service account credentials without impersonated_user_email)

Extra testing will be required, but so far I tend to believe that we should make the following changes :

• Allow authenticating with a service account without setting an impersonated user email
• Update documentation to :
  • Remove any mention of authentication with a user account
  • Add authentication through a service account with admin privileges instructions (required privileges, where to set it, etc)
  • Make authentication through domain-wide delegation secondary. It's relevant to keep it for backwards compatibility, but new users should be discouraged to use it, as the new auth method is both way simpler and more secure. Mentioning it only in the description of the impersonated_user_email field, for example, would make sense to me.

What do you think ?