Google connector with ADC: unexpected end of JSON input

[tsawada]

Preflight Checklist

• [X] I agree to follow the Code of Conduct that this project adheres to.
• [X] I have searched the issue tracker for an issue that matches the one I want to file, without success.
• [X] I am not looking for support or already pursued the available support channels without success.

Version

ghcr.io/dexidp/dex:v2.34.0

Storage Type

Kubernetes

Installation Type

Other (specify below)

Expected Behavior

google connector successfully use Application Default Credential via Compute Engine metadata server

Actual Behavior

failed to initialize server: server: Failed to open connector google: failed to open connector: failed to create connector google: could not create directory service: unable to parse credentials to config: unexpected end of JSON input

Steps To Reproduce

1. deploy dex on GCE / GKE. Use google connector without specifying serviceAccountFilePath
2. check the logs

Additional Information

I installed dex via ArgoCD's install.yaml, and replaced its version from v2.30.2 to v2.34.0. My setup worked OK before, with a key file in serviceAccountFilePath.

I tried running the following code on the same GKE pod by kubectl exec, and it went all good.

func main() {
        cred, err := google.FindDefaultCredentials(context.Background())
        if err != nil {
                fmt.Errorf("FindDefaultCredentials: %w", err)
        }
        fmt.Printf("Json: %v\n", cred.JSON)
        cfg, err := google.JWTConfigFromJSON(cred.JSON, admin.AdminDirectoryGroupReadonlyScope)
        if err != nil {
                fmt.Errorf("JWTConfigFromJSON: %w\n", err)
        }
        fmt.Printf("cfg: %v\n", cfg)
}

The above code works OK with following output, which is something I expected.

cred: &{my-project-name 0xc0000aed00 []}
Json:
cfg: <nil>

The code does basically the same as dex v2.34.0, so I am not sure why it doesn't work in dex.

cc @ichbinfrog

Configuration

connectors:
    - type: google
      id: google
      name: Google
      config:
        issuer: https://accounts.google.com
        clientID: $GOOGLE_CLIENT_ID
        clientSecret: $GOOGLE_CLIENT_SECRET
        redirectURI: https://my.domain/api/dex/callback
        hostedDomains: ['my.domain']
        serviceAccountFilePath: /tmp/keyfile.json
        adminEmail: [email protected]

Logs

Dex Version: v2.34.0-dirty, Go Version: go1.19.1, Go OS/ARCH: linux amd64
config issuer: https://***
config connector: google
config skipping approval screen
config refresh tokens rotation enabled: true
the application default credential is used since the service account file path is not used
failed to initialize server: server: Failed to open connector google: failed to open connector: failed to create connector google: could not create directory service: unable to parse credentials to config: unexpected end of JSON input

[ichbinfrog]

I think I might have isolated the issue with the code (upon further testing in a GKE cluster) and its with this line: https://github.com/dexidp/dex/blob/master/connector/google/google.go#L300

The default behavior for the google.FindDefaultCredentials function is the following:

• On Google Compute Engine, Google App Engine standard second generation runtimes (>= Go 1.11), and Google App Engine flexible environment, it fetches credentials from the metadata server.

credential.JSON is empty in GKE environments, thus the error. I'm working on a fix for this but it feels very convoluted with the admin SDK API and it's a bit tough to test without e2e tests.

[tsawada]

Thank you for investigating this. Yeah, I also suspect it's because of JSON being empty, but I'm still confused because my testing code above worked okay (returning <nil> config successfully) in my GKE env with an empty JSON. Do you have any theory what is the different between my testing code snippet and dex?

[xrstf]

Is this a breaking change? From the code change in #2530 it seems that

if serviceAccountFilePath == "" && email == "" {
	return nil, nil
}

was removed and so now as an admin I have to configure additional stuff for the google connector to work? Due to the security issue fixed in 2.34 we just tried to bump our Dex installations from 2.32 to 2.35.1 and things suddenly crashlooped. So far our configuration was rather simple:

  - type: google
    id: google
    name: Google
    config:
      issuer: https://accounts.google.com
      clientID: yadda.yadda.yadda.apps.googleusercontent.com
      clientSecret: thisIsNotARealSecret
      redirectURI: https://initech.example.com/dex/callback
      hostedDomains:
      - initech.com
      - initroid.com

Our Dex pods run in Kubernetes on GKE nodes.

[sagikazarmark]

@xrstf it does seem to be yet another regression with the Google connector. Annoying...

There is a new patch version coming up upgrading Go to the latest patch version. I believe I can patch this one as well: #2699

As for the original issue: we need a better understanding of how the admin service works. Previous assumption was that an admin user has to be impersonated in order to use it. We need to verify that assumption/find out how authorization works with the admin service.

As a quick fix though, we could disable impersonation if JSON is empty for a default credential.

Edit: @xrstf in your case the groups scope is not configured as far as I can tell, so the fix I'm about to submit should fix your case. As soon as the groups scope is added to the list, you will have to change your configuration.

[sagikazarmark]

See #2700

[sagikazarmark]

Also #2122 could solve this problem as well.

[sagikazarmark]

Please comment on #2701

[haydentherapper]

FYI, deploying 2.35.1, we saw an error:

failed to initialize server: server: Failed to open connector https://accounts.google.com: failed to open connector: failed to create connector https://accounts.google.com: could not create directory service: unable to parse credentials to config: unexpected end of JSON input

[sagikazarmark]

Thanks for reporting. I believe the solution to that is going to be refactoring the code based on #2122

[milesarmstrong]

Hiya @sagikazarmark 👋

I'm not too sure which issue / PR to comment on, but we're also hitting problems with Workload Identity / ADC in GKE.

We're running ArgoCD, deployed with the Helm chart and using the bundled dex deployment.

We followed these instructions to get group membership.

We don't see any errors/warnings in the logs on startup (with or without adminEmail), but we don't see group membership in the ArgoCD UI.

It looks like some commits went into master since the latest release which are related to this.

Do you know when the next release is planned for?

[sagikazarmark]

@milesarmstrong we are still trying to determine the correct solution for the problem.

Chances are the latest release does NOT work with workload identity. The current workaround is manually passing an IAM service account to the Dex workload. It should work with the latest release.

I know it's not ideal, but that's the currently available workaround.

Although we wanted to fix the problem in the next release (which is due this week), it may not just be part of it.

[torfjor]

Pitching in here that we managed to get dex running with groups on Anthos Bare Metal (no metadata server) with workload identity using these changes.

The issue is that google.JWTConfigFromJSON will fail for Anthos setups because they use federated tokens and their credential setup for ADC is a little different than on GKE:

{
  "type": "external_account",
  "audience": "identitynamespace:WORKLOAD_IDENTITY_POOL:IDENTITY_PROVIDER",
  "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/GSA_NAME@GSA_PROJECT_ID.iam.gserviceaccount.com:generateAccessToken",
  "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
  "token_url": "https://sts.googleapis.com/v1/token",
  "credential_source": {
    "file": "/var/run/secrets/tokens/gcp-ksa/token"
  }
}

See the docs for reference.

[jangjaelee]

I think I might have isolated the issue with the code (upon further testing in a GKE cluster) and its with this line: https://github.com/dexidp/dex/blob/master/connector/google/google.go#L300

The default behavior for the google.FindDefaultCredentials function is the following:

• On Google Compute Engine, Google App Engine standard second generation runtimes (>= Go 1.11), and Google App Engine flexible environment, it fetches credentials from the metadata server.

credential.JSON is empty in GKE environments, thus the error. I'm working on a fix for this but it feels very convoluted with the admin SDK API and it's a bit tough to test without e2e tests.

It is as follows according to the Google Oauth2 document.

https://pkg.go.dev/golang.org/x/oauth2/google#Credentials

// JSON contains the raw bytes from a JSON credentials file.
// This field may be nil if authentication is provided by the
// environment and not with a credentials file, e.g. when code is
// running on Google Cloud Platform.

I tried with using GKE Workload Identity function to authentication from Google Workspace. But it could not success. Because Google return empty(nil) credentials.

I think google oauth2 does not support Google Kubernetes Engine environment.

[kjvellajr]

I have also verified the changes made by @torfjor work flawlessly when running Dex on GKE using Workload Identity to retrieve Google groups. In my case the Dex server is running as auth for ArgoCD. I attached a Workload Identity GSA to the Dex server that has the Groups Reader Admin role as outlined by https://support.google.com/a/answer/9807615?hl=en#zippy=%2Cassign-a-role-to-a-service-account.

[vsychov]

I've been trying to understand the issue and why it's not working, and I've come to the following conclusions (I might be wrong, but this is my current understanding):

1. To send requests to Google Workspace (GWS) to retrieve the list of groups, it's necessary to obtain a token that is impersonated with a GWS admin user.

2. To achieve this using the service account key, the Google SDK forms a JWT token (with iss set to the service account email and sub set to the GWS admin email). This JWT is then signed with the private key (obtained from the service account JSON file) and a request is made to https://oauth2.googleapis.com/token. The response to this request provides an access token (ya29.*****).

3. Subsequently, this token is used to send requests to GWS on behalf of the GWS admin user.

4. However, when working with GKE Workload Identity, we cannot access the service account's private key. All we can obtain is a ready-made access token (ya29.*****). Therefore, we are unable to form a request to https://oauth2.googleapis.com/token (with correct iss and sub) and consequently, we cannot obtain a token that impersonates an admin user.

Based on the limitations, it seems that it's fundamentally not possible to achieve impersonation using GKE Workload Identity in this scenario.

[torfjor]

It is possible to achieve what you want by constructing your desired JWT and passing it to the IAM Service Account Credentials API (see the signJwt method).

Nowadays you can grant service accounts workspace admin roles without domain wide delegation. That's probably a more comfortable route.

[vsychov]

@torfjor, thank you for your suggestion. I tested it, and can confirm that it indeed works.

https://cloud.google.com/sdk/gcloud/reference/iam/service-accounts/sign-jwt

[vsychov]

I have submitted a pull request addressing the mentioned issue: https://github.com/dexidp/dex/pull/2989. It would be greatly appreciated if someone could test it.