Headlamp OIDC with GKE

[slitsevych]

After numerous and various attempts it does not seem that Headlamp OIDC is working with GKE Identity Service and the only way to login is to use service account token.

1. Enabled Identity Service with GKE (https://cloud.google.com/kubernetes-engine/docs/how-to/oidc)
2. Changed default ClientConfig to the following

apiVersion: authentication.gke.io/v2alpha1
kind: ClientConfig
metadata:
  name: default
  namespace: kube-public
spec:
  authentication:
  - name: oidc
    oidc:
      clientID: <<CLIENT_ID>>
      groupPrefix: 'oidc:'
      groupsClaim: groups
      issuerURI: https://<<KEYCLOACK_URL/realms/<<MY_REALM>>
      kubectlRedirectURI: http://localhost:8000/callback
      scopes: email,profile
      userClaim: email
      userPrefix: 'oidc:'
  certificateAuthorityData: <<certificateAuthorityData>>
  internalServer: ""
  name: <<CLUSTER_NAME>>
  server: https://<<IP_ADDRESS_OF_gke-oidc-envoy LB service>>:443

server is set to the IP address of the gke-oidc-envoy LB service in "anthos-identity-service" namespace

3. Default login through kubectl oidc login --login-config=client-config.yaml --cluster=<<CLUSTER_NAME>> where client-config.yaml is the same client configuration as shown above + clientSecret - works just fine. I'm able to login with email that's mapped to "oidc:<<MY_EMAIL/USERNAME>>" and with clusterRoleBinding in place I'm able to gain the access.

4. In Headlamp though, using the following config:

   clientID: <<CLIENT_ID>>
   clientSecret: <<CLIENT_SECRET>>
   issuerURL: https://<<KEYCLOACK_URL/realms/<<MY_REALM>>
   scopes: email,profile

I cannot get any permissions/access. Moreover I don't see any relevant logs in headlamp at all, only messages like ""events is forbidden: User "system:anonymous" cannot list resource "events" in API group "" at the cluster scope" appearing in the browser console

---

As such I have a couple of questions:

1. Is there any working example of Headlamp OIDC with GKE (if any) or is it broken at this point and awaiting a fix?
2. Is it possible to see/enable some debug logs of Headlamp OIDC flow with API server or at least any hints on that matter in logs?

---

And on a somewhat unrelated note, is there a way to suppress error message

{"level":"error","source":"/headlamp/backend/cmd/headlamp.go","line":757,"error":"expiry time not found or invalid","time":"2024-12-06T07:41:50Z","message":"failed to get expiry time"}

when using service token without expiry (set as a secret)?

[joaquimrocha]

cc/ @ashu8912

[youvegotmoxie]

I am also facing this issue.

I noticed requests from Headlamp go directly to the API server instead of being routed through the OIDC proxy which I think needs to be done for authorization to work properly based on my own experience with GKE and OIDC:

Apr 24 20:11:59.882: headlamp/headlamp-856f4b7878-5hg54:47750 (ID:58218) -> 10.32.240.130:443 (ID:16777217) to-stack FORWARDED (TCP Flags: RST)
Apr 24 20:11:59.883: headlamp/headlamp-856f4b7878-5hg54:47750 (ID:58218) <- 10.32.240.130:443 (ID:16777217) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
Apr 24 20:11:59.883: headlamp/headlamp-856f4b7878-5hg54:47750 (ID:58218) -> 10.32.240.130:443 (ID:16777217) to-stack FORWARDED (TCP Flags: RST)
Apr 24 20:12:00.032: headlamp/headlamp-856f4b7878-5hg54:47762 (ID:58218) -> 10.32.240.130:443 (ID:16777217) to-stack FORWARDED (TCP Flags: SYN)

[BenTheElder]

I suspect this is the same as https://github.com/kubernetes-sigs/headlamp/issues/266#issuecomment-2842359584

[illume]

GKE with OIDC (GKE Identity Service) is broken currently.

However, GKE Identity Service uses access_token. So I expect a new PR merged recently into main branch will help with this:

• https://github.com/kubernetes-sigs/headlamp/pull/3177

---

There are some new flags to use access_token rather than id_token.

Note, the values for the flags below are Untested for GKE Identity Service...

-oidc-validator-client-id=YOUR-GOOGLE-WORKFORCE-CLIENT-ID
-oidc-validator-idp-issuer-url=https://accounts.google.com
-oidc-use-access-token

Some existing flags may apply to GKE Identity Service Untested

-oidc-username-claim=email
-oidc-groups-claim=groups

[mgalesloot]

I have a working setup with accounts.google.com, providing an oidc secret to headlamp helm chart with:

OIDC_CLIENT_ID=<redacted> 
OIDC_CLIENT_SECRET=<redacted> 
OIDC_ISSUER_URL='https://accounts.google.com'
OIDC_SCOPES='email,profile'

This config works fine in headlamp already for quite some time. In the helm chart I do not specify command line args for headlamp, so we are using the defaults.

In Headlamp v0.31 this config is broken. The env vars for OIDC_VALIDATOR are now required. I was able to fix this by adding to the helm chart:

env:
    - name: OIDC_VALIDATOR_CLIENT_ID
      value: $(OIDC_CLIENT_ID)
    - name: OIDC_VALIDATOR_ISSUER_URL
      value: $(OIDC_ISSUER_URL)

I think this is caused by the changes in the helm chart here: https://github.com/kubernetes-sigs/headlamp/blob/main/charts/headlamp/templates/deployment.yaml#L212

[daviddob]

I believe this is true in the event you are configuring OIDC using an external secret. The chart logic is pretty messy/complex given support for the 3+ different methods of configuring OIDC in the chart, may be worth streamlining this as a separate breaking change in the future to truly clean up the methods of config and reduce complexity.

As the option to override validation parameters is a more niche config, to resolve this regression we can drop these two lines from the chart

https://github.com/kubernetes-sigs/headlamp/blob/019ef5d39539275ada0270fa8cd8a6da0415510d/charts/headlamp/templates/deployment.yaml#L221-L222

and instead guide users to supply the flags via the following extraArgs the the event they want to leverage validation overrides while using external secrets

extraArgs:
  - "-oidc-validator-client-id=$(OIDC_VALIDATOR_CLIENT_ID)"
  - "-oidc-validator-idp-issuer-url=$(OIDC_VALIDATOR_ISSUER_URL)"

[joaquimrocha]

Ah, I have (hopefully) a fix for the validator-* here: https://github.com/kubernetes-sigs/headlamp/pull/3438