Eclipse che Dashboard issue (devworkspacetemplates and devworkspace 401)

[rakeshreddyrg09]

Describe the bug

Hi All, I am installing eclipse che on my k3d k8s cluster using helm chart next version. Install is Successful and able to hit the dashboard URL, but once the oidc authentication is verified and the callback is also successful but I am getting below error Failed to fetch available workspaces, reason: Failed to fetch the list of devWorkspaces. Unable to list devworkspaces: unauthorized

here is my CheCluster manifest

I am using NFS as my storage class.

apiVersion: org.eclipse.che/v1
kind: CheCluster
metadata:
  name: eclipse-che
  namespace: eclipse-che
spec:
  auth:
    externalIdentityProvider: true
    openShiftoAuth: false
    identityProviderURL: "https://test.eu.auth0.com/"   ##  changed for security reasons
    oAuthClientName: "abc"   ##  changed for security reasons
    oAuthSecret: "xyz"   ##  changed for security reasons
  database:
    chePostgresDb: ""
    chePostgresHostName: ""
    chePostgresPort: ""
    chePostgresSecret: ""
    externalDb: false
  devWorkspace:
    enable: true
  k8s:
    ingressClass: kong  ## changed for security reasons
    ingressDomain: "eclipse-che.test-sample.com"    ##  changed for security reasons
    tlsSecretName: che-tls
  metrics:
    enable: true
  server:
    customCheProperties:
      CHE_OIDC_USERNAME__CLAIM: "email"
    cheClusterRoles: ""
    tlsSupport: true
    # cheLogLevel: "DEBUG"
    cheWorkspaceClusterRole: ""
    gitSelfSignedCert: false
    nonProxyHosts: ""
    proxyPort: ""
    proxySecret: ""
    proxyURL: ""
    workspaceNamespaceDefault: <username>-che
  storage:
    postgresPVCStorageClassName: "nfs"
    preCreateSubPaths: true
    pvcClaimSize: 10Gi
    pvcStrategy: common
    workspacePVCStorageClassName: "nfs"




### Che version

next (development version)

### Steps to reproduce

NA

### Expected behavior

NA

### Runtime

Kubernetes (vanilla), other (please specify in additional context)

### Screenshots

![che-error](https://user-images.githubusercontent.com/48629246/174765810-d494aa47-93ee-40bd-b5c7-da5a80423fb7.png)
![chedashboard](https://user-images.githubusercontent.com/48629246/174766040-63b3b80a-25ff-466c-a7c1-5a575e76fa22.png)



### Installation method

chectl/next

### Environment

Linux

### Eclipse Che Logs

_No response_

### Additional context

_No response_

[rakeshreddyrg09]

@Divine1 please have look

[Divine1]

@rakeshreddyrg09

if you take a look at this question , you will notice the details that are provided for analysis. could you share those information as well?

[rakeshreddyrg09]

@Divine1 here are the screenshots.

[Divine1]

@rakeshreddyrg09

in your Checluster manifest, i can see apiVersion: org.eclipse.che/v1.

The latest version is apiVersion: org.eclipse.che/v2

i would suggest you to use the latest version. below fix is applicable for apiVersion: org.eclipse.che/v2

in your Checluster manifest, i can see CHE_OIDC_USERNAME__CLAIM: "email"

in below link i have detailed my observation on adding CHE_OIDC_USERNAME__CLAIM: "email" and how i fixed the issue https://github.com/eclipse/che/issues/21477

i would suggest you to read through all the details present in this link

[rakeshreddyrg09]

When I apply v2 version facing below issue

*ERROR controller-runtime.manager.controller.checluster Reconciler error {"reconciler group": "org.eclipse.che", "reconciler kind": "CheCluster", "name": "eclipse-che", "namespace": "eclipse-che", "error": "Get "https://eclipse-che.sample.com": dial tcp: lookup eclipse-che.sample.com on 10.43.0.10:53: no such host"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem

Here the manifest i am using

  apiVersion: org.eclipse.che/v2
  kind: CheCluster
  metadata:
    name: eclipse-che
    namespace: eclipse-che
  spec:
    devEnvironments:
      defaultNamespace:
        template: '<username>-che'
      storage:
        pvcStrategy: 'common'
    components:
      cheServer:
        extraProperties:
          CHE_OIDC_USERNAME__CLAIM: email
      database:
        externalDb: false
      metrics:
        enable: true
    networking:
      domain: 'eclipse-che.sample.com'
      tlsSecretName: ''
      auth:
        identityProviderURL: 'https://test.org.auth0.com/'
        oAuthClientName: 'abc'
        oAuthSecret: 'xyz'

[Divine1]

@rakeshreddyrg09 i think there are some stale files in the k8s cluster. i'm not really sure about that.

is it possible for you to create a new k8s cluster and install eclipse che version chectl/0.0.20220613-next.df3ae2c darwin-x64 node-v16.13.2 in the new k8s cluster?

please dont forget to follow the details mentioned in this link

[azatsarynnyy]

cc @ibuziuk

[rakeshreddyrg09]

In v2 version I am not able to find where to give my ingress class name. Can anyone help me. Thank you. @sparkoo @azatsarynnyy @tolusha

[tolusha]

@rakeshreddyrg09 It is not possible to specify ingress class separately, but it is possible by setting annotations for ingress:

spec:
  networking:
    annotations:
      kubernetes.io/ingress.class:  "nginx"
      nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
      nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
      nginx.ingress.kubernetes.io/ssl-redirect: "true"
      nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"
      nginx.org/websocket-services: "che-gateway"

Check you current ingress annotations not to miss anything

[rakeshreddyrg09]

I have added below lines

networking:
    annotations:
      kubernetes.io/ingress.class:  "nginx"

in https://github.com/eclipse-che/che-operator/blob/main/helmcharts/next/templates/org_v2_checluster.yaml and values file then its started working.

But still when i try to access the dashboard getting the same error.

Failed to fetch available workspaces, reason: Failed to fetch the list of devWorkspaces. Unable to list devworkspaces: unauthorized 401

[sparkoo]

is OIDC server, you're using to authenticate Che, same OIDC as Kubernetes is using?

[rakeshreddyrg09]

sorry i didn't understand your question, but my OIDC is hosted in organization level and che is running on local k8s cluster on my laptop.

[sparkoo]

Do you have k8s cluster configured with OIDC Authentication? https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens

[rakeshreddyrg09]

no i don't have, I am using a normal k3d cluster. do I have to enable oidc oauth for my cluster for this to work

[sparkoo]

yes. Che is using OIDC token to talk with Kubernetes API. So Kubernetes has to be configured with same OIDC to understand the token.

[rakeshreddyrg09]

i will check on this and update @sparkoo , thank you for your help.

[sparkoo]

You can get inspired how we're doing that for minikube in chectl https://github.com/che-incubator/chectl/blob/main/src/tasks/platforms/minikube.ts#L135

[olexii4]

@rakeshreddyrg09 Do you still have this '401' problem?

[rakeshreddyrg09]

yes

[timosur]

I am also experieing the same error. I am using Eclipse Che (latest next version) on AKS with Azure AD enabled. I have configured my CheCluster in the following way:

components:
    cheServer:
      debug: false
      extraProperties:
        CHE_OIDC_AUTH__SERVER__URL: https://login.microsoftonline.com/[HIDDEN]/v2.0/
        CHE_OIDC_EMAIL__CLAIM: unique_name
        CHE_OIDC_OIDC__PROVIDER: https://login.microsoftonline.com/[HIDDEN]/v2.0/
        CHE_OIDC_USERNAME__CLAIM: unique_name
      logLevel: INFO
    dashboard: {}
    database:
      credentialsSecretName: postgres-credentials
      externalDb: false
      postgresDb: dbche
      postgresHostName: postgres
      postgresPort: "5432"
      pvc:
        claimSize: 1Gi
    devWorkspace: {}
    devfileRegistry: {}
    imagePuller:
      enable: false
      spec: {}
    metrics:
      enable: true
    pluginRegistry: {}
  containerRegistry: {}
  devEnvironments:
    defaultNamespace:
      template: <username>-che
    secondsOfInactivityBeforeIdling: 1800
    secondsOfRunBeforeIdling: -1
    storage:
      pvcStrategy: common
  networking:
    auth:
      gateway:
        configLabels:
          app: che
          component: che-gateway-config
      identityProviderURL: https://login.microsoftonline.com/[HIDDEN]/v2.0/
      identityToken: access_token
      oAuthClientName: [HIDDEN]
      oAuthScope: openid email profile api://[HIDDEN]/user.read
      oAuthSecret: [HIDDEN]
    domain: [HIDDEN]
    tlsSecretName: che-tls

I have followed the following Tutorial in your Blog: https://che.eclipseprojects.io/2022/07/25/@karatkep-installing-eclipse-che-on-aks.html

But still receiving the error described in this bug.

Anyone can help? Also happy to contribute, but I have no idea where the bug could be located.

[timosur]

I just found the code where it is crashing: https://github.com/eclipse-che/che-dashboard/blob/f934c42c2cccca7ea5b6d36590032ea0d867a52a/packages/dashboard-backend/src/devworkspace-client/services/api/workspaceApi.ts#L36 Problem seems to be, that the request is being declined by the k8s API server. But I am not able to debug the issue any further as debugging level is not configurable for the Dashboard service. @olexii4 I saw that you have done a lot of stuff in the Dashboard service. Are you maybe able to assist here?

[tolusha]

@timosur I am not sure if it is needed to both:

CHE_OIDC_EMAIL__CLAIM: unique_name
CHE_OIDC_USERNAME__CLAIM: unique_name

[timosur]

@tolusha I have also tried to only use CHE_OIDC_EMAIL__CLAIM: unique_name, as described in the Blog (https://che.eclipseprojects.io/2022/07/25/@karatkep-installing-eclipse-che-on-aks.html#how-did-we-solve-the-third-challenge) But without any success.

[tolusha]

Pls, remove user's namespace before trying a different configuration, since RoleBindings won't be updated automatically.

[timosur]

@tolusha I have tried a fresh install with only CHE_OIDC_EMAIL__CLAIM: unique_name set. But still receiving 401 for /devworkspaces and /devworkspacetemplates

[gustavoromerobenitez]

I experience the same 401 errors in GKE, after a fresh install and successful authentication with the external OIDC (I've tried both Google accounts and an external Keycloak). Just like the other reports above, I access the dashboard and navigate to the Workspaces or Create Workspace pages and see the red pop-up on the top right of the screen. My environment:

• Che 7.52 installed via chectl in new GKE cluster with identity service enabled.

chectl server:deploy --installer operator \
                     --telemetry off \
                     --che-operator-cr-patch-yaml patch.yaml \
                     --platform k8s \
                     --domain ${DOMAIN} \
                     --skip-oidc-provider-check

• OIDC has been configured in the cluster following this document. However, the document does not specify that once the identity-service is enabled, all authenticated API calls should go through the gke-oidc-envoy proxy (Working configuration below).
• Che has been configured to use the same OIDC provider as GKE
• I'm able to log in to the dashboard using the OIDC provider.
• The OIDC Client Config has been tested using kubectl oidc login --cluster=${CLUSTER_NAME} --login-config=gke-oidc-login-google.yaml followed by kubectl get pods.

che-gateway/oauth-proxy errors:

GET / "/dashboard/api/namespace/<user-namespace>/devworkspaces" HTTP/1.1 "Mozilla/5.0 (X11; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/103.0" 401 96 0.028
GET / "/dashboard/api/namespace/<user-namespace>/devworkspacetemplates" HTTP/1.1 "Mozilla/5.0 (X11; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/103.0" 401 105 0.037

GKE's gke-oidc-envoy logs for the same request show no errors:

"GET /api/v1/namespaces?labelSelector=app.kubernetes.io%2Fpart-of%3Dche.eclipse.org%2Capp.kubernetes.io%2Fcomponent%3Dworkspaces-namespace HTTP/1.1" 200 - 0 94 13 8 "-" "okhttp/3.12.12" "150932bd-7fd9-4a59-874a-32c94ad1c926" "gke-oidc-envoy.anthos-identity-service" "10.56.0.1:443"
"GET /api/v1/namespaces?labelSelector=app.kubernetes.io%2Fpart-of%3Dche.eclipse.org%2Capp.kubernetes.io%2Fcomponent%3Dworkspaces-namespace HTTP/1.1" 200 - 0 94 4 3 "-" "okhttp/3.12.12" "6947e1ea-7c04-4ad9-977f-cd8ffd05139c" "gke-oidc-envoy.anthos-identity-service" "10.56.0.1:443"
"GET /api/v1/namespaces/<user-namespace> HTTP/1.1" 200 - 0 674 5 4 "-" "okhttp/3.12.12" "bcf7a82b-9108-4a28-8109-a024328852ce" "gke-oidc-envoy.anthos-identity-service" "10.56.0.1:443"
"GET /api/v1/namespaces?labelSelector=app.kubernetes.io%2Fpart-of%3Dche.eclipse.org%2Capp.kubernetes.io%2Fcomponent%3Dworkspaces-namespace HTTP/1.1" 200 - 0 94 6 4 "-" "okhttp/3.12.12" "b6f1bfaa-e0e2-43ee-9d38-17bf0099adbf" "gke-oidc-envoy.anthos-identity-service" "10.56.0.1:443"
"GET /api/v1/namespaces?labelSelector=app.kubernetes.io%2Fpart-of%3Dche.eclipse.org%2Capp.kubernetes.io%2Fcomponent%3Dworkspaces-namespace HTTP/1.1" 200 - 0 94 6 3 "-" "okhttp/3.12.12" "34080239-8af9-4806-9a78-b82e6f2a68a4" "gke-oidc-envoy.anthos-identity-service" "10.56.0.1:443"
"GET /api/v1/namespaces/<user-namespace> HTTP/1.1" 200 - 0 674 4 3 "-" "okhttp/3.12.12" "a07fd071-d285-493f-8ab7-2c2222670292" "gke-oidc-envoy.anthos-identity-service" "10.56.0.1:443"

CheCluster configuration:

spec:
  networking:  
    annotations:
      kubernetes.io/ingress.class:  "nginx"
      nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
      nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
      nginx.ingress.kubernetes.io/ssl-redirect: "true"
      nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"
      nginx.org/websocket-services: "che-gateway"
    tlsSecretName: che-tls
    domain: ${DOMAIN}
    auth:
      identityProviderURL: https://accounts.google.com
      identityToken: id_token
      oAuthClientName: ${oauth_client_id}
      oAuthSecret: ${oauth_client_secret}
      oAuthScope: email     
      openShiftoAuth: false    
  components:    
    cheServer:
      debug: true
      logLevel: DEBUG
    extraProperties:
        CHE_OIDC_EMAIL__CLAIM: email
        CHE_OIDC_USERNAME__CLAIM: email
        CHE_INFRA_KUBERNETES_MASTER__URL: https://gke-oidc-envoy.anthos-identity-service
        
    dashboard:
      deployment:
        containers:
          - env:
              - name: CHE_INFRA_KUBERNETES_MASTER__URL
                value: https://gke-oidc-envoy.anthos-identity-service
              - name: CHE_INFRA_KUBERNETES_PORT
                value: tcp://${GKE_OIDC_ENVOY_IP}:443
              - name: CHE_INFRA_KUBERNETES_PORT_443_TCP_ADDR
                value: ${GKE_OIDC_ENVOY_IP}
              - name: CHE_INFRA_KUBERNETES_PORT_443_TCP
                value: tcp://${GKE_OIDC_ENVOY_IP}:443
              - name: CHE_INFRA_KUBERNETES_SERVICE_HOST
                value: ${GKE_OIDC_ENVOY_IP}

• GKE OIDC Configuration:

  apiVersion: authentication.gke.io/v2alpha1
  kind: ClientConfig
  metadata:
    name: default
    namespace: kube-public
  spec:
    name: ${CLUSTER_NAME}
    server: https://${GKE_OIDC_ENVOY_IP}:443
    certificateAuthorityData: ${GKE_OIDC_ENVOY_CA}
    authentication:
      - name: oidc
        oidc:
          clientID: ${oauth_client_id}
          clientSecret: ${oauth_client_secret}
          extraParams: resource=token-groups-claim,prompt=consent,access_type=offline
          issuerURI: https://accounts.google.com
          cloudConsoleRedirectURI: https://console.cloud.google.com/kubernetes/oidc
          kubectlRedirectURI: http://localhost:10000/callback
          scopes: email
          userClaim: email
          groupsClaim: groups

I'd appreciate if you can prioritize investigating this issue, given that the failure point has already been highlighted by @timosur. Please let me know if you need any more details or logs.

Once I get Che 7.52 to work on GKE, I'll happily contribute with a GKE installation process blog similar to the excellent blog about Che installation on AKS.

[sreeramu]

I am also getting 401 error for /dashboard/api/namespace/user1-che/devworkspacetemplates , /dashboard/api/namespace/user1-che/devworkspaces with minikube setup is anyone is able to get the fix for it.?

I have this issue with 7.53.0 but with 7.54.0-next version it is working without this error.

[gustavoromerobenitez]

The errors persist on version 7.54 deployed on GKE via chectl.

[timosur]

@Divine1 @tolusha @sparkoo Any update? Problem still blocks us from using Eclipse Che in our Azure AKS RBAC Azure AD joined setup.

[Divine1]

@Divine1 @tolusha @sparkoo Any update? Problem still blocks us from using Eclipse Che in our Azure AKS RBAC Azure AD joined setup.

Have you tried this approach? https://github.com/eclipse/che/issues/21477#issuecomment-1160516560