[Bug] GrafanaDashboard import does not work when oauth is enabled

[manvindar]

Describe the bug When we create a grafana dashboard custom resource, it is not able to import dashboard and I get an error in events stating "error getting folders, expected status 200 but got 401". I believe it is not able to hit /api/folders after enabling oauth.

here's the full log of controller manager

2022-04-21T14:34:18.339Z        ERROR   failed to get or create namespace folder for dashboard  {"folder": "default", "dashboard": "", "error": "error getting folders, expected status 200 but got 401"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/log.(*DelegatingLogger).Error
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/log/deleg.go:144
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).reconcileDashboards
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:283
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).Reconcile
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:101
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.SetupWithManager.func1
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:184
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.SetupWithManager.func2
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:193
2022-04-21T14:34:18.339Z        ERROR   error updating dashboard        {"error": "error getting folders, expected status 200 but got 401"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/log.(*DelegatingLogger).Error
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/log/deleg.go:144
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).manageError
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:468
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).reconcileDashboards
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:284
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).Reconcile
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:101
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.SetupWithManager.func1
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:184
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.SetupWithManager.func2
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:193

Version Full semver version of the operator being used v4.0.1

Expected behavior Dashboard should be imported in grafana

Suspect component/Location where the bug might be occuring Please provide this if you know where this bug might occur otherwise leave as unknown

Screenshots

Runtime (please complete the following information):

• Grafana Operator Version - 4
• Environment: Kubernetes
• Deployment type: AWS, local

[nissessenap]

@manvindar can you update to the latest version of the operator and try again? Allot have changed since v4.0.0.

[manvindar]

@NissesSenap I tried the latest version and got the same error

[nissessenap]

@manvindar can you share how your yaml of your grafana and your dashboard looks like? We need a way of reproducing this issue. I think it sounds strange that the admin account that we use to talk to the API would get some issues when using oauth since it's a bearer token. So unless you have enforced something in grafana that makes that impossible to use I think shouldn't be a problem.

[manvindar]

hi @NissesSenap Please find the yamls below

###################### Creating GRAFANA for Omara cluster   ###########################################
apiVersion: integreatly.org/v1alpha1
kind: Grafana
metadata:
  name: basic
  labels:
    grafana: omara-monitoring-grafana-basic
spec:
  config:
    server:
      domain: "grafana.DOMAIN_NOT_DEFINED"
      root_url: "https://grafana.DOMAIN_NOT_DEFINED"
      enforce_domain: true
    auth:
      disable_login_form: true
      oauth_auto_login: true
      signout_redirect_url: https://id.DOMAIN_NOT_DEFINED/auth/realms/omara-core/protocol/openid-connect/logout?redirect_uri=https://grafana.DOMAIN_NOT_DEFINED/login
    auth.generic_oauth:
      enabled: true
      allow_sign_up: true
      client_id: "omara-monitoring-grafana-basic"
      client_secret: "SECRET_NOT_DEFINED"
      scopes: "openid profile email"
      role_attribute_path: "realm_access | contains(roles[*], 'platform-administrators') && 'Admin' || 'Viewer'"
      auth_url: "https://id.DOMAIN_NOT_DEFINED/auth/realms/omara-core/protocol/openid-connect/auth"
      token_url: "https://id.DOMAIN_NOT_DEFINED/auth/realms/omara-core/protocol/openid-connect/token"
      api_url: "https://id.DOMAIN_NOT_DEFINED/auth/realms/omara-core/protocol/openid-connect/userinfo"
    security:
      admin_user: "admin"
      admin_password: "admin"
      allow_embedding: true
      cookie_samesite: none
      cookie_secure: true
    smtp:
      enabled: true
      host: "mail-service:25"
      skip_verify: true
      from_address: [email protected]
      from_name: Grafana
  dataStorage:
    class: hostpath
    accessModes:
      - ReadWriteOnce
    size: 10Gi
  deployment:
    securityContext:
      fsGroup: 472
      runAsUser: 472
      runAsGroup: 472
    replicas: 1
    labels:
      app: "omara-monitoring-grafana"
  dashboardLabelSelector:
    - matchExpressions:
        - { key: app, operator: In, values: [ grafana ] }
  resources:
    requests:
      cpu: 500m
      memory: 50Mi

apiVersion: integreatly.org/v1alpha1
kind: GrafanaDashboard
metadata:
  labels:
    app: grafana
  name: omara-infrastructure
spec:
  json: |
        {
          ...
        }

[manvindar]

Hi @NissesSenap

do you have any update on this?

[nissessenap]

@manvindar the operator is rather simple, it works by using the admin account, and grabs the credentials.

Then it just goes towards the API. I don't know enough about grafana generic oauth solution but it wouldn't surprise me if it's possible to disable basic auth towards the API and enforce oauth 100%.

If this would happen you would break the operator.

Could you please verify that you can reach the API using the basic auth solution?

Below you can find a simple example on how to do it taken from our e2e tests.

NAMESPACE="grafana-operator-system"
HEADER='-H Accept:application/json -H Content-Type:application/json'
PASSWORD=$(kubectl -n $NAMESPACE get secrets grafana-admin-credentials --template={{.data.GF_SECURITY_ADMIN_PASSWORD}} | base64 -d)
kubectl port-forward -n $NAMESPACE service/grafana-service 3000:3000 &

# Check to see that your grafana instance is up:
curl localhost:3000/api/health
# Check to see if we can talk to the folders grafana API
curl $HEADER "http://admin:$PASSWORD@localhost:3000/api/search?folderIds=0"

[manvindar]

hi @NissesSenap I get the below error, it is trying to redirect to public endpoint

[nissessenap]

@manvindar please update the client config to prefer the service.

grafana
  spec:
    client::
      preferService: true

This should probably be more highlighted in the docs. Would love to get a PR with this information if it solves the issue for you.

[manvindar]

@NissesSenap I tried that also, but it doesn't seem to affect anything

[raviqqe]

@manvindar Did you check this already? https://github.com/grafana-operator/grafana-operator/issues/659#issuecomment-1009923994

[manvindar]

yes, I tried that too, but it's not working, also in that example, they are using anonymous auth which works anyways. If I curl API after enabling OAuth using the service name then it redirects me to the ingress url as I commented above.

[manvindar]

@NissesSenap any update on this?

[raviqqe]

Is it possible that you deleted the original admin account created by the operator? I think you need to keep it in the Grafana instance although you may not be using it manually.

[pb82]

@manvindar can you confirm that the admin credential secret is still available and that the operator is using those credentials?

[weisdd]

@manvindar I've had an oidc-enabled grafana for over a year, and it all worked just fine. I think the issue here is due to two parameters that you set in the server section:

spec:
  config:
    server:
      domain: "grafana.DOMAIN_NOT_DEFINED" # <---- remove it
      root_url: "https://grafana.DOMAIN_NOT_DEFINED"
      enforce_domain: true # <---- remove it

If you remove those two, grafana should not redirect operator's requests to the main domain anymore, and everything should work just fine. Better to combine it with @NissesSenap recommendation:

spec:
  client:
    preferService: true

[pb82]

@manvindar would the solution from @weisdd work for you?

[github-actions[bot]]

This issue hasn't been updated for a while, marking as stale, please respond within the next 7 days to remove this label