databricks-grafana using OAuth2 pass-trough

I want to use OAuth2 pass-trough as authentication method to Databricks. I test it with a datasource that I provision with the following YAML:

apiVersion: 1
datasources:
- name: databricks-alarming
  type: mullerpeter-databricks-datasource
  access: proxy
  url: ''
  isDefault: false
  database: ''
  user: ''
  password: ''
  jsonData:
    hostname: <MY_HOSTNAME>
    path: <MY_PATH>
    timeInterval: 1ms
    maxRetryDuration: 120
    authenticationMethod: oauth2_pass_through
    oauthScopes: 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default
  secureJsonData:
    token: <TOKEN>  # Not required for oauth.

In the UI, I received the following error. .

The logs read as follows:

logger=plugin.mullerpeter-databricks-datasource t=2025-11-03T09:11:48.351060578Z level=info msg="Plugin Request Completed" dsName=databricks-alarming duration=140.203µs pluginId=mullerpeter-databricks-datasource uname="<MY_NAME>" dsUid=P82544355E3A152A6 endpoint=checkHealth status=ok statusSource=plugin
logger=plugin.mullerpeter-databricks-datasource t=2025-11-03T09:11:48.350892113Z level=info msg="CheckHealth called" request="map[Headers:map[http_X-Datasource-Uid:P82544355E3A152A6 http_X-Grafana-Id:<...> http_X-Grafana-Org-Id:1] PluginContext:map[APIVersion: AppInstanceSettings:<nil> DataSourceInstanceSettings:map[APIVersion: BasicAuthEnabled:false BasicAuthUser: Database: DecryptedSecureJSONData:map[token:<...>] ID:1 JSONData:map[authenticationMethod:oauth2_pass_through hostname:<...> maxRetryDuration:120 oauthScopes:2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default path:<...> timeInterval:1ms] Name:databricks-alarming Type:mullerpeter-databricks-datasource UID:P82544355E3A152A6 URL: Updated:2025-10-30T13:53:10Z User:] GrafanaConfig:map[] OrgID:1 PluginID:mullerpeter-databricks-datasource PluginVersion:1.3.7 User:map[Email:<...> Login:<...> Name:<...> Role:Admin] UserAgent:map[]]]"
logger=plugin.mullerpeter-databricks-datasource t=2025-11-03T09:11:48.351019391Z level=info msg="Token is empty"
logger=context userId=2 orgId=1 uname=<...> t=2025-11-03T09:11:48.351173309Z level=info msg="Request Completed" method=GET path=/api/datasources/uid/P82544355E3A152A6/health status=400 remote_addr=192.168.1.1 time_ms=17 duration=17.728997ms size=103 referer=<...> handler=/api/datasources/uid/:uid/health status_source=server

Do you have an idea of what is going wrong? I use Grafana 12.2.0 and plugin version 1.3.7.

Nov 03 '25 09:11 liebsc21

Do you have any hint to troubleshoot this?

Nov 13 '25 07:11 liebsc21

Hi @liebsc21 Can you try to add oauthPassThru: true in the jsonData of your yaml?

Nov 13 '25 08:11 mullerpeter

This results in the same. However, I post the logs of the first try, which seem to be more comprehensive.

logger=plugin.mullerpeter-databricks-datasource t=2025-11-13T15:57:55.999130848Z level=info msg="Init Databricks SQL DB"
logger=plugin.mullerpeter-databricks-datasource t=2025-11-13T15:57:55.999429261Z level=info msg="Token is empty"
logger=plugin.mullerpeter-databricks-datasource t=2025-11-13T15:57:55.999151957Z level=info msg="Store Databricks SQL DB Connection"
logger=plugin.mullerpeter-databricks-datasource t=2025-11-13T15:57:55.999335796Z level=info msg="CheckHealth called" request="map[Headers:map[http_X-Datasource-Uid:P82544355E3A152A6 http_X-Grafana-Id:eyJhbGciOiJFUzI1NiIsImtpZCI6ImlkLTIwMjUtMTEtZXMyNTYiLCJ0eXAiOiJqd3QifQ.eyJhdWQiOiJvcmc6MSIsImF1dGhlbnRpY2F0ZWRCeSI6Im9hdXRoX2F6dXJlYWQiLCJlbWFpbCI6InNlYmFzdGlhbi5saWVic2NobmVyQHN1bmZpcmUuZGUiLCJleHAiOjE3NjMwNTAwNjEsImlhdCI6MTc2MzA0OTQ2MSwiaWRlbnRpZmllciI6ImJmNDBoazA1YWcycmthIiwiaXNzIjoiaHR0cHM6Ly90ZXN0LmdyYWZhbmEuc3VuZmlyZS5kZS8iLCJuYW1lIjoiU2ViYXN0aWFuIExpZWJzY2huZXIiLCJuYW1lc3BhY2UiOiJkZWZhdWx0Iiwicm9sZSI6IkFkbWluIiwic3ViIjoidXNlcjoyIiwidHlwZSI6InVzZXIiLCJ1c2VybmFtZSI6InNlYmFzdGlhbi5saWVic2NobmVyQHN1bmZpcmUuZGUifQ.V2HLxAErdUwSGsCwS-5Y3421rXYR6yRrIx6suYzilp8fWD7dft48afjwjgfrRkHEk54TwlRmXDlOqwKCmMPBWA http_X-Grafana-Org-Id:1] PluginContext:map[APIVersion: AppInstanceSettings:<nil> DataSourceInstanceSettings:map[APIVersion: BasicAuthEnabled:false BasicAuthUser: Database: DecryptedSecureJSONData:map[token:<TOKEN>] ID:1 JSONData:map[authenticationMethod:oauth2_pass_through hostname:<HOSTNAME> maxRetryDuration:120 oauthScopes:2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default path:<...> timeInterval:1ms] Name:databricks-alarming Type:mullerpeter-databricks-datasource UID:P82544355E3A152A6 URL: Updated:2025-11-13T15:11:15Z User:] GrafanaConfig:map[] OrgID:1 PluginID:mullerpeter-databricks-datasource PluginVersion:1.3.7 User:map[Email:<...> Login:<...> Name:<...> Role:Admin] UserAgent:map[]]]"
logger=plugin.mullerpeter-databricks-datasource t=2025-11-13T15:57:55.999097736Z level=info msg="ConnectionSettings Parse Error" err="json: cannot unmarshal number into Go struct field ConnectionSettingsRawJson.maxRetryDuration of type string"
logger=context userId=2 orgId=1 uname=<...> t=2025-11-13T15:57:56.00049824Z level=info msg="Request Completed" method=GET path=/api/datasources/uid/P82544355E3A152A6/health status=400 remote_addr=192.168.1.1 time_ms=10 duration=10.114036ms size=103 referer=<...>/connections/datasources/edit/P82544355E3A152A6 handler=/api/datasources/uid/:uid/health status_source=server
logger=plugin.mullerpeter-databricks-datasource t=2025-11-13T15:57:55.99946616Z level=info msg="Plugin Request Completed" pluginId=mullerpeter-databricks-datasource statusSource=plugin dsName=databricks-alarming dsUid=P82544355E3A152A6 status=ok uname="<...>" duration=661.673µs endpoint=checkHealth

Nov 13 '25 19:11 liebsc21

Do you have an idea about how to proceed? Time is a bit pressing for me.

Nov 17 '25 06:11 liebsc21

Try to setup the datasource via the UI from scratch. Do you get the same issue? The error you post seems to be related to something different (wrong type for maxRetryDuration)

Nov 17 '25 06:11 mullerpeter

In the UI, I cannot set the scope to 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default. Do I miss something?

Filling only hostname, path and auth method, gives the logs:

logger=plugin.mullerpeter-databricks-datasource t=2025-11-17T13:37:15.885881939Z level=info msg="CheckHealth called" request="map[Headers:map[Authorization:Bearer eyJ0eXAiOiJKV1QiLCJub25jZSI6IjlDaDJLZG9CakVkOFZicmo3TVNrM18xWldGano4dVlWRmlTWWh1c2hualkiLCJhbGciOiJSUzI1NiIsIng1dCI6InJ0c0ZULWItN0x1WTdEVlllU05LY0lKN1ZuYyIsImtpZCI6InJ0c0ZULWItN0x1WTdEVlllU05LY0lKN1ZuYyJ9.eyJhdWQiOiIwMDAwMDAwMy0wMDAwLTAwMDAtYzAwMC0wMDAwMDAwMDAwMDAiLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC85MjBjN2Y2Mi1kYjBhLTQ1OTAtYWI5My02NmM3ZGJmYTkwOGUvIiwiaWF0IjoxNzYzMzg1NzA3LCJuYmYiOjE3NjMzODU3MDcsImV4cCI6MTc2MzM5MDI4NywiYWNjdCI6MCwiYWNyIjoiMSIsImFjcnMiOlsicDEiXSwiYWlvIjoiQVpRQWEvOGFBQUFBM2g0dFRUVmJLeWFkTHVhNHcvU2hiRzZZbFJWVDJZWUlRbVZGNms1aHZMTlhOQTBDT1Y3cTBVQ0pZV1RoTkRSdzBFR01UZEs0dWx2dFJTTTQrRHZsTUNsbER3ZTMyUVJWaXRvZk03ZjVYeWNFQVdETy9HS3FuNnRvZ256SStDTU9pZGQxbmJpTktLYmRUL3I2ak05andrNElBM29EZzFlMDBBTFpYR21hREI0UXI1MFBYMkR2ckNiMDNIVy92KzNzIiwiYW1yIjpbInB3ZCIsIm1mYSJdLCJhcHBfZGlzcGxheW5hbWUiOiJHcmFmYW5hIC0gQXp1cmUiLCJhcHBpZCI6IjA1ZTE5Yzc2LWQ1YjUtNDNkNi1hMmIyLTNkNDM3MDg4OTJhYiIsImFwcGlkYWNyIjoiMSIsImZhbWlseV9uYW1lIjoiTGllYnNjaG5lciIsImdpdmVuX25hbWUiOiJTZWJhc3RpYW4iLCJpZHR5cCI6InVzZXIiLCJpcGFkZHIiOiI4OS4yNDcuMTY5Ljg4IiwibmFtZSI6IlNlYmFzdGlhbiBMaWVic2NobmVyIiwib2lkIjoiMmYzNDIyODQtOGVmYS00ZDU5LThhMzAtNjM5ZWUzNTlkZTFmIiwib25wcmVtX3NpZCI6IlMtMS01LTIxLTEyNTA0MDk1NC0zMTE3NDU2ODg2LTQxMjY0ODc0NC00Mzk0IiwicGxhdGYiOiIzIiwicHVpZCI6IjEwMDMyMDAwRTQ0MjZGQjQiLCJyaCI6IjEuQVhRQVluOE1rZ3Jia0VXcmsyYkgyX3FRamdNQUFBQUFBQUFBd0FBQUFBQUFBQUFfQWFKMEFBLiIsInNjcCI6ImVtYWlsIG9wZW5pZCBwcm9maWxlIiwic2lkIjoiMDBhOTY3NDktZmRiYS00YWU3LWY5YjgtMjNjNzQ4ZWFjMzBjIiwic2lnbmluX3N0YXRlIjpbImttc2kiXSwic3ViIjoiY0NGTTJxVVBwdW56ak1XMzN1aE1LVTR1RkRZY0pEUDZMaUl6ZlVhVXcxdyIsInRlbmFudF9yZWdpb25fc2NvcGUiOiJFVSIsInRpZCI6IjkyMGM3ZjYyLWRiMGEtNDU5MC1hYjkzLTY2YzdkYmZhOTA4ZSIsInVuaXF1ZV9uYW1lIjoic2ViYXN0aWFuLmxpZWJzY2huZXJAc3VuZmlyZS5kZSIsInVwbiI6InNlYmFzdGlhbi5saWVic2NobmVyQHN1bmZpcmUuZGUiLCJ1dGkiOiJBTWVEMHd2SzFVT2Y1LVJJeS0tUEFBIiwidmVyIjoiMS4wIiwid2lkcyI6WyJiNzlmYmY0ZC0zZWY5LTQ2ODktODE0My03NmIxOTRlODU1MDkiXSwieG1zX2FjZCI6MTcyMzUzNDAxOCwieG1zX2FjdF9mY3QiOiI5IDMiLCJ4bXNfZnRkIjoiT09oelFiSFpZSUtVYVFDMDB5TFdIWko2a1otODVRYngzWldQOUlvZGNib0JaWFZ5YjNCbGQyVnpkQzFrYzIxeiIsInhtc19pZHJlbCI6IjEgOCIsInhtc19zdCI6eyJzdWIiOiJFLUlRTVlxZGtuSGJpcWlFa3VPUjFIOF9ZMTBVWkYyUW1mN0NERDViU2lrIn0sInhtc19zdWJfZmN0IjoiMyAxNCIsInhtc190Y2R0IjoxNDQ4NjEzNzQwLCJ4bXNfdGRiciI6IkVVIiwieG1zX3RudF9mY3QiOiIxMCAzIn0.buBX4uXzeL2N3TbyBRkaEyjB0AK1WODpIWbri5WMkPl9QllS-J4GRuQ2HlOZppwI1z_RIPnHTir_sH_4pZdbDUML_Rj6BCbWNsfOFspjH2nmx9xzdXCocz0-MOi-W5gQxgvnd1ANH_Xi9s3ROFZfUdBOiO-8XxQchDRTX3_zTj25Wh2JxBMLoX-O-m2JJz1fdHXN2iy4QCVB9znFDs8jQgZcSU7KEAcrHA5xa02MEdvt6-HfnVpvk9Bojvt6mSsAZQY33woWhRW2uYgL0PRxIKWo780Eh8fo2ikCeqVrNh5V9ChsIjNcSvFMC_uJkBEfBQUZ9sPa8RYd5rf2bPz1og X-Id-Token:<LONG_TOKEN> http_X-Datasource-Uid:ef4edz9ybetj4b http_X-Grafana-Id:eyJhbGciOiJFUzI1NiIsImtpZCI6ImlkLTIwMjUtMTEtZXMyNTYiLCJ0eXAiOiJqd3QifQ.eyJhdWQiOiJvcmc6MSIsImF1dGhlbnRpY2F0ZWRCeSI6Im9hdXRoX2F6dXJlYWQiLCJlbWFpbCI6InNlYmFzdGlhbi5saWVic2NobmVyQHN1bmZpcmUuZGUiLCJleHAiOjE3NjMzODcxODMsImlhdCI6MTc2MzM4NjU4MywiaWRlbnRpZmllciI6ImVmNGVkeDFzb3BsYThkIiwiaXNzIjoiaHR0cHM6Ly90ZXN0LmdyYWZhbmEuc3VuZmlyZS5kZS8iLCJuYW1lIjoiU2ViYXN0aWFuIExpZWJzY2huZXIiLCJuYW1lc3BhY2UiOiJkZWZhdWx0Iiwicm9sZSI6IkFkbWluIiwic3ViIjoidXNlcjoyIiwidHlwZSI6InVzZXIiLCJ1c2VybmFtZSI6InNlYmFzdGlhbi5saWVic2NobmVyQHN1bmZpcmUuZGUifQ.Z04ry2tn3jv0lN-5ME9h48cy9R-yReLVH1Sv0hrZuzXG7eb4xQAbGoc4EMUgE759WZgKeybuC1n5dLapWBxwdw http_X-Grafana-Org-Id:1] PluginContext:map[APIVersion: AppInstanceSettings:<nil> DataSourceInstanceSettings:map[APIVersion: BasicAuthEnabled:false BasicAuthUser: Database: DecryptedSecureJSONData:<nil> ID:3 JSONData:map[authenticationMethod:oauth2_pass_through hostname:<...> oauthPassThru:true path:<...> pdcInjected:false port:443] Name:third Type:mullerpeter-databricks-datasource UID:ef4edz9ybetj4b URL: Updated:2025-11-17T13:37:15Z User:] GrafanaConfig:map[] OrgID:1 PluginID:mullerpeter-databricks-datasource PluginVersion:1.3.7 User:map[Email:<...> Login:<...> Name:<...> Role:Admin] UserAgent:map[]]]"
logger=plugin.mullerpeter-databricks-datasource t=2025-11-17T13:37:15.885984812Z level=info msg="Token updated"
logger=plugin.mullerpeter-databricks-datasource t=2025-11-17T13:37:15.885734653Z level=info msg="Init Databricks SQL DB"
logger=plugin.mullerpeter-databricks-datasource t=2025-11-17T13:37:15.8857703Z level=info msg="Store Databricks SQL DB Connection"
logger=context userId=2 orgId=1 uname=<...> t=2025-11-17T13:37:16.100077507Z level=info msg="Request Completed" method=GET path=/api/datasources/uid/ef4edz9ybetj4b/health status=400 remote_addr=192.168.1.1 time_ms=226 duration=226.946433ms size=468 referer=<...> handler=/api/datasources/uid/:uid/health status_source=server
logger=plugin.mullerpeter-databricks-datasource t=2025-11-17T13:37:16.099811207Z level=info msg="Plugin Request Completed" dsName=third dsUid=ef4edz9ybetj4b duration=214.15822ms endpoint=checkHealth pluginId=mullerpeter-databricks-datasource status=ok statusSource=plugin uname="<...>"
time=2025-11-17T13:37:35.707Z level=INFO source=/src/pkg/api/middleware/logger.go:15 msg="request complete" method=GET mux_pattern="" uri=/render/version status=200 status_text=OK duration=31.369µs
t=2025-11-17T13:37:35.707439285Z level=error caller=logger.go:234 time=2025-11-17T13:37:35.707421051Z msg="cleaning up inactive secure values" error="fetching inactive secure values that need to be cleaned up: acquiring leases for inactive secure values: leasing inactive secure values: Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'to_update AS (\n  SELECT guid FROM (\n    SELECT \n      guid,\n      ROW_NUMBER() O' at line 1"

Nov 17 '25 13:11 liebsc21

The scope has to be set in your SSO Auth Provider Config in Grafana and not in the plugin. OAuth Pass Trough only works if you setup the authentication part correctly and you are signed in via SSO. (Maybe refer to this doc: https://grafana.com/docs/grafana/latest/setup-grafana/configure-access/configure-authentication/entraid/)

Nov 17 '25 13:11 mullerpeter

Thanks for the hint! I did not set up SSO for this Grafana instance. However, if I add the scope 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default with the admin user, I cannot login with SSO anymore. Also, I noticed that the Client authentication was not selected (see screenshot) and the scope was not set initially, though I configured it in the data source provisioning file (1 data source with PAT, 1 with oauth2_pass_through) . Do you have any ideas?

Nov 17 '25 14:11 liebsc21

I don't think the issue is related to my plugin, but rather something with your SSO setup in Grafana. Where I can't be of much help.

The logs from your previous comment show that pass through is working, the plugin is receiving the token, but the scope for Databricks user impersonation is missing.

To maybe clear up a bit of confusion. The oauthScopes setting you can set in my plugin are only used when "OAuth2 Client Credentials Authentication" is set as the Auth method. That's also why you can't set them in the UI when you choose OauthPasstrough.

With passthrough these scopes already have to be added in the SSO authentication of Grafana. As the plugin just uses the Token as is and can't modify the scopes.

Nov 18 '25 03:11 mullerpeter

Ahh, thanks for the clarification. Then it really seems to be the SSO setup in Grafana and this issue can be closed. Thanks a lot for your help!

Nov 18 '25 06:11 liebsc21