Databricks Connect requires static access_token; no supported OAuth client-credentials auto-refresh for CI / Long runs

[marcelhfm]

Hi there, we are currently adapting sqlmesh + databricks into our workflows and actively migrating our dlt pipelines to sqlmesh. So far we're very pleased with working with sqlmesh!

Summary Locally we're running sqlmesh + databricks connect with long lived PATs, for our CI-workflows (or deployed workflows in general) we're currently using a databricks service principle and manually minting a short-lived OAuth access token. While this works for short runs, long runs break because of the token expiry (expiry is ~1h).

We're mainly looking for recommendations on how to deal with this situation with the current implementation. In general we're open to contributing to sqlmesh to fix this behavior, however we wanted to ensure we're not doing something wrong first, though.

Current behavior

• connection.type: databricks with Databricks Connect requires:
  • databricks_connect_server_hostname
  • databricks_connect_access_token
  • databricks_connect_cluster_id or databricks_connect_use_serverless
• The adapter creates a DatabricksSession once with the provided token. When the token expires, the session dies. No built-in refresh.

Expected behavior

• Ability to configure client-credentials OAuth for Connect in CI so tokens are minted and refreshed automatically during long SQLMesh runs, or
• An official pattern to refresh tokens and rebuild the Connect session transparently.

Repro steps Mint SP OAuth token via client id/secret. Run sqlmesh plan/apply using Connect. Let execution exceed token TTL (~1h). Observe Connect session errors due to expired token.