OAuth token caching fails in Docker/K8S

[sithson]

Describe the bug

OAuth token caching fails in Docker/K8S.

Steps To Reproduce

1. Setup dbt profile for U2M OAuth to Databricks
2. Start dbt Docker container and do ”dbt run”
3. In the web popup, authenticate yourself to Databricks
4. You will be authed
5. You will get error message for ”Cannot save auth token” and dbt run will continue as expected
6. On the next dbt run you will be informed ”Cannot load cached auth token” and you have to start over from step (3) again

Expected behavior

After step (4) above, the OAuth token will be cached for subsequent dbt runs without the need to re-auth within the token lifetime.

Screenshots and log output

root@e074d3d1b4e7:/app/dbt# dbt run
Found 666 models, 42 tests, 18 sources, 0 exposures, 0 metrics, 923 macros, 0 groups, 0 semantic models
Databricks adapter: could not retrieved saved token
Databricks adapter: could not save token

System information

The output of dbt --version:

root@e074d3d1b4e7:/app/dbt# dbt --version
Core:
  - installed: 1.7.4
  - latest:    1.7.4 - Up to date!

Plugins:
  - duckdb:     1.7.1 - Up to date!
  - databricks: 1.7.3 - Up to date!
  - spark:      1.7.1 - Up to date!

The operating system you're using:

• Windows 10
• Docker image with Ubuntu v22.04

The output of python --version:

• Python 3.10.12

Additional context

Add any other context about the problem here.

[benc-db]

@andrefurlan-db do you have any context? I would not have expected the token to be cached between run, but the output here does suggest that it is a planned behavior.

[sithson]

Is the planned behaviour to store the token?

[sithson]

The context from the user:

• Subsequent dbt runs by the the engineer without having to auth after each dbt run

[sithson]

I would not have expected the token to be cached between run, but the output here does suggest that it is a planned behavior.

Looking at the source code, it first tries to get from cache, and after successful auth it tries to store it. https://github.com/databricks/dbt-databricks/blob/main/dbt/adapters/databricks/connections.py#L361 https://github.com/databricks/dbt-databricks/blob/main/dbt/adapters/databricks/connections.py#L386

But:

• When runtime is Docker/K8S, we have no running backend services, ex. for keyring is depending.

[benc-db]

Talked to Andre offline, he suggested following the steps here for 'Using Keyring on headless Linux systems in a Docker container' could help.

[sithson]

Thx benc! Will check if it fits us, and if it works.

Talked to Andre offline, he suggested following the steps here for 'Using Keyring on headless Linux systems in a Docker container' could help.

[thijs-nijhuis]

FYI, I have remove my comment on a similar issue on windows 11 as that seems to have a different reason and I didn't want to block closing this one. I created a new issue for the windows 11 here: https://github.com/databricks/dbt-databricks/issues/563