Default credentials when running on a cluster

[bsamseth]

The CLI comes pre-installed in recent runtimes, like 10.4 LTS. However, no credentials are set up by default:

When running the CLI on a cluster this could get sensible defaults. Similar to how dbutils work, where the commands you run are connected to your user, the default profile could use the current host and auto-created credentials for the executing user. I don't think this is something that can be implemented in this repo by itself, but it feels like a natural place to raise this as a possible enhancement.

This would provide programmatic access to all the APIs that are supported, filling in what you can't do with dbutils currently. Many times I've resorted to calling the REST APIs "manually" from Python in order to get at certain API features. While possible, it doesn't work well with other team members as we have to explicitly set/manage credentials in the code. A "transparent" way to use the Databricks CLI as the current user would streamline this a lot.

Let me know if this sounds like something you'd want to add 😄

[mroy-seedbox]

We do this by using secrets to store the API token in the env var used by Databricks CLI:

os.environ['DATABRICKS_TOKEN'] = dbutils.secrets.get('scope', 'databricks_api_token')

Simple enough, but your suggestion would make it even simpler! 😄

[bsamseth]

@mroy-seedbox I've done the same, but as I mentioned the downside is that you end up having to share a token for everone to use, making it hard to manage in a good way. There are certainly ways around it, but it just seems like a nice "quality of life" improvment if this was built in 😊