[FEATURE] Add Support for Mounted Locations with dbutils.fs.ls() or dbfs.exists()

[williamdphillips]

Problem Statement We are unable to use fs-related commands with mounted location. I am using the python sdk and tried using workspace_client.dbutils.fs.ls and workspace_client.dbfs.exists but both of them fail with 403 (forbidden) with no option to provide a cluster id for authentication.

I would expect that, similar to client.dbutils.fs.mount(), the cluster id specified in the config would be utilized to authenticate to the mounted location. When I run dbutils.fs.ls directly on Databricks, there is no issue with a 403.

Proposed Solution Allow the cluster id specified in the config to be used when trying to use fs-related commands with a mounted location. Or, provide some way for the user of the client to specify authentication information for the mounted location they are trying to interact with.

[williamdphillips]

As a workaround, we have decided to implement BlobServiceClient from azure-core and check status of files directly in our SA. Still, the above would be a useful feature to avoid needing a workaround.

[mgyucht]

This would be nice. However, it isn't always possible to read data from a mount point without a cluster. The direction we're going in instead is to encourage users to use Unity Catalog volumes and external locations and to use the Files API to interact with objects in your storage account. There are some blockers to this right now, namely that ls is not yet implemented in the Files API, but it is currently under development. Would this work for you?

[williamdphillips]

@mgyucht I agree, that would be a good solution as it would more natively handle the authentication piece with UC rather than needing to specify a cluster with the spark configs to allow authentication to the SA.