delta-rs
delta-rs copied to clipboard
delta-io
Azure SAS support
Description
Currently, authentication to Azure ADLS Gen2 is supported only via the storage account key. As the key is very powerful, in many cases it would be more secure to use SAS tokens for more restricted access.
Use Case
Alternative, more secure way of authenticating to Azure ADLS Gen2 service.
Related Issue(s)
- #767
I could be completely of base but I am a little confused how SAS keys could be used in this context, as unless I'm mistaken they contain a single object path hard-coded. They are effectively pre-signed URLs if I'm not mistaken.
Perhaps an alternative might be to support token auth, which will be included in the next object_store release. This allows creating an app in Active Directory, and then associating roles with the corresponding Service Principal. This allows fine-grained access control.
Good point, my question may well be infeasible/ignorant. I have to admit I don't know this project's codebase well enough but aren't all storage requests communicated through REST API calls? Does the ADLS2 interface rule out passing SAS tokens in the request or something... It seems that something similar to SAS has been supported in earlier versions [1] I didn't search the git history, though.
Thanks also for the alternative suggestion with token auth / AD, too.
[1] StackOverflow: How to read from Azure Blob Storage with Python delta-rs
So I think you can generate SAS keys at the container level, and not the specific blob, so your request should be possible, ignore me :smile:. FWIW token auth is more flexible, and I would argue the better way to handle this - if only because SAS appears to be some Azure-specific quirk, but we should support both :+1:
It seems that something similar to SAS has been supported in earlier versions
It is possible the switch to using object_store lost support for this, I'm not familiar enough with this delta-rs. I maintain object_store, not this crate :smile:
Account key authentication is not available in my use case, so it appears I can't use deltalake until SAS/token support is available.
Is there a workaround?
object_store 0.5 which is in the process of being released includes support for SAS tokens.
In the short-term you may be able to use patch.crates-io to update to a new revision, as there haven't been any breaking changes.
I just merged a PR to allow passing in the "raw" SAS key and forwarding it to object_store. This will be available in the next python release.
