adlfs icon indicating copy to clipboard operation
adlfs copied to clipboard

Pass on azure sdk keyword arguments

Open ptrba opened this issue 2 years ago • 0 comments

Motivation

The current implementation wraps azure-storage-blob methods without giving access to advanced keyword parameters. In rare but some cases (#375) it is necessary to tune those parameters. But, passing on all keyword arguments to azure SDK methods has potentially unintended effects and possible conflicts as discussed in (draft #364).

Suggestion here

  • There is a tool (tools/azure_sdk_kwargs.py), which checks all azure SDK methods for potential conflicts and generates a docstring for the AzureBlobFileSystem constructor. This is only a semi-automated process, but good enough. It only needs to be rerun for major changes in the azure SDK, which means touching the code anyway.
  • Known conflicting arguments are explicitly excluded from being passed on to the azure SDK methods.

This solution is sufficiently safe considering the fact that this is an advanced feature. At the same time, it is flexible enough to deal with changes in the azure SDK contracts. The docstring points to a specific version of the azure SDK. If the user decides to work with a different SDK or wants to use hidden undocumented arguments, he can use any keyword argument (except for the explicitly excluded ones) if he knows what he's doing.

This is a cleanup of the ideas discussed in draft #364.

ptrba avatar Dec 01 '22 16:12 ptrba