lakeFS icon indicating copy to clipboard operation
lakeFS copied to clipboard

[Feature Request]: Support additional protocols syntax for Azure imports, e.g. `abfss://`

Open MigQ2 opened this issue 1 year ago • 4 comments

Currently importing from Azure Blob storage or ADLS only supports paths using https:// protocol.

However, for many Azure applications, the abfss:// protocol is preferred and has a different syntax (or even wasbs://)

It would be great if it could be possible to use the abfss:// syntax for creating repositories, underlying Azure paths, ot import paths.

Even if lakefs still uses https:// underneath, it would be very convenient to be able to copy-paste an abfss:// path into the import UI or in lakectl and not having to edit it manually, as it's not just changing the protocol prefix, the path syntax is quite different.

Maybe a path translator functionality from abfss:// to https:// could make this work easily.

The only thing I can think of that could give problems is the adls prefix needed for ADLS Gen2 imports, which I think is non-standard Azure syntax and I don't know if that could be explicitly expressed in an abfss:// path

image

MigQ2 avatar Jun 21 '23 23:06 MigQ2

@MigQ2 Thanks for the suggestion. We will definitely look into supporting the abfs[s] scheme. Please note the wasb was deprecated by Azure - so we are not planning on supporting this scheme. Regarding the important note you added about the import hint in lakeFS - from the Azure documentation if I understand correctly, this scheme is only available for ADLS Gen2. If that's the case, we can assume that when given this scheme we are always dealing with an ADLS Gen2 account

N-o-Z avatar Jul 11 '23 14:07 N-o-Z

@ozkatz please prioritize

idanovo avatar Aug 31 '23 08:08 idanovo

This issue is now marked as stale after 90 days of inactivity, and will be closed soon. To keep it, mark it with the "no stale" label.

github-actions[bot] avatar Nov 30 '23 01:11 github-actions[bot]

@talSofer @ozkatz needs prioritization

N-o-Z avatar Nov 30 '23 06:11 N-o-Z