duckdb_azure icon indicating copy to clipboard operation
duckdb_azure copied to clipboard

IO Error: AzureBlobStorageFileSystem open file error reading Iceberg written by Spark and Nessie catalog

Open krsma33 opened this issue 6 months ago • 2 comments

Azure file connection format parsing issue.

I have a following usecase. Writing iceberg format using Spark and Nessie using iceberg rest protocol. Nessie supports azure file access format with @ notation (e.g. abfss://warehouse@<storage_account>.dfs.core.windows.net, and hence all paths in metadata.json files are in that format.

DuckDb on the other hand supports path notation (e.g. abfss://<storage_account>.dfs.core.windows.net/warehouse) and I think because of it my query fails with following error IO Error: AzureBlobStorageFileSystem open file 'abfss://warehouse@<storage_account>.dfs.core.windows.net/some-path/some-file.avro'

I think the problem is that DuckDb doesn't handle @ notation for azure schema.

Please let me know if there is a workaround.

krsma33 avatar Aug 13 '24 11:08 krsma33