duckdb_azure
duckdb_azure copied to clipboard
IO Error: AzureBlobStorageFileSystem open file error reading Iceberg written by Spark and Nessie catalog
Azure file connection format parsing issue.
I have a following usecase. Writing iceberg format using Spark and Nessie using iceberg rest protocol. Nessie supports azure file access format with @ notation (e.g. abfss://warehouse@<storage_account>.dfs.core.windows.net, and hence all paths in metadata.json files are in that format.
DuckDb on the other hand supports path notation (e.g. abfss://<storage_account>.dfs.core.windows.net/warehouse) and I think because of it my query fails with following error IO Error: AzureBlobStorageFileSystem open file 'abfss://warehouse@<storage_account>.dfs.core.windows.net/some-path/some-file.avro'
I think the problem is that DuckDb doesn't handle @ notation for azure schema.
Please let me know if there is a workaround.