Cannot ATTACH to Unity Catalog (via HTTPS rest)

[akomisarek]

Hello!

I wanted to test Iceberg Extension with Unity Catalog as a Iceberg Catalog and facing issues with attaching:

 ATTACH 'warehouse' AS iceberg_catalog (
     TYPE iceberg,
     SECRET iceberg_secret, -- pass a specific secret name to prevent ambiguity
     ENDPOINT 'https://{workspace}.cloud.databricks.com/api/2.1/unity-catalog/iceberg-rest'
  );

returns:

Invalid Error:
'https' scheme is not supported.

This is taken from the documentation which states:

ATTACH 'warehouse' AS iceberg_catalog (
   TYPE iceberg,
   SECRET iceberg_secret, -- pass a specific secret name to prevent ambiguity
   ENDPOINT https://rest_endpoint.com
);

which seems invalid as without quotes it doesn't work at all:

D ATTACH 'warehouse' AS iceberg_catalog (
     TYPE iceberg,
     SECRET iceberg_secret, -- pass a specific secret name to prevent ambiguity
     ENDPOINT https://rest_endpoint.com
  );
Parser Error:
syntax error at or near ":"

I tried without https but it gives:

Unknown error for HTTP GET to 'http://{workspace}.cloud.databricks.com/api/2.1/unity-catalog/iceberg-rest/v1/config?warehouse=new_test'

Steps to reproduce:

1. Install duckdb via: curl https://install.duckdb.org | sh
2. Install and load Iceberg extension:

install iceberg;
load iceberg;

[Tishj]

Hmm I think I traced this down to the default httplib-backed HTTPUtil implementation in core. If you LOAD httpfs; before the ATTACH I think it should work, as it replaces this default implementation

[akomisarek]

Thank you for your prompt response. I can confirm, indeed, it seems I can connect to the catalogue now! I wouldn't figure it out!

Sorry, another Quick question if I may, my understanding is DuckDB Iceberg should support credentials vending and so does Unity, but have anyone tested them together? I can Vend via Spark, but from DuckDB it doesn't seem to work :( not sure how to debug this best...

D select * from iceberg_catalog.db.unity_example_from_flink;
HTTP Error:
HTTP GET error on 'https://{bucket}.s3.amazonaws.com/users/adam.komisarek/unity/new_test/__unitystorage/catalogs/bf37add7-0fe2-4517-ad1b-16c0fd97fd4e/tables/774a57bf-d308-4c43-bd35-98bb670e9ed1/1000/0010/1000/01011011/774a57bf-d308-4c43-bd35-98bb670e9ed1/_iceberg/00000-0-227dffa4-639c-4708-a9c4-e07a797fab27-00006.parquet' (HTTP 403)

I was testing the debug logs:

       message = {'request': {'type': GET, 'url': 'https://{workspace}.cloud.databricks.com/api/2.1/unity-catalog/iceberg-rest/v1/catalogs/new_test/namespaces/db/tables/unity_example_from_flink', 'headers': {Authorization=Bearer {token}, X-Iceberg-Access-Delegation=vended-credentials, User-Agent='duckdb/v1.3.1(osx_arm64) cli 2063dda3e6'}}, 'response': {'status': OK_200, 'reason': OK, 'headers': {vary=Accept-Encoding, content-length=7282, strict-transport-security='max-age=31536000; includeSubDomains; preload', x-databricks-org-id=2468605022034679, server=databricks, x-request-id=491524d7-8db1-474f-a732-ea23bd136b92, date='Tue, 17 Jun 2025 15:40:46 GMT', x-content-type-options=nosniff, content-type=application/json}}}

Afaik Unity Catalog has dedicated temporary-table-credentials endpoint, but I don't see it being called. In order to get this to work, I need to supply S3 credentials on DuckDB side, but my understanding is it shouldn't be required, maybe I miss some extra setup step, but can't find anything mentioned in the docs. Any pointers, much appreciated!

Do you want me to close this issue, or could we use as one to enhance docs? :)

[Tishj]

That has not been tested

dedicated temporary-table-credentials endpoint

That sounds like we won't support it, I'd really like to not have special case behavior everywhere, REST Catalog implementations should adhere to the spec

or could we use as one to enhance docs? :)

I agree this should be in the docs as a NOTE on the authorization/connecting page 👍

[akomisarek]

Fully agreed with avoiding the workarounds, I will also reach out to Databricks folks if they have an idea/consider fixing this.

Thanks for your help! I will check if I have spare time and if no PR is opened for updating the docs, I will find where to do it, and open on my own!

[Tishj]

It's possible that we are missing support for parts of the API/spec, so if that's the case we can look into implementing the missing support 👍

[nicornk]

Invalid Error: 'https' scheme is not supported.

Hmm I think I traced this down to the default httplib-backed HTTPUtil implementation in core. If you LOAD httpfs; before the ATTACH I think it should work, as it replaces this default implementation

Hi @Tishj , I recently ran into this issue while testing reads from another IRC Implementation

Should I open a dedicated issue here or on the duckdb repository? Happy to try a fix if you give some pointers where httpfs would need to be loaded explicitly when the url contains https.

[carlopi]

I think this has now properly solved, could you try again with any recent version (v1.4.2 I think for sure, likely also v1.4.0 / v1.4.1 but not completely sure)?