duckdb_iceberg icon indicating copy to clipboard operation
duckdb_iceberg copied to clipboard

IOException when querying table with a list<int> column.

Open daturkel opened this issue 11 months ago • 4 comments

Hello, thanks for your great work on duckdb-iceberg!

I'm encountering an issue where attempting to query an iceberg table with iceberg_scan fails if the table has a column which is a list of ints (list<int>).

table_with_lists # a pyiceberg.table.Table object
table_without_lists # another pyiceberg.table.Table object

duckdb.sql("INSTALL ICEBERG; LOAD ICEBERG;")

# querying the table that doesn't feature any lists works fine
duckdb.sql(f"SELECT * FROM iceberg_scan('{table_without_lists.metadata_location}') LIMIT 10")

# querying the table that *does* have lists throws an error
duckdb.sql(f"SELECT * FROM iceberg_scan('{table_with_lists.metadata_location}') LIMIT 10")
# IOException: IO Error: Invalid field found while parsing field: type

This seems to be related to an error parsing the table schema. I was wondering if there maybe was no logic for complex types in the duckdb-iceberg extension, but it appears that there is.

daturkel avatar Mar 28 '24 15:03 daturkel