go-duckdb icon indicating copy to clipboard operation
go-duckdb copied to clipboard

Add support `duckdb_arrow_array_scan`

Open disq opened this issue 1 year ago • 7 comments
trafficstars

Extend the Apache Arrow interface support to expose duckdb_arrow_scan and duckdb_arrow_array_scan functions. (seems like only duckdb_query_arrow_schema is supported for now)

duckdb_arrow_scan duckdb_arrow_array_scan

This would be useful for fast data loading directly from Arrow types.

disq avatar Feb 14 '24 13:02 disq

Hi @levakin, what would be your opinion on this? Is it something we can easily add?

marcboeker avatar Feb 14 '24 18:02 marcboeker

hi @marcboeker , it should be doable. The only tricky thing is Note that after running duckdb_arrow_array_scan, duckdb_destroy_arrow_stream must be called on the out stream. this part. Maybe it can implemented using a pattern with something similar to cancel function being returned.

levakin avatar Feb 16 '24 13:02 levakin

Thanks @levakin for the evaluation. Would it make sense to add a method that returns an ArrayScanner which has to be closed afterwards?

marcboeker avatar Feb 18 '24 17:02 marcboeker

I wanted to take a look at this the other day, but didn't know where to start.

disq avatar Mar 01 '24 09:03 disq