duckdb-rs
duckdb-rs copied to clipboard
Duckdb updates
- Strong types for duck db vectors.
- Ensure correct size are reserved before writing to duckdb vectors.
- Introduce
DataChunkHandlefor manipulating data chunks, and wrap unsafe functions at bay. There are still few handle likeduckdb_appenderthat is better off being wrap instead of sprinkling unsafe everywhere. - Add union, struct type and nested type without the duplicated code. (It should be simple enough to now add other nested types like
Map) - Appender keeps a cached copy of the schema for error handling and insertions.
- Fix large amount of record insertion (Respect the duckdb limit)
- Remove unsafe code for traits that need not be unsafe. Entire idea for rust is to limit usage of unsafe code when providing as a library.
- For unit testing, I also added arrow-convert library which makes it easier to derive arrow arrays directly from rust specific unions/structs than hand writing it.
Fixes or partially fixes the following:
- https://github.com/duckdb/duckdb-rs/issues/211
- https://github.com/duckdb/duckdb-rs/issues/188
- https://github.com/duckdb/duckdb-rs/issues/81
I think the 306 update to the test case could be merged into your changes.
Union feature is blocked by a suitable release i.e which includes the fix https://github.com/duckdb/duckdb/pull/11326 https://github.com/duckdb/duckdb-rs/pull/336
Thanks for the PR! Could you perhaps split this up into smaller chunks so that it is easier to review?
NOTICE: I have abandoned this work. Anyone is welcome to take over and integrate bits and pieces they may find useful.