duckdb-rs icon indicating copy to clipboard operation
duckdb-rs copied to clipboard

Duckdb updates

Open Swoorup opened this issue 1 year ago • 4 comments

  1. Strong types for duck db vectors.
  2. Ensure correct size are reserved before writing to duckdb vectors.
  3. Introduce DataChunkHandle for manipulating data chunks, and wrap unsafe functions at bay. There are still few handle like duckdb_appender that is better off being wrap instead of sprinkling unsafe everywhere.
  4. Add union, struct type and nested type without the duplicated code. (It should be simple enough to now add other nested types like Map)
  5. Appender keeps a cached copy of the schema for error handling and insertions.
  6. Fix large amount of record insertion (Respect the duckdb limit)
  7. Remove unsafe code for traits that need not be unsafe. Entire idea for rust is to limit usage of unsafe code when providing as a library.
  8. For unit testing, I also added arrow-convert library which makes it easier to derive arrow arrays directly from rust specific unions/structs than hand writing it.

Swoorup avatar Jun 11 '24 10:06 Swoorup

Fixes or partially fixes the following:

  • https://github.com/duckdb/duckdb-rs/issues/211
  • https://github.com/duckdb/duckdb-rs/issues/188
  • https://github.com/duckdb/duckdb-rs/issues/81

Swoorup avatar Jun 11 '24 10:06 Swoorup

I think the 306 update to the test case could be merged into your changes.

era127 avatar Jun 11 '24 18:06 era127

Union feature is blocked by a suitable release i.e which includes the fix https://github.com/duckdb/duckdb/pull/11326 https://github.com/duckdb/duckdb-rs/pull/336

Swoorup avatar Jun 16 '24 16:06 Swoorup

Thanks for the PR! Could you perhaps split this up into smaller chunks so that it is easier to review?

Mytherin avatar Jul 10 '24 08:07 Mytherin

NOTICE: I have abandoned this work. Anyone is welcome to take over and integrate bits and pieces they may find useful.

Swoorup avatar Sep 25 '24 11:09 Swoorup