Dealing with arrow nulls?

[tshauck]

Hello!

I'm trying to copy an arrow record batch into a duckdb chunk via record_batch_to_duckdb_data_chunk. Generally things are working however it seems that items that should be null aren't working properly. I think on downcasting or otherwise, things are getting dropped, e.g. a null i32 becomes 0. I'm only tried the basic primitive types.

I've created a little test here: https://github.com/tshauck/duckdb-rs/commit/0d3b87b59a96189f2a6701681cca2e3bf1874a87 (I think validity shouldn't be null).

I'm happy to try to take a look, but any pointers where to start would be helpful.

[wangfenjin]

Thanks for the report. I think the issue may related to the copy function which defined in here. We maybe need to call set_null when the value is null.

It'll be great if you can help on this. Thanks!

[tshauck]

Sure! I guess I got that far, and one step further. I.e. looking at https://github.com/wangfenjin/duckdb-rs/blob/948b87902dad442ea7fb96418a5c90f931881982/src/vtab/vector.rs#L85-L88C6 it isn't clear how best to call set_null without reiterating through the vector which doesn't seem best.