duckdb-rs icon indicating copy to clipboard operation
duckdb-rs copied to clipboard

How to insert lists, maps, and structs?

Open Timmmm opened this issue 3 years ago • 11 comments

Maybe I missed it, but I couldn't see how to insert a row if one of the columns is e.g. TEXT[]. I naively tried params![a_vec_of_str] but ToSql is only implemented for Vec<u8> (I guess for blobs).

The DuckDB docs aren't super clear but it looks like you're supposed to use list_create('a', 'b', 'c') but I'm not sure how to do that with params![].

Timmmm avatar Sep 10 '22 15:09 Timmmm

Haven’t been supported in rust yet

wangfenjin avatar Sep 11 '22 00:09 wangfenjin

@wangfenjin appreciate your work! Just want to insert here that missing insert of lists is a deal breaker for my projects. Hope to see that implemented soon. Cheers!

digizeph avatar Oct 31 '22 21:10 digizeph

Recent support for Union types too. 👍🏼 Natural fits with Rust's enum.

Swoorup avatar Nov 16 '22 10:11 Swoorup

Hope this feature can be supported. If you don't have time, I can write a PR for that.

neverchanje avatar Dec 04 '22 12:12 neverchanje

@neverchanje it'll be great if you can help. Thanks

wangfenjin avatar Dec 04 '22 13:12 wangfenjin

Would be great if we could also be able to retrieve SRUCT, MAP, ENUMs and UNION along with being able to insert them. I think the insert just needs string representation vs the retrieval needs to be convert from arrow format?

Am I correct @wangfenjin ?

Swoorup avatar Dec 04 '22 14:12 Swoorup

Correct.

  1. If you want to support bind struct or other type as params, you may need to update bind_params
  2. If you want to support parse sql result as rust native datatypes like map, you may need to update get
  3. If it's a new native datatypes, you need to impl the FromSql and ToSql trait, example serde_json
  4. You may also want to add new type to enum Value when necessary, if you do that compiler will teach you multiple related places that also need to updated

@Swoorup

wangfenjin avatar Dec 05 '22 01:12 wangfenjin

The overall idea is in order to make it easier for users, we need to implement the convert logic of from/to sql. After doing this, you will be more familiar with the rust API and also the C API

wangfenjin avatar Dec 05 '22 01:12 wangfenjin

And please add some tests to the new features, feel free to support one type a time

wangfenjin avatar Dec 05 '22 01:12 wangfenjin

I think struct/enums/unions would likely need some kind of derivation support via proc macros. This is an interesting example for clickhouse client where the author modified serde to enable support for struct <-> row mapping. See klickhouse_derive https://github.com/Protryon/klickhouse

Swoorup avatar Dec 05 '22 03:12 Swoorup

Correct.

1. If you want to support bind struct or other type as params, you may need to update [bind_params](https://github.com/wangfenjin/duckdb-rs/blob/d2f611f922792ec6943a4f65301c81e247864fc0/src/statement.rs#L414)
2. If you want to support parse sql result as rust native datatypes like map, you may need to update [get](https://github.com/wangfenjin/duckdb-rs/blob/d2f611f922792ec6943a4f65301c81e247864fc0/src/row.rs#L544)
3. If it's a new native datatypes, you need to impl the FromSql and ToSql trait, example [serde_json](https://github.com/wangfenjin/duckdb-rs/blob/d2f611f922792ec6943a4f65301c81e247864fc0/src/types/serde_json.rs#L8-L27)
4. You may also want to add new type to enum [Value](https://github.com/wangfenjin/duckdb-rs/blob/d2f611f922792ec6943a4f65301c81e247864fc0/src/types/value.rs#L10) when necessary, if you do that compiler will teach you multiple related places that also need to updated

@wangfenjin If I try like in point 3, it says (edited out the error; I just had to add the relevant features).

Could you show an example, please, of how I can insert a String (valid JSON String) or a Vec<u8> containing a valid JSON String? If I try, I only see the new row as bytes (hex bytes shown in decimal for each character, and not a string). I created the table beforehand with create table tab (col1 STRING). Even tried the type TEXT and JSON. All just get inserted as bytes. Even with the serde_json example, I still see only bytes in decimal inserted into the table.

Will duckdb-rs add native support for Rust data types (struct/enum at least)?

v1gnesh avatar Sep 05 '23 08:09 v1gnesh