datafusion

datafusion copied to clipboard

Is your feature request related to a problem or challenge?

We have recently issues show that it is time to support Union

#10161 #10139 ScalarValue::iter_to_array #10180 comparison_coercion

I think the first step is able to create Union in slt file.

Describe the solution you'd like

Follow DuckDB Union or others well-known system.

https://duckdb.org/docs/sql/data_types/union.html

First step might be able to create table with union type

query error DataFusion error: Error during planning: Inconsistent data type across values list at row 1 column 0\. Was Int64 but found Utf8
create table t1 (u union(num interger, str varchar)) as values
  (1),
  ('two');

Describe alternatives you've considered

No response

Additional context

1. Built Union on top of Struct
2. Impl with ScalarUDFImpl

I would like to work on this.

As I mentioned here https://github.com/apache/datafusion/issues/7845#issuecomment-2073675288, the most useful thing for me would be less ugly errors when you do try to compare a union with something else.

Also it would be good to be able to force people to cast unions in my case, so it would be nice if we could switch off comparison_coercion somehow.

@vaibhawvipul if you're working on this, you might be interested in my logic in datafusion-contrib/datafusion-functions-json/src/common_union.rs

As I mentioned here #7845 (comment), the most useful thing for me would be less ugly errors when you do try to compare a union with something else.

Also it would be good to be able to force people to cast unions in my case, so it would be nice if we could switch off comparison_coercion somehow.

@vaibhawvipul if you're working on this, you might be interested in my logic in datafusion-contrib/datafusion-functions-json/src/common_union.rs

This was helpful @samuelcolvin. Thank you.

My approach is similar -

• adding support for arrow union datatype
• update planner
• need to support cast
• will be extending functions to have union functions like union_extract.

take

I opened a PR that adds support for Union datatype on sqlparser which I hope can help here. If so, some review would be very appreciated, since sqlparser-rs dev-bandwith is a bit limited. Thanks 🙏