datafusion icon indicating copy to clipboard operation
datafusion copied to clipboard

Add tests for support for `DataType::Union`

Open alamb opened this issue 1 year ago • 0 comments

Is your feature request related to a problem or challenge?

As we saw on https://github.com/apache/datafusion/pull/10139 with @joroKr21 there is not any particularly good way to test the coverage of DataType::Union today with our tests.

Note this is different than the SQL UNION / UNION ALL feature in union.slt

Describe the solution you'd like

I would like some way to test DataType::Union , ideally via sqllogictest -- in union_type.slt

Describe alternatives you've considered

I think the ideal alternative is to create a function that creates a UnionArray from different columns and use that function to create some simple queries / data in union_type.slt

Something like make_union(col1, col2, ...) (similarly to make_array) that results in a single column output of UnionArray

--- create a single UNION with two members (String and F64) with the string variant
select make_union('a'::varchar, null::double)

--- create a single UNION with two members (String and F64) with the f64
select make_union(null::varchar, 1.234)

--- Error: can't figure out which variant to create
select make_union('a', 123.4)

An alternate, option would be to register tables specially for similarly to https://github.com/apache/datafusion/blob/16e3831734358d2c628c7ff281cddd680dc4aa10/datafusion/sqllogictest/src/test_context.rs#L75-L78)

We could create a UnionArray using the examples from https://docs.rs/arrow/latest/arrow/array/struct.UnionArray.html

Additional context

See https://github.com/apache/datafusion/pull/10139#pullrequestreview-2013390566

alamb avatar Apr 21 '24 12:04 alamb