datafusion icon indicating copy to clipboard operation
datafusion copied to clipboard

Validate ScalarUDF output rows and support nulls for `array_has` and `get_field` for `Map`

Open duongcongtoai opened this issue 1 year ago • 2 comments

Which issue does this PR close?

Closes #5735.

Add a small constraint for each UDF to have the same input and output size, with "arrow_typeof" as an exception

2 implementation failed the constraint (also fixed inside this PR)

  • array_has
  • get_field

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

duongcongtoai avatar Apr 20 '24 08:04 duongcongtoai

External error: query failed: DataFusion error: Internal error: UDF returned a different number of rows than expected. Expected: 7, Got: 6.
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
[SQL] select array_has(column1, make_array(5, 6)),
       array_has(column1, make_array(7, NULL)),
       array_has(column2, 5.5),
       array_has(column3, 'o')
from arrays;
at test_files/array.slt:5161

The omitted row is where column1 = null

duongcongtoai avatar Apr 21 '24 07:04 duongcongtoai

Requested change made, please help me check again @alamb

duongcongtoai avatar Apr 28 '24 05:04 duongcongtoai