Kazantsev Maksim
Kazantsev Maksim
I can fix all the already implemented array related functions within the scope of this issue. @kazuyukitanimura
@mbutrovich I couldn't find a built-in implementation of bit_count in the DataFusion project, but i rewrote it using scalarFunc without adding a new proto expr.
Thank you for the feedback! @kazuyukitanimura @parthchandra
Thanks @comphead, but the test failures occurred due to DataFusion's lack of support for dictionary-encoded types. Cause: org.apache.comet.CometNativeException: Error from DataFusion: bitmap_count expects Binary/BinaryView/FixedSizeBinary/LargeBinary as argument, got Dictionary(Int32, Binary).
Created a task: https://github.com/apache/datafusion/issues/18058
The DataFusion implementation produces results that are not compatible with Spark Project [bit_count(col2#2190) AS bit_count(col2)#2218] +- Relation spark_catalog.default.bitwise_count_test[col1#2189L,col2#2190,col3#2191,col4#2192] parquet == Physical Plan == *(1) CometColumnarToRow +- CometProject [bit_count(col2)#2218], [bit_count(col2#2190) AS...
Thanks @comphead, conflicts resolved.
> *** 3 TESTS FAILED *** I think the DataFusion implementation needs modification to be fully compatible with Spark.
After closing the task (https://github.com/apache/datafusion/pull/18322 ), the tests must pass. FYI @comphead
@comphead I'd like to merge https://github.com/apache/datafusion-comet/pull/1602 first, and then update this PR again, but the reverse order is also fine.