datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard

Feat: bitmap_count impl and some refactoring

Open kazantsev-maksim opened this issue 2 months ago • 4 comments

Which issue does this PR close?

Part of: https://github.com/apache/datafusion-comet/issues/2443

Rationale for this change

For the bitmap_count and read_side_padding functions, Spark uses a StaticInvoke wrapper. (https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitmapExpressions.scala#L85)

What changes are included in this PR?

  1. statics.scala - CometStaticInvoke wrapper has been added for the comet functions: bitmap_count, read_side_padding
  2. QueryPlanSerde.scala - removed unused matching cases
  3. CometBitmapExpressionSuite.scala - new UT

How are these changes tested?

A new UT has been added.

kazantsev-maksim avatar Oct 02 '25 18:10 kazantsev-maksim

Codecov Report

:x: Patch coverage is 56.25000% with 7 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 58.74%. Comparing base (f09f8af) to head (728b7c9). :warning: Report is 581 commits behind head on main.

Files with missing lines Patch % Lines
...rc/main/scala/org/apache/comet/serde/statics.scala 50.00% 6 Missing and 1 partial :warning:
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2518      +/-   ##
============================================
+ Coverage     56.12%   58.74%   +2.61%     
- Complexity      976     1452     +476     
============================================
  Files           119      148      +29     
  Lines         11743    13642    +1899     
  Branches       2251     2362     +111     
============================================
+ Hits           6591     8014    +1423     
- Misses         4012     4405     +393     
- Partials       1140     1223      +83     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov-commenter avatar Oct 02 '25 20:10 codecov-commenter

Starting CI

comphead avatar Oct 09 '25 02:10 comphead

Thanks @comphead, but the test failures occurred due to DataFusion's lack of support for dictionary-encoded types.

Cause: org.apache.comet.CometNativeException: Error from DataFusion: bitmap_count expects Binary/BinaryView/FixedSizeBinary/LargeBinary as argument, got Dictionary(Int32, Binary).

kazantsev-maksim avatar Oct 09 '25 15:10 kazantsev-maksim

Created a task: https://github.com/apache/datafusion/issues/18058

kazantsev-maksim avatar Oct 14 '25 16:10 kazantsev-maksim