datafusion-comet feat: Support ANSI mode sum expr

Which issue does this PR close?

Closes #531 . (Partially closes 531) . The ANSI changes to support AVG will be tracked in a new PR

Rationale for this change

DataFusion's default SUM doesn't match Spark's overflow semantics. This implementation ensures we get the exact same behavior as Spark for integer overflow in all 3 eval modes (ANSI , Legacy and Try mode)

What changes are included in this PR?

This PR adds native Rust implementation for SUM aggregation on integer types (byte, short, int, long)

Native changes (Rust): (Inspired from SumDecimal and spark's SUM impl)

New SumInteger aggregate function that handles SUM for all integer types (in coherence with Spark)
Support all eval modes (ANSI , Legacy and Try) ( Implemented code in similar fashion of spark leveraging Option<i64> to represent NULL and numeric values for sum , and an additional parameter called has_all_nulls which is leveraged in Try mode to distinguish if NULL sum is caused by all NULL inputs or the fact that the sum overflowed. (Spark does this with shouldTrackIsEmpty and assigning NULL to running sum which is a long datatype) )
Implemented groups accumulator (which should optimize group by op)

Scala side changes :

Update CometSum to add ANSI support (ANSI and Try)
Change proto schema to pass along eval_mode instead of fail_on_error to support Legacy, ANSI and Try eval modes
Added tests for NULL (had to force cast nulls as java.lang.Long to avoid Scala's auto-boxing feature which auto casts objects to primitive types there by casting nulls to 0s) handling in both simple and GROUP BY agg .
Added tests for both ANSI and non-ANSI mode tests for the input types Int, Short, Byte
Added try_sum overflow tests with GROUP BY (mixed overflow/non-overflow groups)

How are these changes tested?

Unit tests on Scala side to test various aggregation use cases in all 3 eval modes .
Added new tests for groupBy use cases to test out groups accumulator

Oct 17 '25 19:10 coderfender

Draft PR to support sum function - WIP

Oct 17 '25 19:10 coderfender

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 59.41%. Comparing base (f09f8af) to head (126c053). :warning: Report is 768 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2600      +/-   ##
============================================
+ Coverage     56.12%   59.41%   +3.28%     
- Complexity      976     1377     +401     
============================================
  Files           119      167      +48     
  Lines         11743    15330    +3587     
  Branches       2251     2546     +295     
============================================
+ Hits           6591     9108    +2517     
- Misses         4012     4938     +926     
- Partials       1140     1284     +144

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

:snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Oct 30 '25 00:10 codecov-commenter

@andygrove , @comphead I believe this should be ready for review (pending CI)

Nov 07 '25 03:11 coderfender

- Windows support *** FAILED *** (3 seconds, 973 milliseconds)
  Expected only Comet native operators, but found Project.
  plan: Project
  +- Window
     +- CometExchange
        +- CometScan [native_iceberg_compat] parquet

Thats actually interesting why would window fall back, we planning to fallback on windows in https://github.com/apache/datafusion-comet/pull/2726 but the test would still preserve windows, so I think we need to check a fallback reason

Nov 07 '25 22:11 comphead

- Windows support *** FAILED *** (3 seconds, 973 milliseconds)
  Expected only Comet native operators, but found Project.
  plan: Project
  +- Window
     +- CometExchange
        +- CometScan [native_iceberg_compat] parquet
Thats actually interesting why would window fall back, we planning to fallback on windows in #2726 but the test would still preserve windows, so I think we need to check a fallback reason

This is no longer a failing test after I rebased with main branch

Nov 13 '25 00:11 coderfender

@andygrove @comphead . All the tests passed and the PR is rest for review

Nov 13 '25 01:11 coderfender

Thank you @andygrove . Let me rebase and fix the merge conflicts

Nov 20 '25 15:11 coderfender

Here are microbenchmark results:

Legacy mode

OpenJDK 64-Bit Server VM 17.0.16+8-Ubuntu-0ubuntu122.04.1 on Linux 6.8.0-87-generic
AMD Ryzen 9 7950X3D 16-Core Processor
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
agg_sum_integers_no_grouping                       1720           1812         129        167.4           6.0       1.0X
agg_sum_integers_no_grouping: Comet                1171           1179          11        246.0           4.1       1.5X

Running benchmark: TPCDS Micro Benchmarks
  Running case: agg_sum_integers_with_grouping
  Stopped after 2 iterations, 5105 ms
  Running case: agg_sum_integers_with_grouping: Comet
  Stopped after 2 iterations, 2803 ms

OpenJDK 64-Bit Server VM 17.0.16+8-Ubuntu-0ubuntu122.04.1 on Linux 6.8.0-87-generic
AMD Ryzen 9 7950X3D 16-Core Processor
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
agg_sum_integers_with_grouping                     2529           2553          34        113.9           8.8       1.0X
agg_sum_integers_with_grouping: Comet              1396           1402           8        206.3           4.8       1.8X

ANSI mode

OpenJDK 64-Bit Server VM 17.0.16+8-Ubuntu-0ubuntu122.04.1 on Linux 6.8.0-87-generic
AMD Ryzen 9 7950X3D 16-Core Processor
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
agg_sum_integers_no_grouping                       1751           1802          72        164.5           6.1       1.0X
agg_sum_integers_no_grouping: Comet                1358           1360           3        212.1           4.7       1.3X

Running benchmark: TPCDS Micro Benchmarks
  Running case: agg_sum_integers_with_grouping
  Stopped after 2 iterations, 5563 ms
  Running case: agg_sum_integers_with_grouping: Comet
  Stopped after 2 iterations, 3668 ms

OpenJDK 64-Bit Server VM 17.0.16+8-Ubuntu-0ubuntu122.04.1 on Linux 6.8.0-87-generic
AMD Ryzen 9 7950X3D 16-Core Processor
TPCDS Micro Benchmarks:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
agg_sum_integers_with_grouping                     2772           2782          14        103.9           9.6       1.0X
agg_sum_integers_with_grouping: Comet              1834           1834           1        157.1           6.4       1.5X

Nov 20 '25 16:11 andygrove

@andygrove , Please take a look whenever you get a chance . I made changes per review and fixed merge conflicts

Nov 21 '25 04:11 coderfender

@andygrove I rebased with main and fixed merge conflicts . Please review whenever you get a chance

Dec 05 '25 22:12 coderfender

Resolved scalafix error

Dec 05 '25 22:12 coderfender

@andygrove , @martin-g could you please take a (hopefully final) look at the PR and let me know if you think we need further changes ?

Dec 11 '25 20:12 coderfender

@andygrove , @martin-g could you please take a (hopefully final) look at the PR and let me know if you think we need further changes ?

The changes in the PR are pretty substantial (~800 new lines of code) so there is a lot to review. I don't see anything that concerns me at this point, but I would like to run some benchmarks to check that there are no regressions in the non ANSI case. Unfortunately, we don't have any existing microbenchmarks that work with ANSI enabled, so we need to create something.

Dec 11 '25 21:12 andygrove

Thank you @andygrove let me go ahead and work on generating benchmarks for ANSI supported aggregate expressions

Dec 11 '25 22:12 coderfender

Created #2883 to support benchmarking with ANSI mode

Dec 12 '25 00:12 coderfender

Ran benchmarking with ANSI mode enabled for SUM aggregated function only and here are the results

Running benchmark: Grouped HashAgg Exec: single group key (cardinality 100), single aggregate SUM, ansi mode enabled : true
  Running case: SQL Parquet - Spark (SUM) ansi mode enabled : true
  Stopped after 32 iterations, 2013 ms
  Running case: SQL Parquet - Comet (SUM) ansi mode enabled : true
  Stopped after 38 iterations, 2032 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: single group key (cardinality 100), single aggregate SUM, ansi mode enabled : true:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) ansi mode enabled : true                                                                   48             63          17        219.2           4.6       1.0X
SQL Parquet - Comet (SUM) ansi mode enabled : true                                                                   36             53          17        289.7           3.5       1.3X

Running benchmark: Grouped HashAgg Exec: single group key (cardinality 1024), single aggregate SUM, ansi mode enabled : true
  Running case: SQL Parquet - Spark (SUM) ansi mode enabled : true
  Stopped after 30 iterations, 2071 ms
  Running case: SQL Parquet - Comet (SUM) ansi mode enabled : true
  Stopped after 36 iterations, 2011 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: single group key (cardinality 1024), single aggregate SUM, ansi mode enabled : true:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) ansi mode enabled : true                                                                    47             69          13        221.2           4.5       1.0X
SQL Parquet - Comet (SUM) ansi mode enabled : true                                                                    35             56          33        303.5           3.3       1.4X

Running benchmark: Grouped HashAgg Exec: single group key (cardinality 1048576), single aggregate SUM, ansi mode enabled : true
  Running case: SQL Parquet - Spark (SUM) ansi mode enabled : true
  Stopped after 33 iterations, 2049 ms
  Running case: SQL Parquet - Comet (SUM) ansi mode enabled : true
  Stopped after 33 iterations, 2035 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: single group key (cardinality 1048576), single aggregate SUM, ansi mode enabled : true:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) ansi mode enabled : true                                                                       50             62          17        211.4           4.7       1.0X
SQL Parquet - Comet (SUM) ansi mode enabled : true                                                                       37             62          33        281.9           3.5       1.3X

Running benchmark: Grouped HashAgg Exec: multiple group keys (cardinality 100), single aggregate SUM
  Running case: SQL Parquet - Spark (SUM) isANSIMode: true
  Stopped after 32 iterations, 2026 ms
  Running case: SQL Parquet - Comet (SUM) isANSIMode: true
  Stopped after 34 iterations, 2010 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: multiple group keys (cardinality 100), single aggregate SUM:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) isANSIMode: true                                                    49             63          23        213.9           4.7       1.0X
SQL Parquet - Comet (SUM) isANSIMode: true                                                    36             59          23        287.7           3.5       1.3X

Running benchmark: Grouped HashAgg Exec: multiple group keys (cardinality 1024), single aggregate SUM
  Running case: SQL Parquet - Spark (SUM) isANSIMode: true
  Stopped after 29 iterations, 2131 ms
  Running case: SQL Parquet - Comet (SUM) isANSIMode: true
  Stopped after 43 iterations, 2014 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: multiple group keys (cardinality 1024), single aggregate SUM:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) isANSIMode: true                                                     48             73          32        217.1           4.6       1.0X
SQL Parquet - Comet (SUM) isANSIMode: true                                                     36             47          18        290.1           3.4       1.3X

Running benchmark: Grouped HashAgg Exec: multiple group keys (cardinality 1048576), single aggregate SUM
  Running case: SQL Parquet - Spark (SUM) isANSIMode: true
  Stopped after 2 iterations, 2952 ms
  Running case: SQL Parquet - Comet (SUM) isANSIMode: true
  Stopped after 5 iterations, 2276 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: multiple group keys (cardinality 1048576), single aggregate SUM:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) isANSIMode: true                                                      1463           1476          18          7.2         139.5       1.0X
SQL Parquet - Comet (SUM) isANSIMode: true                                                       396            455          92         26.5          37.8       3.7X

Running benchmark: Grouped HashAgg Exec: single group key (cardinality 100), multiple aggregates SUM isANSIMode: true
  Running case: SQL Parquet - Spark (SUM) isANSIMode: true
  Stopped after 13 iterations, 2075 ms
  Running case: SQL Parquet - Comet (SUM) isANSIMode: true
  Stopped after 35 iterations, 2051 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: single group key (cardinality 100), multiple aggregates SUM isANSIMode: true:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) isANSIMode: true                                                                     56            160          78        188.0           5.3       1.0X
SQL Parquet - Comet (SUM) isANSIMode: true                                                                     34             59          45        309.5           3.2       1.6X

Running benchmark: Grouped HashAgg Exec: single group key (cardinality 1024), multiple aggregates SUM isANSIMode: true
  Running case: SQL Parquet - Spark (SUM) isANSIMode: true
  Stopped after 16 iterations, 2103 ms
  Running case: SQL Parquet - Comet (SUM) isANSIMode: true
  Stopped after 37 iterations, 2016 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: single group key (cardinality 1024), multiple aggregates SUM isANSIMode: true:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) isANSIMode: true                                                                      55            131          97        189.6           5.3       1.0X
SQL Parquet - Comet (SUM) isANSIMode: true                                                                      38             54          30        273.0           3.7       1.4X

Running benchmark: Grouped HashAgg Exec: single group key (cardinality 1048576), multiple aggregates SUM isANSIMode: true
  Running case: SQL Parquet - Spark (SUM) isANSIMode: true
  Stopped after 20 iterations, 2118 ms
  Running case: SQL Parquet - Comet (SUM) isANSIMode: true
  Stopped after 22 iterations, 2025 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: single group key (cardinality 1048576), multiple aggregates SUM isANSIMode: true:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) isANSIMode: true                                                                         55            106          71        189.8           5.3       1.0X
SQL Parquet - Comet (SUM) isANSIMode: true                                                                         35             92          73        301.9           3.3       1.6X

Running benchmark: Grouped HashAgg Exec: single group key (cardinality 100), single aggregate SUM on decimal
  Running case: SQL Parquet - Spark (SUM) ansi mode enabled : true
  Stopped after 2 iterations, 2658 ms
  Running case: SQL Parquet - Comet (SUM) ansi mode enabled : true
  Stopped after 9 iterations, 2233 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: single group key (cardinality 100), single aggregate SUM on decimal:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) ansi mode enabled : true                                                  1257           1329         102          8.3         119.9       1.0X
SQL Parquet - Comet (SUM) ansi mode enabled : true                                                   237            248          15         44.2          22.6       5.3X

Running benchmark: Grouped HashAgg Exec: single group key (cardinality 1024), single aggregate SUM on decimal
  Running case: SQL Parquet - Spark (SUM) ansi mode enabled : true
  Stopped after 2 iterations, 2615 ms
  Running case: SQL Parquet - Comet (SUM) ansi mode enabled : true
  Stopped after 8 iterations, 2222 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: single group key (cardinality 1024), single aggregate SUM on decimal:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) ansi mode enabled : true                                                   1307           1308           0          8.0         124.7       1.0X
SQL Parquet - Comet (SUM) ansi mode enabled : true                                                    257            278          29         40.7          24.6       5.1X

Running benchmark: Grouped HashAgg Exec: single group key (cardinality 1048576), single aggregate SUM on decimal
  Running case: SQL Parquet - Spark (SUM) ansi mode enabled : true
  Stopped after 2 iterations, 8222 ms
  Running case: SQL Parquet - Comet (SUM) ansi mode enabled : true
  Stopped after 2 iterations, 2853 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: single group key (cardinality 1048576), single aggregate SUM on decimal:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) ansi mode enabled : true                                                      4082           4111          42          2.6         389.3       1.0X
SQL Parquet - Comet (SUM) ansi mode enabled : true                                                      1412           1427          20          7.4         134.7       2.9X




Running benchmark: Grouped HashAgg Exec: single group key (cardinality 100), single aggregate SUM, ansi mode enabled : false

  Running case: SQL Parquet - Spark (SUM) ansi mode enabled : false
  Stopped after 9 iterations, 2064 ms
  Running case: SQL Parquet - Comet (SUM) ansi mode enabled : false
  Stopped after 12 iterations, 2025 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: single group key (cardinality 100), single aggregate SUM, ansi mode enabled : false:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) ansi mode enabled : false                                                                  221            229          12         47.5          21.0       1.0X
SQL Parquet - Comet (SUM) ansi mode enabled : false                                                                  153            169          25         68.5          14.6       1.4X

Running benchmark: Grouped HashAgg Exec: single group key (cardinality 1024), single aggregate SUM, ansi mode enabled : false
  Running case: SQL Parquet - Spark (SUM) ansi mode enabled : false
  Stopped after 9 iterations, 2207 ms
  Running case: SQL Parquet - Comet (SUM) ansi mode enabled : false
  Stopped after 12 iterations, 2053 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: single group key (cardinality 1024), single aggregate SUM, ansi mode enabled : false:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) ansi mode enabled : false                                                                   226            245          30         46.4          21.6       1.0X
SQL Parquet - Comet (SUM) ansi mode enabled : false                                                                   165            171           4         63.4          15.8       1.4X

Running benchmark: Grouped HashAgg Exec: single group key (cardinality 1048576), single aggregate SUM, ansi mode enabled : false
  Running case: SQL Parquet - Spark (SUM) ansi mode enabled : false
  Stopped after 2 iterations, 4153 ms
  Running case: SQL Parquet - Comet (SUM) ansi mode enabled : false
  Stopped after 3 iterations, 2645 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: single group key (cardinality 1048576), single aggregate SUM, ansi mode enabled : false:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) ansi mode enabled : false                                                                     2069           2077          11          5.1         197.3       1.0X
SQL Parquet - Comet (SUM) ansi mode enabled : false                                                                      841            882          49         12.5          80.2       2.5X

Running benchmark: Grouped HashAgg Exec: multiple group keys (cardinality 100), single aggregate SUM
  Running case: SQL Parquet - Spark (SUM) isANSIMode: false
  Stopped after 4 iterations, 2340 ms
  Running case: SQL Parquet - Comet (SUM) isANSIMode: false
  Stopped after 8 iterations, 2320 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: multiple group keys (cardinality 100), single aggregate SUM:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) isANSIMode: false                                                  574            585          12         18.3          54.8       1.0X
SQL Parquet - Comet (SUM) isANSIMode: false                                                  269            290          33         39.0          25.6       2.1X

Running benchmark: Grouped HashAgg Exec: multiple group keys (cardinality 1024), single aggregate SUM
  Running case: SQL Parquet - Spark (SUM) isANSIMode: false
  Stopped after 2 iterations, 5877 ms
  Running case: SQL Parquet - Comet (SUM) isANSIMode: false
  Stopped after 2 iterations, 2924 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: multiple group keys (cardinality 1024), single aggregate SUM:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) isANSIMode: false                                                  2896           2939          60          3.6         276.2       1.0X
SQL Parquet - Comet (SUM) isANSIMode: false                                                  1400           1462          88          7.5         133.5       2.1X

Running benchmark: Grouped HashAgg Exec: multiple group keys (cardinality 1048576), single aggregate SUM
  Running case: SQL Parquet - Spark (SUM) isANSIMode: false
   Stopped after 2 iterations, 13239 ms
  Running case: SQL Parquet - Comet (SUM) isANSIMode: false
  Stopped after 2 iterations, 10603 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: multiple group keys (cardinality 1048576), single aggregate SUM:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) isANSIMode: false                                                     6580           6620          57          1.6         627.5       1.0X
SQL Parquet - Comet (SUM) isANSIMode: false                                                     5265           5302          52          2.0         502.1       1.2X

Running benchmark: Grouped HashAgg Exec: single group key (cardinality 100), multiple aggregates SUM isANSIMode: false
  Running case: SQL Parquet - Spark (SUM) isANSIMode: false
  Stopped after 6 iterations, 2011 ms
  Running case: SQL Parquet - Comet (SUM) isANSIMode: false
  Stopped after 9 iterations, 2165 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: single group key (cardinality 100), multiple aggregates SUM isANSIMode: false:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) isANSIMode: false                                                                    290            335          74         36.2          27.7       1.0X
SQL Parquet - Comet (SUM) isANSIMode: false                                                                    207            241          59         50.6          19.8       1.4X

Running benchmark: Grouped HashAgg Exec: single group key (cardinality 1024), multiple aggregates SUM isANSIMode: false
  Running case: SQL Parquet - Spark (SUM) isANSIMode: false
  Stopped after 6 iterations, 2031 ms
  Running case: SQL Parquet - Comet (SUM) isANSIMode: false
  Stopped after 9 iterations, 2281 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: single group key (cardinality 1024), multiple aggregates SUM isANSIMode: false:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) isANSIMode: false                                                                     303            339          54         34.6          28.9       1.0X
SQL Parquet - Comet (SUM) isANSIMode: false                                                                     227            253          32         46.2          21.7       1.3X

Running benchmark: Grouped HashAgg Exec: single group key (cardinality 1048576), multiple aggregates SUM isANSIMode: false
  Running case: SQL Parquet - Spark (SUM) isANSIMode: false
  Stopped after 2 iterations, 6469 ms
  Running case: SQL Parquet - Comet (SUM) isANSIMode: false
  Stopped after 2 iterations, 3608 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: single group key (cardinality 1048576), multiple aggregates SUM isANSIMode: false:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) isANSIMode: false                                                                       3222           3235          18          3.3         307.3       1.0X
SQL Parquet - Comet (SUM) isANSIMode: false                                                                       1562           1804         343          6.7         148.9       2.1X

Running benchmark: Grouped HashAgg Exec: single group key (cardinality 100), single aggregate SUM on decimal
  Running case: SQL Parquet - Spark (SUM) ansi mode enabled : false
  Stopped after 2 iterations, 2433 ms
  Running case: SQL Parquet - Comet (SUM) ansi mode enabled : false
  Stopped after 8 iterations, 2034 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: single group key (cardinality 100), single aggregate SUM on decimal:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) ansi mode enabled : false                                                 1210           1217          10          8.7         115.4       1.0X
SQL Parquet - Comet (SUM) ansi mode enabled : false                                                  249            254           4         42.0          23.8       4.9X

Running benchmark: Grouped HashAgg Exec: single group key (cardinality 1024), single aggregate SUM on decimal
  Running case: SQL Parquet - Spark (SUM) ansi mode enabled : false
  Stopped after 2 iterations, 2483 ms
  Running case: SQL Parquet - Comet (SUM) ansi mode enabled : false
  Stopped after 7 iterations, 2099 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: single group key (cardinality 1024), single aggregate SUM on decimal:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) ansi mode enabled : false                                                  1235           1242          10          8.5         117.7       1.0X
SQL Parquet - Comet (SUM) ansi mode enabled : false                                                   255            300          54         41.2          24.3       4.8X

Running benchmark: Grouped HashAgg Exec: single group key (cardinality 1048576), single aggregate SUM on decimal
  Running case: SQL Parquet - Spark (SUM) ansi mode enabled : false
  Stopped after 2 iterations, 8194 ms
  Running case: SQL Parquet - Comet (SUM) ansi mode enabled : false
  Stopped after 2 iterations, 2820 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Grouped HashAgg Exec: single group key (cardinality 1048576), single aggregate SUM on decimal:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (SUM) ansi mode enabled : false                                                     4069           4097          40          2.6         388.0       1.0X
SQL Parquet - Comet (SUM) ansi mode enabled : false                                                     1381           1410          42          7.6         131.7       2.9X

Dec 12 '25 05:12 coderfender

Rebased with main

Dec 15 '25 22:12 coderfender