datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard

Re-enable tests for FIRST/LAST

Open andygrove opened this issue 8 months ago • 0 comments

What is the problem the feature request solves?

During the upgrade to DataFusion 47.0.0 () it was necessary to disable some tests that use FIRST and LAST because the behavior of these functions changed. These functions are non-determistic in Spark but previous versions of DataFusion had the same behavior as Spark for the single partition case used in the unit tests.

This issue is for re-enabling the tests somehow, or for implementing a Comet-specific implementation of FIRST/LAST to match Spark behavior (just for the single partition case used in tests).

These are the tests (all in CometAggregateSuite):

  ignore("single group-by column + aggregate column, multiple batches, no null") {
  ignore("multiple group-by columns + single aggregate column (first/last), with nulls") {
  ignore("multiple group-by columns + multiple aggregate column (first/last), with nulls") {
  ignore("all types first/last, with nulls") {
  ignore("first/last") {

Describe the potential solution

No response

Additional context

No response

andygrove avatar Apr 14 '25 13:04 andygrove