address failure caused by method signature change in SPARK-48791

[parthchandra]

Describe the bug

Comet BatchReader uses a private method in Spark TaskMetrics.externalAccums. In the latest codebase of Spark a fix for SPARK-48791 has changed the signature of this method which causes Comet to fail with

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 2) (localhost executor driver): java.lang.NoSuchMethodError: 'scala.collection.mutable.Buffer org.apache.spark.executor.TaskMetrics.externalAccums()'

Steps to reproduce

Build Spark from latest code base and tru to run the HiveParquetSuite with Comet enabled.

Expected behavior

The test should pass

Additional context

No response

[kazuyukitanimura]

Reopening as there is a follow up PR that changes the method signature again https://github.com/apache/spark/commit/c3ba8fa69cbb88d5880a203f4b98c8bceb1c436b

[andygrove]

@parthchandra is this still an issue?

[parthchandra]

Seems like a perennial issue. This signature changes in every release it appears (it is private after all). https://github.com/apache/datafusion-comet/issues/1576

[kazuyukitanimura]

Fixed as a part of #1482 closing

[mkgada]

Would this fix https://github.com/apache/datafusion-comet/issues/1576 as well? @kazuyukitanimura

[kazuyukitanimura]

I looked into #1576 and looks like @mkgada you are using Spark 3.5.0?

https://datafusion.apache.org/comet/user-guide/installation.html We are currently only testing with 3.5.5 and 3.5.2-4 (partially)