spark-rapids
spark-rapids copied to clipboard
Fix tests failures in orc_cast_test.py
FAILED ../../../../integration_tests/src/main/python/orc_cast_test.py::test_casting_among_integer_types
This one is particularly interesting. The failure indicates a missing GPU metric:
2024-06-12 18:30:10,128 [Executor task launch worker for task 0.0 in stage 41.0 (TID 164)] ERROR org.apache.spark.executor.Executor - Exception in task 0.0 in stage 41.0 (TID 164)
java.util.NoSuchElementException: key not found: gpuDecodeTime
at scala.collection.immutable.Map$EmptyMap$.apply(Map.scala:243) ~[scala-library-2.13.14.jar:?]
at scala.collection.immutable.Map$EmptyMap$.apply(Map.scala:239) ~[scala-library-2.13.14.jar:?]
at com.nvidia.spark.rapids.OrcTableReader.next(GpuOrcScan.scala:2884) ~[spark-shared/:?]
at com.nvidia.spark.rapids.OrcTableReader.next(GpuOrcScan.scala:2859) ~[spark-shared/:?]
at com.nvidia.spark.rapids.CachedGpuBatchIterator$.$anonfun$apply$1(GpuDataProducer.scala:159) ~[spark-shared/:?]
I'm double checking to see if this fails in non-Spark-4 builds.
I've unassigned myself from this bug. This should be sorted out after the missing GPU metrics are addressed.
This passes now that the gpuDecodeTime errors have been resolved -- #11429