spark-rapids icon indicating copy to clipboard operation
spark-rapids copied to clipboard

Fix tests failures in orc_cast_test.py

Open razajafri opened this issue 1 year ago • 1 comments

FAILED ../../../../integration_tests/src/main/python/orc_cast_test.py::test_casting_among_integer_types

razajafri avatar Jun 08 '24 05:06 razajafri

This one is particularly interesting. The failure indicates a missing GPU metric:

2024-06-12 18:30:10,128 [Executor task launch worker for task 0.0 in stage 41.0 (TID 164)] ERROR org.apache.spark.executor.Executor - Exception in task 0.0 in stage 41.0 (TID 164)
java.util.NoSuchElementException: key not found: gpuDecodeTime
        at scala.collection.immutable.Map$EmptyMap$.apply(Map.scala:243) ~[scala-library-2.13.14.jar:?]
        at scala.collection.immutable.Map$EmptyMap$.apply(Map.scala:239) ~[scala-library-2.13.14.jar:?]
        at com.nvidia.spark.rapids.OrcTableReader.next(GpuOrcScan.scala:2884) ~[spark-shared/:?]
        at com.nvidia.spark.rapids.OrcTableReader.next(GpuOrcScan.scala:2859) ~[spark-shared/:?]
        at com.nvidia.spark.rapids.CachedGpuBatchIterator$.$anonfun$apply$1(GpuDataProducer.scala:159) ~[spark-shared/:?]

I'm double checking to see if this fails in non-Spark-4 builds.

mythrocks avatar Jun 12 '24 18:06 mythrocks

I've unassigned myself from this bug. This should be sorted out after the missing GPU metrics are addressed.

mythrocks avatar Jul 24 '24 05:07 mythrocks

This passes now that the gpuDecodeTime errors have been resolved -- #11429

rwlee avatar Oct 17 '24 00:10 rwlee