spark-rapids icon indicating copy to clipboard operation
spark-rapids copied to clipboard

Fix tests failures in join_test.py

Open razajafri opened this issue 1 year ago • 2 comments

FAILED ../../../../integration_tests/src/main/python/join_test.py::test_broadcast_hash_join_fix_fallback_by_inputfile
FAILED ../../../../integration_tests/src/main/python/join_test.py::test_broadcast_nested_join_fix_fallback_by_inputfile
FAILED ../../../../integration_tests/src/main/python/join_test.py::test_broadcast_nested_loop_join_special_case_count
FAILED ../../../../integration_tests/src/main/python/join_test.py::test_broadcast_nested_loop_join_special_case_group_by_count
FAILED ../../../../integration_tests/src/main/python/join_test.py::test_broadcast_nested_loop_join_with_condition
FAILED ../../../../integration_tests/src/main/python/join_test.py::test_cartesian_join_special_case_count
FAILED ../../../../integration_tests/src/main/python/join_test.py::test_cartesian_join_special_case_group_by_count
FAILED ../../../../integration_tests/src/main/python/join_test.py::test_distinct_join
FAILED ../../../../integration_tests/src/main/python/join_test.py::test_half_cache_join
FAILED ../../../../integration_tests/src/main/python/join_test.py::test_left_broadcast_nested_loop_join_condition_missing_count
FAILED ../../../../integration_tests/src/main/python/join_test.py::test_right_broadcast_nested_loop_join_condition_missing_count

razajafri avatar Jun 08 '24 05:06 razajafri

Two of these tests (test_broadcast_hash_join_fix_fallback_by_inputfile, and test_broadcast_nested_join_fix_fallback_by_inputfile) seem to be failing with the same missing gpuDecodeTime metric:

E                   Caused by: java.util.NoSuchElementException: key not found: gpuDecodeTime
E                       at scala.collection.immutable.Map$EmptyMap$.apply(Map.scala:243)
E                       at scala.collection.immutable.Map$EmptyMap$.apply(Map.scala:239)
E                       at com.nvidia.spark.rapids.OrcTableReader.next(GpuOrcScan.scala:2884)
E                       at com.nvidia.spark.rapids.OrcTableReader.next(GpuOrcScan.scala:2859)

This comes from the ORC file reader.

The rest of the tests seem to run fine, when ANSI mode is disabled.

mythrocks avatar Jun 12 '24 21:06 mythrocks

I've unassigned myself from this bug. This should be resolved after the missing metrics are addressed.

mythrocks avatar Jul 24 '24 05:07 mythrocks