spark-rapids icon indicating copy to clipboard operation
spark-rapids copied to clipboard

Fix tests failures in subquery_test.py

Open razajafri opened this issue 1 year ago • 2 comments

FAILED ../../../../integration_tests/src/main/python/subquery_test.py::test_scalar_subquery_struct
FAILED ../../../../integration_tests/src/main/python/subquery_test.py::test_scalar_subquery_struct
FAILED ../../../../integration_tests/src/main/python/subquery_test.py::test_scalar_subquery_struct
FAILED ../../../../integration_tests/src/main/python/subquery_test.py::test_scalar_subquery_array
FAILED ../../../../integration_tests/src/main/python/subquery_test.py::test_scalar_subquery_struct
FAILED ../../../../integration_tests/src/main/python/subquery_test.py::test_scalar_subquery_array
FAILED ../../../../integration_tests/src/main/python/subquery_test.py::test_scalar_subquery_array
FAILED ../../../../integration_tests/src/main/python/subquery_test.py::test_scalar_subquery_struct
FAILED ../../../../integration_tests/src/main/python/subquery_test.py::test_scalar_subquery_struct
FAILED ../../../../integration_tests/src/main/python/subquery_test.py::test_scalar_subquery_struct
FAILED ../../../../integration_tests/src/main/python/subquery_test.py::test_scalar_subquery_struct
FAILED ../../../../integration_tests/src/main/python/subquery_test.py::test_scalar_subquery_array
FAILED ../../../../integration_tests/src/main/python/subquery_test.py::test_scalar_subquery_array
FAILED ../../../../integration_tests/src/main/python/subquery_test.py::test_scalar_subquery_struct
FAILED ../../../../integration_tests/src/main/python/subquery_test.py::test_scalar_subquery_array
FAILED ../../../../integration_tests/src/main/python/subquery_test.py::test_scalar_subquery_array
FAILED ../../../../integration_tests/src/main/python/subquery_test.py::test_scalar_subquery_struct
FAILED ../../../../integration_tests/src/main/python/subquery_test.py::test_scalar_subquery_struct
FAILED ../../../../integration_tests/src/main/python/subquery_test.py::test_scalar_subquery_array
FAILED ../../../../integration_tests/src/main/python/subquery_test.py::test_scalar_subquery_array
FAILED ../../../../integration_tests/src/main/python/subquery_test.py::test_scalar_subquery_map

razajafri avatar Jun 08 '24 05:06 razajafri

Some context here: These tests pass when ANSI mode is disabled.

mythrocks avatar Jun 12 '24 17:06 mythrocks

More info: subquery_test.py::test_scalar_subquery_struct fails because of a possible bug in libcudf:

E                       Suppressed: com.nvidia.spark.rapids.jni.GpuRetryOOM: injected RetryOOM
E                               at ai.rapids.cudf.ColumnView.hasNonEmptyNulls(Native Method)
E                               at ai.rapids.cudf.ColumnView.hasNonEmptyNulls(ColumnView.java:5218)
E                               at ai.rapids.cudf.AssertEmptyNulls.assertNullsAreEmpty(AssertEmptyNulls.java:33)
E                               at ai.rapids.cudf.ColumnView.<init>(ColumnView.java:58)
...

Somewhere along the way, it appears that the output columns have non-empty null rows. This will need to be fixed in libcudf. I'll raise a separate bug to track this. Edit: https://github.com/NVIDIA/spark-rapids/issues/11090

mythrocks avatar Jun 24 '24 21:06 mythrocks