spark-rapids
spark-rapids copied to clipboard
Fix tests failures in url_test.py
FAILED ../../../../integration_tests/src/main/python/url_test.py::test_parse_url_query_with_key
FAILED ../../../../integration_tests/src/main/python/url_test.py::test_parse_url_query_with_key_column
FAILED ../../../../integration_tests/src/main/python/url_test.py::test_parse_url_supported
FAILED ../../../../integration_tests/src/main/python/url_test.py::test_parse_url_with_key
All these tests pass when ANSI mode is disabled.
New failures with Spark-4.0 release jar:
FAILED ../../../../integration_tests/src/main/python/url_test.py::test_parse_url_with_key[invalid][DATAGEN_SEED=1749852444, TZ=UTC, INJECT_OOM] - pyspark.errors.exceptions.captured.IllegalArgumentException: Part of the plan is not columnar class org.apache.spark.sql.execution.ProjectExec
Project [invoke(ParseUrlEvaluator(null,invalid,key,false).evaluate(a#20, invalid, key)) AS parse_url(a, invalid, key)#21]
+- Scan ExistingRDD[a#20]
FAILED ../../../../integration_tests/src/main/python/url_test.py::test_parse_url_with_key[path][DATAGEN_SEED=1749852444, TZ=UTC, INJECT_OOM] - pyspark.errors.exceptions.captured.IllegalArgumentException: Part of the plan is not columnar class org.apache.spark.sql.execution.ProjectExec
Project [invoke(ParseUrlEvaluator(null,path,key,false).evaluate(a#18, path, key)) AS parse_url(a, path, key)#19]
+- Scan ExistingRDD[a#18]
FAILED ../../../../integration_tests/src/main/python/url_test.py::test_parse_url_with_key_fallback[FILE][DATAGEN_SEED=1749852444, TZ=UTC, INJECT_OOM, ALLOW_NON_GPU(ProjectExec,ParseUrl)] - py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.sql.rapids.ExecutionPlanCaptureCallback.assertDidFallBack.
: java.lang.AssertionError: assertion failed: Could not find ParseUrl in the GPU plans:
*(1) Project [invoke(ParseUrlEvaluator(null,FILE,key,true).evaluate(a#20, FILE, key)) AS parse_url(a, FILE, key)#21]
+- *(1) Scan ExistingRDD[a#20]
at scala.Predef$.assert(Predef.scala:279)
at org.apache.spark.sql.rapids.ShimmedExecutionPlanCaptureCallbackImpl.assertDidFallBack(ShimmedExecutionPlanCaptureCallbackImpl.scala:153)
at org.apache.spark.sql.rapids.ShimmedExecutionPlanCaptureCallbackImpl.assertDidFallBack(ShimmedExecutionPlanCaptureCallbackImpl.scala:165)
at org.apache.spark.sql.rapids.ExecutionPlanCaptureCallback$.assertDidFallBack(ExecutionPlanCaptureCallback.scala:91)
at org.apache.spark.sql.rapids.ExecutionPlanCaptureCallback.assertDidFallBack(ExecutionPlanCaptureCallback.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:184)
at py4j.ClientServerConnection.run(ClientServerConnection.java:108)
at java.base/java.lang.Thread.run(Thread.java:833)
FAILED ../../../../integration_tests/src/main/python/url_test.py::test_parse_url_with_key[PATH][DATAGEN_SEED=1749852444, TZ=UTC, INJECT_OOM] - pyspark.errors.exceptions.captured.IllegalArgumentException: Part of the plan is not columnar class org.apache.spark.sql.execution.ProjectExec
Project [invoke(ParseUrlEvaluator(null,PATH,key,false).evaluate(a#18, PATH, key)) AS parse_url(a, PATH, key)#19]
+- Scan ExistingRDD[a#18]
====================================================== 30 failed, 1 xfailed, 702 warnings in 28.02s ======================================================