spark-rapids icon indicating copy to clipboard operation
spark-rapids copied to clipboard

Fix tests failures in url_test.py

Open razajafri opened this issue 1 year ago • 1 comments

FAILED ../../../../integration_tests/src/main/python/url_test.py::test_parse_url_query_with_key
FAILED ../../../../integration_tests/src/main/python/url_test.py::test_parse_url_query_with_key_column
FAILED ../../../../integration_tests/src/main/python/url_test.py::test_parse_url_supported
FAILED ../../../../integration_tests/src/main/python/url_test.py::test_parse_url_with_key

razajafri avatar Jun 08 '24 05:06 razajafri

All these tests pass when ANSI mode is disabled.

mythrocks avatar Jun 13 '24 00:06 mythrocks

New failures with Spark-4.0 release jar:

FAILED ../../../../integration_tests/src/main/python/url_test.py::test_parse_url_with_key[invalid][DATAGEN_SEED=1749852444, TZ=UTC, INJECT_OOM] - pyspark.errors.exceptions.captured.IllegalArgumentException: Part of the plan is not columnar class org.apache.spark.sql.execution.ProjectExec
Project [invoke(ParseUrlEvaluator(null,invalid,key,false).evaluate(a#20, invalid, key)) AS parse_url(a, invalid, key)#21]
+- Scan ExistingRDD[a#20]
FAILED ../../../../integration_tests/src/main/python/url_test.py::test_parse_url_with_key[path][DATAGEN_SEED=1749852444, TZ=UTC, INJECT_OOM] - pyspark.errors.exceptions.captured.IllegalArgumentException: Part of the plan is not columnar class org.apache.spark.sql.execution.ProjectExec
Project [invoke(ParseUrlEvaluator(null,path,key,false).evaluate(a#18, path, key)) AS parse_url(a, path, key)#19]
+- Scan ExistingRDD[a#18]
FAILED ../../../../integration_tests/src/main/python/url_test.py::test_parse_url_with_key_fallback[FILE][DATAGEN_SEED=1749852444, TZ=UTC, INJECT_OOM, ALLOW_NON_GPU(ProjectExec,ParseUrl)] - py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.sql.rapids.ExecutionPlanCaptureCallback.assertDidFallBack.
: java.lang.AssertionError: assertion failed: Could not find ParseUrl in the GPU plans:
*(1) Project [invoke(ParseUrlEvaluator(null,FILE,key,true).evaluate(a#20, FILE, key)) AS parse_url(a, FILE, key)#21]
+- *(1) Scan ExistingRDD[a#20]

	at scala.Predef$.assert(Predef.scala:279)
	at org.apache.spark.sql.rapids.ShimmedExecutionPlanCaptureCallbackImpl.assertDidFallBack(ShimmedExecutionPlanCaptureCallbackImpl.scala:153)
	at org.apache.spark.sql.rapids.ShimmedExecutionPlanCaptureCallbackImpl.assertDidFallBack(ShimmedExecutionPlanCaptureCallbackImpl.scala:165)
	at org.apache.spark.sql.rapids.ExecutionPlanCaptureCallback$.assertDidFallBack(ExecutionPlanCaptureCallback.scala:91)
	at org.apache.spark.sql.rapids.ExecutionPlanCaptureCallback.assertDidFallBack(ExecutionPlanCaptureCallback.scala)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:184)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:108)
	at java.base/java.lang.Thread.run(Thread.java:833)
FAILED ../../../../integration_tests/src/main/python/url_test.py::test_parse_url_with_key[PATH][DATAGEN_SEED=1749852444, TZ=UTC, INJECT_OOM] - pyspark.errors.exceptions.captured.IllegalArgumentException: Part of the plan is not columnar class org.apache.spark.sql.execution.ProjectExec
Project [invoke(ParseUrlEvaluator(null,PATH,key,false).evaluate(a#18, PATH, key)) AS parse_url(a, PATH, key)#19]
+- Scan ExistingRDD[a#18]

====================================================== 30 failed, 1 xfailed, 702 warnings in 28.02s ======================================================

nartal1 avatar Jun 13 '25 22:06 nartal1