spark-rapids
spark-rapids copied to clipboard
Spark 3.4 changes to cast broke build
Is your feature request related to a problem? Please describe. Looks like changes in Saprk 3.4 cast broke the plugin 3.4 build:
https://github.com/apache/spark/commit/46175d1362062035fb93f87f25d61a9b711359ab https://github.com/apache/spark/commit/a4f96afdc147bfee3e0f195b7bcf3dfa882ad511
EDIT: Please aware of the followup change
- [SPARK-40054][SQL] Restore the error handling syntax of try_cast() https://github.com/apache/spark/commit/f8d51b9940
- [SPARK-40222][SQL] Numeric try_add/try_divide/try_subtract/try_multiply should throw error from their children https://github.com/apache/spark/commit/d3cf9310cc
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:30: object CastBase is not a member of package org.apache.spark.sql.catalyst.expressions
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:37: not found: type CastBase
ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCanonicalize.scala:58: not found: type CastBase
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCanonicalize.scala:58: value timeZoneId is not a member of org.apache.spark.sql.catalyst.expressions.Expression
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCanonicalize.scala:58: value needsTimeZone is not a member of org.apache.spark.sql.catalyst.expressions.Expression
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCanonicalize.scala:59: value withTimeZone is not a member of org.apache.spark.sql.catalyst.expressions.Expression
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:55: value child is not a member of type parameter INPUT
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:56: value dataType is not a member of type parameter INPUT
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:157: value timeZoneId is not a member of type parameter INPUT
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:30: Unused import
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:927: not found: type CastBase
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:929: not found: type CastBase
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:2102: recursive value x$15 needs type
EDIT: Please aware of the followup change [SPARK-40054][SQL] Restore the error handling syntax of try_cast() https://github.com/apache/spark/commit/f8d51b9940
Per @amahussein -- this bug now exists in Databricks 3.3
Trying to build for DB 11.3 / Spark 3.3.0. Apparently the castBase was ported to 3.3 on databricks.
[INFO] Compiling 313 Scala sources and 59 Java sources to /home/ubuntu/spark-rapids/sql-plugin/target/spark330db/classes...
[ERROR] /home/ubuntu/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:32: object CastBase is not a member of package org.apache.spark.sql.catalyst.expressions
[ERROR] import org.apache.spark.sql.catalyst.expressions.{Cast, CastBase, Expression, NullIntolerant, TimeZoneAwareExpression}
[ERROR] ^
[ERROR] /home/ubuntu/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:40: not found: type CastBase
[ERROR] final class CastExprMeta[INPUT <: CastBase](