blaze
blaze copied to clipboard
ClassCastException when have specify UDF
Describe the bug
I met this exception when my program have specify UDF
java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.sql.catalyst.expressions.ScalaUDF.f of type scala.Function1 in instance of org.apache.spark.sql.catalyst.expressions.ScalaUDF
at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287)
at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2293)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
at org.apache.spark.sql.blaze.NativeConverters$.$anonfun$deserializeExpression$4(NativeConverters.scala:1095)
at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2764)
at org.apache.spark.sql.blaze.NativeConverters$.$anonfun$deserializeExpression$2(NativeConverters.scala:1094)
at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2764)
at org.apache.spark.sql.blaze.NativeConverters$.deserializeExpression(NativeConverters.scala:1093)
at org.apache.spark.sql.blaze.SparkUDFWrapperContext.
To Reproduce Steps to reproduce the behavior:
- refer delta 2.3.0
- use delta format to load data, delta is base on parquet
Expected behavior
it should works whatever UDF
Desktop (please complete the following information):
- OS: Centos7
- Version compile base on latest code