blaze icon indicating copy to clipboard operation
blaze copied to clipboard

ClassCastException when have specify UDF

Open kettlelinna opened this issue 1 year ago • 2 comments

Describe the bug

I met this exception when my program have specify UDF

java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.sql.catalyst.expressions.ScalaUDF.f of type scala.Function1 in instance of org.apache.spark.sql.catalyst.expressions.ScalaUDF at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287) at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2293) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) at org.apache.spark.sql.blaze.NativeConverters$.$anonfun$deserializeExpression$4(NativeConverters.scala:1095) at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2764) at org.apache.spark.sql.blaze.NativeConverters$.$anonfun$deserializeExpression$2(NativeConverters.scala:1094) at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2764) at org.apache.spark.sql.blaze.NativeConverters$.deserializeExpression(NativeConverters.scala:1093) at org.apache.spark.sql.blaze.SparkUDFWrapperContext.(SparkUDFWrapperContext.scala:42)

To Reproduce Steps to reproduce the behavior:

  1. refer delta 2.3.0
  2. use delta format to load data, delta is base on parquet

Expected behavior

it should works whatever UDF

Desktop (please complete the following information):

  • OS: Centos7
  • Version compile base on latest code

kettlelinna avatar Jan 21 '24 11:01 kettlelinna