Support Spark 3.5: Catalyst StaticInvoke constructor signature changed (8-arg vs 9-arg with Option)

[anudeepkonaboina-db]

Context On some Spark 3.5 builds, StaticInvoke has a 9-arg signature; RF binaries expect 8-arg, throwing NoSuchMethodError:

NoSuchMethodError:
org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke.<init>(
Ljava/lang/Class;
Lorg/apache/spark/sql/types/DataType;
Ljava/lang/String;
Lscala/collection/Seq;
Lscala/collection/Seq;
Z Z Z
)V

Environment (secondary)

• Spark: 3.5.x (runtime exposes 9-arg StaticInvoke)
• Scala: 2.12.x
• Note: Observed on Databricks Runtime 15.4 (a vendor build of Spark 3.5)

Evidence

• Runtime constructor(s) seen (Also in saprk 4.0):

public org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke(
    java.lang.Class,
    org.apache.spark.sql.types.DataType,
    java.lang.String,
    scala.collection.Seq,
    scala.collection.Seq,
    boolean, boolean, boolean,
    scala.Option
  )

• Spark 3.5/3.4 constructor(s) :

public org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke(
    java.lang.Class,
    org.apache.spark.sql.types.DataType,
    java.lang.String,
    scala.collection.Seq,
    scala.collection.Seq,
    boolean, boolean, boolean
  )

• Error was trying to call the 8-arg ctor (no Option) and failed with NoSuchMethodError: StaticInvoke.(Class, DataType, String, Seq, Seq, boolean, boolean, boolean)

Proposal

• Introduce a tiny compatibility in RF encoder code paths that constructs StaticInvoke via reflection:
  • First try the 9-arg signature (… boolean, boolean, boolean, Option)
  • Fallback to the 8-arg signature (… boolean, boolean, boolean)
• This keeps sources binary-compatible across Spark 3.4 and 3.5 variants. This is currently available in Spark 4.0