spark-rapids [AUDIT] test the RAPIDS plugin with the spark-connect plugin

Spark connect modifies how jars are associated with the spark context / session https://github.com/apache/spark/commit/caa3df48d94ff2e7c824a87acf51ab4978e18098

Add tests ensuring, spark-rapids works with connect.

Oct 04 '23 00:10 gerashegalov

@mattahrens @sameerz How important do you guys think this feature is for 24.04?

Mar 11 '24 20:03 razajafri

I presume the main driver for connect will be support for an LTS of Databricks 14 https://docs.databricks.com/en/release-notes/runtime/14.0.html. 14.3 LTS was released on Feb 1 https://docs.databricks.com/en/release-notes/runtime/index.html

Mar 12 '24 17:03 gerashegalov

What about Apache Spark 3.5.1? Since 14.3 is branching off of it.

Mar 12 '24 17:03 razajafri

What about Apache Spark 3.5.1? Since 14.3 is branching off of it.

Upstream users have a choice, whereas it does not look like Shared Cluster users can opt out https://docs.databricks.com/en/release-notes/runtime/14.0.html#introducing-spark-connect-in-shared-cluster-architecture

Mar 12 '24 17:03 gerashegalov

We can revisit when we work on 4.0 or Databricks 14.x shims. For 3.5.1 it is not critical.

Mar 15 '24 01:03 sameerz