[Spark] Update sbt testing task of Delta Connect Client to automatically use the Spark latest jars

[longvu-db]

trafficstars

Which Delta project/connector is this regarding?

• [X] Spark
• [ ] Standalone
• [ ] Flink
• [ ] Kernel
• [ ] Other (fill in here)

Description

Currently, in order to test Delta Connect Client with Spark Connect, we always fetch the Spark 4.0 First Preview jars, which were created in May 2024.

Since then, lots of things have changed in Spark, so the jars are outdated and the testing no longer works due to binary incompatibility between the latest version of Delta and Spark 4.0 First Preview version.

Therefore, we are updating the sbt testing task of the Delta Connect Client project to automatically fetch and use the latest jars from Spark nightly releases.

How was this patch tested?

Ran build/sbt -DsparkVersion=master clean connectServer/assembly connectClient/test.

Observed that the latest jar releases of Spark are downloaded and the Delta Connect Client test suite's (DeltaTableSuite) ran successfully.

Does this PR introduce any user-facing changes?

No.