Added Shims for adding Databricks 14.3 Support [databricks]

[razajafri]

This PR adds Shims for Databricks 14.3

[razajafri]

build

[razajafri]

@gerashegalov @mythrocks PTAL

[gerashegalov]

Please update Scala 2.13 poms

[razajafri]

@gerashegalov I have addressed your concerns. PTAL

[gerashegalov]

Re failing package-tests (350db143, false) , need to make sure that the db buildvers are not included in the Github Workflow build matrix

[razajafri]

Re failing package-tests (350db143, false) , need to make sure that the db buildvers are not included in the Github Workflow build matrix

Hang on, I will have to modify the workflow file and remove the change from pom.xml

[gerashegalov]

build

[razajafri]

build

[mythrocks]

It's looking good to go, now that the build is passing as well, no?

[NvTimLiu]

@razajafri Can you enable Databricks-14.3 in the pre-merge CI pipeline in this PR as below?

https://github.com/NVIDIA/spark-rapids/blob/branch-24.12/jenkins/Jenkinsfile-blossom.premerge-databricks#L94

values '11.3', '12.2', '13.3', '14.3'

[razajafri]

build

[razajafri]

@razajafri Can you enable Databricks-14.3 in the pre-merge CI pipeline in this PR as below?

https://github.com/NVIDIA/spark-rapids/blob/branch-24.12/jenkins/Jenkinsfile-blossom.premerge-databricks#L94

values '11.3', '12.2', '13.3', '14.3'

I have updated it but this will turn the integration tests on. We don't want the integration tests to run against Databricks 14.3 because we aren't ready.

[razajafri]

build

[razajafri]

build

[razajafri]

build

AWS_INSUFFICIENT_INSTANCE_CAPACITY_FAILURE

[razajafri]

build

[pxLi]

build

[pxLi]

build

AWS_INSUFFICIENT_INSTANCE_CAPACITY_FAILURE

This change requires 8 clusters (4 shims x 2 parallelism) for the CI. We're considering using some escalation strategy, such as switching to different instance types or cloud providers or temporarily disabling parallel runs for Databricks runtimes.

also cc @NvTimLiu

BTW my last trigger works fine