[TASK][MEDIUM] Make KSHC binary compatible with multiple Spark versions

[pan3793]

What's the level of this task?

MEDIUM

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Search before creating

• [X] I have searched in the task list and found no similar tasks.

Mentor

• [X] I have sufficient expertise on this task, and I volunteer to be a mentor of this task to guide contributors through the task.

Skill requirements

• Knowledge of GitHub Actions, Java Reflections, Hive, Spark

Background and Goals

Currently, the KSHC(Kyuubi Spark Hive Connector) supports Spark 3.3, 3.4, and 3.5, but unlike the Spark engine, the KSHC jar built with Spark 3.5 can not work on Spark 3.4 runtime.

Implementation steps

• Tune the KSHC code to fix binary-compatible issues
• Tune the test code and set up the GitHub Actions workflow to do cross-version verification

Additional context

Introduction of 2024H1 Kyuubi Code Contribution Program

[beryllw]

Looks like what I worked for before #5384 , please assign to me if possible.

[pan3793]

@beryllw thank you

[SteNicholas]

@pan3793, any update? I would like to take this ticket.

[zhouyifan279]

@pan3793 @SteNicholas I created a PR #6436 to do KSHC cross-version test. I'd like to take this ticket if it's OK.