[KYUUBI #6265] Resource isolation in Spark Scala mode

[beryllw]

:mag: Description

Issue References 🔗

This pull request fixes #6265

Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Types of changes :bookmark:

• [ ] Bugfix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Behavior Without This Pull Request :coffin:

Behavior With This Pull Request :tada:

Related Unit Tests

---

Checklist 📝

• [ ] This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 82.14286% with 5 lines in your changes are missing coverage. Please review.

Project coverage is 58.45%. Comparing base (1591157) to head (e3c652e). Report is 12 commits behind head on master.

Files	Patch %	Lines
...e/kyuubi/engine/spark/operation/ExecuteScala.scala	66.66%	3 Missing :warning:
...a/org/apache/spark/sql/hive/HiveClientHelper.scala	85.71%	1 Missing and 1 partial :warning:

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #6331      +/-   ##
============================================
- Coverage     58.47%   58.45%   -0.03%     
  Complexity       24       24              
============================================
  Files           653      655       +2     
  Lines         39769    39904     +135     
  Branches       5472     5484      +12     
============================================
+ Hits          23256    23326      +70     
- Misses        14030    14078      +48     
- Partials       2483     2500      +17     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[pan3793]

can you summarize your design/change on the PR description?

[beryllw]

can you summarize your design/change on the PR description?

I've been quite busy recently,, sorry for not replying in time.

For now ExecuteScala use spark.sharedState.jarClassLoader to get the URLs for loading classes and resources, and then repl loaded.SharedState do not support isolated class loaded.

SparkSession use HiveSessionResourceLoader to support isolated class loaded.So we can get the URLs for loading classes and resources by HiveClientImpl.clientLoader.classLoader.getURLs.

Spark 3.5 use JobArtifactSet to support add jar/resources to different path.

We can use it to support add resources isolated.And then repl loaded isolated classes/resources from HiveSessionResourceLoader.

[beryllw]

Only Spark 3.5 supports JobArtifactState. I need to use reflection to solve the compatibility problem, but I don’t know how to use reflection to call def withActiveJobArtifactState[T](state: JobArtifactState)(block: => T): T = {}.

[beryllw]

@pan3793 cc, and need some help, thank you.