microsoft
Add warning log - newly created Hyperspace context for different Spark Session
What is the context for this pull request?
- Tracking Issue: n/a
- Parent Issue: n/a
- Dependencies: n/a
What changes were proposed in this pull request?
Currently, Hyperspace keeps only one HyperspaceContext for each Hyperspace object. So if an app uses multiple concurrent Spark sessions using one Hyperspace object & one thread, a new Hyperspace context is continuously created for each request.
As Hyperspace object is not thread-safe, we tried to force - "One client thread" should use only "one Spark Session", but we can't because it's possible to access from multiple threads with the same SparkSession (e.g. delta lake - broadcast join query execution).
So for now, we left some warning log in that case, so that users could notice it might an ineffective use of Hyperspace.
Does this PR introduce any user-facing change?
Yes, warning log can be left if a Hyperspace object is accessed with different Spark Session.
How was this patch tested?
tested on local env
Could you update the PR description as it seems out of date? Basically, we are preventing hyperspace context from being re-created in certain scenarios, and making sure too many contexts are not created.
