Run cells with specific language

[shay1bz]

Description

Addresses the same functionality as #598 but with some differences, this PR is for allowing users to execute cells with specific languages against the same Spark Session. Given that #598 is nearly 5 years old and seems abandoned, this implementation provides a minimal, backward-compatible solution.

Due to a lack of familiarity with the sparkmagic code base, no tests have been added. I kindly request guidance on how to implement unit tests.

The changes have been manually tested on a notebook, ensuring compatibility with shared DataFrames and UDFs between Scala and Python.

Checklist

• [x] Wrote a description of my changes above
• [x] Formatted my code with black
• [x] Added a bullet point for my changes to the top of the CHANGELOG.md file
• [ ] Added or modified unit tests to reflect my changes
• [x] Manually tested with a notebook
• [ ] If adding a feature, there is an example notebook and/or documentation in the README.md file

[shay1bz]

@devstein would you mind taking a look at this?

[devstein]

Hey @shay1bz 👋 sorry for the late response. Are you still opening to contributing? I can help with testing. We'll also want to add an example notebook

[shay1bz]

Hi @devstein . Yes, I’d be happy to move this PR forward. We've been using it in my organization for a while, and it’s proven to be very useful.

[devstein]

@shay1bz Amazing 🙌

Let me know how I can help

[shay1bz]

@devstein I can provide the example notebook, but I'll need some assistance with writing the unit tests. From my perspective, we mainly need to verify that the language parameter, when specified, is correctly passed to the Livy HTTP request.

[shay1bz]

Hi @devstein, I have added an example notebook. How can we move forward?