[LIVY-702]: Submit Spark apps to Kubernetes

[askhatri]

This pull request (PR) is the foundational PR for adding Kubernetes support in Apache Livy, originally found here. This update includes a newer version of the Kubernetes client and adds code to display the Spark UI.

Summary of the Proposed Changes

This PR introduces a method to submit Spark applications to a Kubernetes cluster. The key points covered include:

• Submitting batch sessions
• Submitting interactive sessions
• Monitoring sessions, collecting logs, and gathering diagnostic information
• Restoring session monitoring after restarts
• Garbage collection (GC) of created Kubernetes resources

JIRA link: https://issues.apache.org/jira/browse/LIVY-702

How was this patch tested?

• Unit Tests: The patch has been verified through comprehensive unit tests.
• Manual Testing: Conducted manual testing using Kubernetes on Docker Desktop.
  • Environment: Helm charts.

For detailed instructions on testing using Helm charts, please refer to the documentation available at livycluster.

[gyogal]

Thanks for working on this @askhatri and for retesting and updating the original code in PR #249 . If there are no objections, we could merge this PR and any smaller updates or fixes could be added in follow-up tickets before the 0.9 release.

[vikas-saxena02]

can this chnaged be merged if there are not objections?

[vikas-saxena02]

@gyogal @lmccay , can you please take a look at this PR?

[jshmchenxi]

@askhatri Thanks for pushing this feature forward!

As a side note, could you please add the original author, @jahstreet, to the author list? We've been using this feature for a year and truly appreciate Alex's initial effort in bringing it to Livy.

[askhatri]

Hi @jshmchenxi , I have added the original author, @jahstreet, to the author list as suggested by you. Please let me know incase if any further change or correction required

[jahstreet]

Hi @jshmchenxi , I have added the original author, @jahstreet, to the author list as suggested by you. Please let me know incase if any further change or correction required

Thx for mentioning mate, appreciate the credits 🙏 .

[vikas-saxena02]

@askhatri the latest push was forced and I can see many of the checks have failed, can you please look into this?

[askhatri]

Hi @vikas-saxena02, I have re-triggered the checks now by adding ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true.

[vikas-saxena02]

@jahstreet @askhatri is there any documentation that you can share that explains how to use livy run jobs on spark on kubernetes?

[vikas-saxena02]

/approve

[jahstreet]

@jahstreet @askhatri is there any documentation that you can share that explains how to use livy run jobs on spark on kubernetes?

Yes, we will talk about packaging it. Until then, people can refer:

• (newer linked to this PR @askhatri 's setup) https://github.com/askhatri/livycluster
• (older my setup) https://github.com/JahstreetOrg/spark-on-kubernetes-helm
• Jira issue: https://issues.apache.org/jira/browse/LIVY-979

[vikas-saxena02]

/assign @gyogal

[askhatri]

Hi @vikas-saxena02, I have assigned to @gyogal as suggested by you...!

[vikas-saxena02]

Thanks @askhatri !! Hopefully this should be merged soon.

[gyogal]

Thanks everyone for your input! It seems like there are no objections and the overall feedback is positive. If you find any issues once this PR is merged, please feel free to raise a ticket.