[FEATURE] Support exposing extra TCP ports in Spark Driver via K8s Ingress

[hiboyang]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?

I plan to extend Spark UI Ingress and support exposing extra TCP ports in Spark Driver via K8s Ingress. This is to support some use cases like exposing Spark Connect server side port in Driver to network traffic outside of the K8s cluster. Another use case is to run Jupyter Notebook Server or Kernel Gateway inside Spark Driver, so user could use Jupyter Notebook within the Spark application.

Describe the solution you would like

Similar like Spark UI Ingress, I would add extra configuration in Spark Application CRD to allow an application to specify what extra ports to expose via K8s Ingress.

Describe alternatives you have considered

Another alternative is to use Spark conf to specify what extra ports to expose, not modifying Spark Application CRD. However, there is already SparkUIOptions field in Spark Application CRD, it looks natural to add similar fields instead of using Spark conf.

Additional context