[FEATURE] Kyuubi proxy spark on kubernetes driver pod ui

[zwangsheng]

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Search before asking

• [X] I have searched in the issues and found no similar issues.

Describe the feature

Help proxy spark on kubernetes cluster deploy-mode spark ui.

Motivation

When Spark On Kubernetes with cluster deploy-mode, spark driver runs in the pods, using the Kubernetes network. Users cannot directly access the spark ui. Related to Spark On Kubernetes Doc[2], you need to kubectl port-forward for each spark application, which is very inconvenient. So related to Kyuubi Issue 2681[1], we design using kyuubi to help port forward spark ui.

Describe the solution

See more detail with Google Doc https://docs.google.com/document/d/1FOMELnyAN9JSKeYi3qSgcVukvj8IUzYk7KF0eW5CNc8/edit#

Additional context

#2681

SubTask

• [ ] Spark SQL Engine register zookeeper with it's ui info [SUBTASK] [KYUUBI #3420] Spark SQL Engine register zk with ui info
• [ ] Spark SQL Engine register etcd with it's ui info
• [ ] Kyuubi read engine ui info from HA and port forward request
• [ ] Kyuubi Web UI engine tab related to ui

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

[zwangsheng]

Ping @pan3793 @yaooqinn @ulysses-you

[minyk]

we're planning similar feature for our product, but using svc object of spark-driver pod is our approach. we think this is more reliable than ip address in the zookeeper.

[zwangsheng]

we're planning similar feature for our product, but using svc object of spark-driver pod is our approach. we think this is more reliable than ip address in the zookeeper.

Thanks for your reply. This issue aims to help Kyuubi to have the ability to expose Spark UI On Kubernetes, so as to more conveniently implement Spark On Kubernetes solutions without relying on external solutions. Of course, we also welcome other solutions that can help us to implement Access UI.