[SPARK-47954][K8S] Support creating ingress entry for external UI access

[pan3793]

What changes were proposed in this pull request?

Ingress a kind of K8s resource to expose HTTP and HTTPS routes from outside the K8s cluster to services within the K8s cluster. It's a common use case that uses ingress to access Spark UI for Spark jobs running on K8s in cluster mode.

This PR aims to add such a built-in feature to allow users to enable Ingress for Spark jobs which running on K8s cluster mode.

State: functionality completed; manuelly tested; ready for review to collect feedback TODO: UT and docs

Why are the changes needed?

Simplify Spark live UI access from outside of the K8s.

Does this PR introduce any user-facing change?

This PR adds a new feature, but is disabled by default.

How was this patch tested?

I will supply the unit tests later.

Maunnely tested on an internal K8s cluster.

$ build/sbt clean package -Pkubernetes
$ bin/docker-image-tool.sh -r ****** -b java_image_tag=17 -t SPARK-47954 build
$ bin/docker-image-tool.sh -r ****** -t SPARK-47954 push
$ SPARK_PREPEND_CLASSES=true bin/spark-submit \
	--master=k8s://https://***********.org:6443 \
	--deploy-mode cluster \
	--driver-class-path='/opt/spark/examples/jars/*' \
	--driver-memory=512m \
	--executor-memory=512m \
	--executor-cores=1 \
	--num-executors=1 \
	--conf spark.kubernetes.context='*****' \
	--conf spark.kubernetes.authenticate.driver.serviceAccountName=***** \
	--conf spark.kubernetes.namespace=spark \
	--conf spark.kubernetes.file.upload.path=hdfs://*****/spark-staging \
	--conf spark.kubernetes.container.image=******/spark:SPARK-47954 \
	--conf spark.kubernetes.driver.ingress.enabled=true \
	--conf spark.kubernetes.driver.ingress.host={{APP_ID}}.bigdata*********.com \
	--conf spark.kubernetes.driver.ingress.ingressClassName=contour \
	--class org.apache.spark.examples.SparkPi \
	spark-internal 1000

$ kubectl get ingress org-apache-spark-examples-sparkpi-a947e18f0c055185-driver-svc-ingress
NAME                                                                    CLASS     HOSTS                                                         ADDRESS       PORTS   AGE
org-apache-spark-examples-sparkpi-a947e18f0c055185-driver-svc-ingress   contour   spark-cbf851baf8f74a1aa64b94d6e1d1b350.bigdata*********.com   10.49.128.6   80      2m32s

Was this patch authored or co-authored using generative AI tooling?

No.

[mridulm]

+CC @zhouyejoe

[pan3793]

cc @dongjoon-hyun @yaooqinn @LuciferYang @EnricoMi appreciate it if you could have a look on this idea

[dongjoon-hyun]

Gentle ping, @pan3793 . It would be great if we can land this before Apache Spark 4.0.0-preview.

To @cloud-fan , just a question, when do you think you are going to cut 4.0.0-preview tag?

[pan3793]

@dongjoon-hyun sorry for late, I'm a little busy these days, will address comments soon

[dongjoon-hyun]

For the record, Apache Spark 4.0.0-preview is scheduled on next Monday.

[dongjoon-hyun]

Just FYI, please take your time. We can target this for Apache Spark 4.0.0.

[github-actions[bot]]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!