[LIVY-703] Show SparkUI link on Livy UI when running on Kubernetes

[jahstreet]

What changes were proposed in this pull request?

Jira

This PR is one of the PRs in the series related to the splitting of the PR https://github.com/apache/incubator-livy/pull/167 to multiple PRs to ease and speed up review and merge processes.

This PR targets branch of the base PR https://github.com/apache/incubator-livy/pull/249 and should be merged after it.

This PR implements a way to build Spark UI links on Livy UI when running on Kubernetes. This patch is required to unlock further integrations with Kubernetes Ingress resources (ref: https://github.com/apache/incubator-livy/pull/167).

Refer https://github.com/jahstreet/incubator-livy/pull/35 to compare changes.

How was this patch tested?

Unit tests.

Manual testing with Kubernetes on Docker Desktop for Mac v2.1.0.1. Environment - Helm charts:

• cluster-base with custom-values.yaml:

nginx-ingress:
  controller:
    service:
      loadBalancerIP: 127.0.0.1 # my-cluster.example.com IP address (from /etc/hosts)
      loadBalancerSourceRanges: []
cluster-autoscaler:
  enabled: false
oauth2-proxy:
  enabled: false

• spark-cluster with custom-values.yaml:

livy:
  image:
    pullPolicy: Never
    tag: 0.7.0-incubating-spark_2.4.3_2.11-hadoop_3.2.0-dev
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
      kubernetes.io/tls-acme: "true"
      nginx.ingress.kubernetes.io/rewrite-target: /$1
    path: /livy/?(.*)
    hosts:
    - my-cluster.example.com
    tls:
    - secretName: spark-cluster-tls
      hosts:
      - my-cluster.example.com
  persistence:
    enabled: true
  env:
    LIVY_LIVY_UI_BASE1PATH: {value: "/livy"}
    LIVY_SPARK_KUBERNETES_CONTAINER_IMAGE_PULL1POLICY: {value: "Never"}
    LIVY_SPARK_KUBERNETES_CONTAINER_IMAGE: {value: "sasnouskikh/livy-spark:0.7.0-incubating-spark_2.4.3_2.11-hadoop_3.2.0-dev"}
    LIVY_LIVY_SERVER_SESSION_STATE0RETAIN_SEC: {value: "300s"}
    LIVY_LIVY_SERVER_KUBERNETES_ALLOWED1NAMESPACES: {value: "default,test"}
    LIVY_LIVY_UI_KUBERNETES_SPARKUI_ENABLED: {value: "true"}
    LIVY_LIVY_UI_KUBERNETES_SPARKUI_LINK0FORMAT: {value: "/spark-ui/%s"}
historyserver:
  enabled: false
jupyterhub:
  enabled: true
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
      kubernetes.io/tls-acme: "true"
    hosts:
    - my-cluster.example.com
    pathSuffix: ''
    tls:
    - secretName: spark-cluster-tls
      hosts:
      - my-cluster.example.com
  hub:
    baseUrl: /jupyterhub
    publicURL: "https://my-cluster.example.com"
    activeServerLimit: 10
    # $> openssl rand -hex 32
    cookieSecret: 41b85e5f50222b1542cc3b38a51f4d744864acca5e94eeb78c6e8c19d89eb433
    pdb:
      enabled: true
      minAvailable: 0
  proxy:
    # $> openssl rand -hex 32
    secretToken: cc52356e9a19a50861b22e08c92c40b8ebe617192f77edb355b9bf4b74b055de
    pdb:
      enabled: true
      minAvailable: 0
  cull:
    enabled: false
    timeout: 300
    every: 60

• Interactive sessions - Jupyter notebook on JupyterHub with Sparkmagic
• Batch sessions - SparkPi:

curl -k -H 'Content-Type: application/json' -X POST \
  -d '{
        "name": "SparkPi-01",
        "className": "org.apache.spark.examples.SparkPi",
        "numExecutors": 2,
        "file": "local:///opt/spark/examples/jars/spark-examples_2.11-2.4.3.jar",
        "args": ["10000"],
        "conf": {
            "spark.kubernetes.namespace": "<namespace>"
        }
      }' "https://my-cluster.example.com/livy/batches"

[codecov-io]

Codecov Report

Merging #252 into master will decrease coverage by 1.22%. The diff coverage is 42.58%.

@@             Coverage Diff              @@
##             master     #252      +/-   ##
============================================
- Coverage     67.92%   66.69%   -1.23%     
- Complexity      939      956      +17     
============================================
  Files           102      103       +1     
  Lines          5876     6189     +313     
  Branches        891      946      +55     
============================================
+ Hits           3991     4128     +137     
- Misses         1314     1464     +150     
- Partials        571      597      +26

Impacted Files	Coverage Δ	Complexity Δ
...ain/java/org/apache/livy/rsc/driver/RSCDriver.java	79.25% <0%> (-1.51%)	45 <0> (ø)
rsc/src/main/java/org/apache/livy/rsc/RSCConf.java	88.18% <100%> (+0.33%)	9 <1> (+1)	:arrow_up:
...main/scala/org/apache/livy/server/LivyServer.scala	32.88% <20%> (-0.46%)	11 <1> (ø)
...e/livy/server/interactive/InteractiveSession.scala	68.93% <33.33%> (-0.44%)	46 <0> (ø)
...ala/org/apache/livy/utils/SparkKubernetesApp.scala	42.2% <42.2%> (ø)	14 <14> (?)
...rc/main/scala/org/apache/livy/utils/SparkApp.scala	45.23% <5.55%> (-30.77%)	1 <0> (ø)
...rver/src/main/scala/org/apache/livy/LivyConf.scala	95.02% <85%> (-1%)	22 <1> (+1)
...ain/scala/org/apache/livy/utils/SparkYarnApp.scala	67.32% <0%> (+1.3%)	40% <0%> (ø)	:arrow_down:
... and 3 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update c7313c5...4050c0d. Read the comment docs.

[jahstreet]

Feel free to tag me for review on future incremental PRs that touch the Livy core, conf, and ui code (especially the UI stuff since I wrote that whole library)

Thanks man, I appreciate that!