[ZEPPELIN-6090] Drop JDK8 from CI

[Reamer]

trafficstars

What is this PR for?

This pull request removes the test with JDK 8. I have kept the JDK matrix to make it easier to switch to the next JDK later. The Livy integration tests have been removed as Livy does not currently support JDK 11. Take a look into, for more information https://lists.apache.org/thread/sk2hc5b5mg7t8sm8rc5mjg9v8n1ozwcz

What type of PR is it?

Improvement

What is the Jira issue?

• https://issues.apache.org/jira/browse/ZEPPELIN-6090

How should this be tested?

• CI

Questions:

• Does the license files need to update? No
• Is there breaking changes for older versions? No
• Does this needs documentation? No

[pan3793]

@Reamer Livy 0.8 seems tested with Spark 3.2.3 https://github.com/apache/incubator-livy/blob/v0.8.0-incubating/dev/docker/livy-dev-spark/Dockerfile#L20

[Reamer]

I am currently testing with Spark version 3.5 and the tests are running except for one.

Error:  Failures: 
Error:    LivyInterpreterIT.testPySparkInterpreter:407 %text 'JavaPackage' object is not callable
Traceback (most recent call last):
  File "/home/runner/work/zeppelin/zeppelin/spark-3.5.3-bin-hadoop3/python/lib/pyspark.zip/pyspark/sql/session.py", line 1444, in createDataFrame
    data, schema, samplingRatio, verifySchema  # type: ignore[arg-type]
  File "/home/runner/work/zeppelin/zeppelin/spark-3.5.3-bin-hadoop3/python/lib/pyspark.zip/pyspark/sql/session.py", line 1485, in _create_dataframe
    rdd, struct = self._createFromLocal(map(prepare, data), schema)
  File "/home/runner/work/zeppelin/zeppelin/spark-3.5.3-bin-hadoop3/python/lib/pyspark.zip/pyspark/sql/session.py", line 1093, in _createFromLocal
    struct = self._inferSchemaFromList(data, names=schema)
  File "/home/runner/work/zeppelin/zeppelin/spark-3.5.3-bin-hadoop3/python/lib/pyspark.zip/pyspark/sql/session.py", line 954, in _inferSchemaFromList
    prefer_timestamp_ntz = is_timestamp_ntz_preferred()
  File "/home/runner/work/zeppelin/zeppelin/spark-3.5.3-bin-hadoop3/python/lib/pyspark.zip/pyspark/sql/utils.py", line 153, in is_timestamp_ntz_preferred
    return jvm is not None and jvm.PythonSQLUtils.isTimestampNTZPreferred()
TypeError: 'JavaPackage' object is not callable
 ==> expected: <SUCCESS> but was: <ERROR>

[pan3793]

I tested Livy 0.8.0 with Spark 3.2.4, using JDK 11, saw another failure, and have no idea ...

Caused by: java.lang.ClassCastException: class Bank cannot be cast to class Bank (Bank is in unnamed module of loader org.apache.spark.repl.ExecutorClassLoader @35007c02; Bank is in unnamed module of loader scala.tools.nsc.interpreter.IMain$TranslatingClassLoader @4dc4e5ff)

[Reamer]

@pan3793 Thanks for https://github.com/apache/zeppelin/pull/4880 I've adjusted this pull request.