Dataflint causes cluster startup failure on Databricks Runtime 16.2

[guyazu-oligo]

Describe the bug When trying to launch a cluster on DBR 16.2 with a global init script to install Dataflint, there's an issue when loading a dependency:

ERROR DriverDaemon$: XXX Fatal uncaught exception. Terminating driver.
java.lang.NoClassDefFoundError: org/json4s/JsonAST$JValue
at org.apache.spark.dataflint.DataflintSparkUIInstaller.loadUI(DataflintSparkUILoader.scala:104)
at org.apache.spark.dataflint.DataflintSparkUIInstaller.install(DataflintSparkUILoader.scala:92)
at org.apache.spark.dataflint.DataflintSparkUILoader$.install(DataflintSparkUILoader.scala:116)
at io.dataflint.spark.SparkDataflintDriverPlugin.registerMetrics(SparkDataflintPlugin.scala:26)
at org.apache.spark.internal.plugin.DriverPluginContainer.$anonfun$registerMetrics$1(PluginContainer.scala:76)

This is likely because on DBR 16.2 the json4s component was updated - see the docs here

Environemnt spark version: 3.5.2 platform: Databricks dataflint version: 0.3.0

To Reproduce Steps to reproduce the behavior:

1. Add global init script as suggested in the docs
2. Create a new all purpose cluster on DBR 16.2
3. Let the cluster start up
4. See the error when its trying to boot up

Expected behavior Cluster should start up on DBR 16.2 with Dataflint installed.

Screenshots

Additional context

[menishmueli]

@guyazu-oligo thanks for the detailed report! I managed to recreate the problem on our databricks test environment. We will try to add support to DBR 16.2 in DataFlint version 0.3.1

DataFlint depends on the existing json interface, so we will need to see how can we make it work in this version while not breaking old DBR versions.

[menishmueli]

Version 0.3.1 didn't solve the problem, we tried to use different signatures but it didn't work.

We will try a different approach next version.

[menishmueli]

Hi @guyazu-oligo , apologies for the long wait!

Fixed the problem in version v0.4.1 it was a challenging fix.. Tested it on DBR 16.4 and it works.

Closing this issue, let me know if you have any issues and in this case I will reopen it.