Support SBT 1.xx and latest databricks

[jdongca2003]

Main changes: 1) support SBT 1.6.2 2) support spark 3.1.* and latest databricks in Azure 3) Skip spark dependency in assembly 4) Package the python script based on standard pip package

[agupta004]

@julioasotodv @jdongca2003 I am looking forward to use spark_tree_plotting library in Databricks. Any updates on this pull request?

[Arnold1]

@agupta004 @jdongca2003 @julioasotodv any idea how to use that jar file in Databricks? I created the jar file with:

$ git config --get remote.origin.url
https://github.com/jdongca2003/spark-tree-plotting

$ git branch
* master
$ git log
commit b3bc0f7a6d2aa977527609b6405472fd1f17693e (HEAD -> master, origin/master, origin/HEAD)
Author: JIANXIONG DONG <[email protected]>
Date:   Fri Feb 11 19:25:17 2022 -0800

    package the python scripts based on standard pip package

$ sbt version
[info] welcome to sbt 1.6.2 (Oracle Corporation Java 1.8.0_231)
[info] loading settings for project spark-tree-plotting-build from assembly.sbt ...
[info] loading project definition from /Users/arnold1/spark_plotting_test/spark-tree-plotting/project
[info] loading settings for project spark-tree-plotting from build.sbt ...
[info] set current project to spark-tree-plotting (in build file:/Users/arnold1/spark_plotting_test/spark-tree-plotting/)
[info] 0.3

$ sbt clean; sbt compile; sbt assembly
[info] welcome to sbt 1.6.2 (Oracle Corporation Java 1.8.0_231)
[info] loading settings for project spark-tree-plotting-build from assembly.sbt ...
[info] loading project definition from /Users/arnold1/spark_plotting_test/spark-tree-plotting/project
[info] loading settings for project spark-tree-plotting from build.sbt ...
[info] set current project to spark-tree-plotting (in build file:/Users/arnold1/spark_plotting_test/spark-tree-plotting/)
[success] Total time: 1 s, completed Jun 1, 2022 10:56:37 PM
[info] welcome to sbt 1.6.2 (Oracle Corporation Java 1.8.0_231)
[info] loading settings for project spark-tree-plotting-build from assembly.sbt ...
[info] loading project definition from /Users/arnold1/spark_plotting_test/spark-tree-plotting/project
[info] loading settings for project spark-tree-plotting from build.sbt ...
[info] set current project to spark-tree-plotting (in build file:/Users/arnold1/spark_plotting_test/spark-tree-plotting/)
[info] Executing in batch mode. For better performance use sbt's shell
[info] compiling 1 Scala source to /Users/arnold1/spark_plotting_test/spark-tree-plotting/target/scala-2.12/classes ...
[warn] there was one feature warning; re-run with -feature for details
[warn] one warning found
[success] Total time: 17 s, completed Jun 1, 2022 10:57:06 PM
[info] welcome to sbt 1.6.2 (Oracle Corporation Java 1.8.0_231)
[info] loading settings for project spark-tree-plotting-build from assembly.sbt ...
[info] loading project definition from /Users/arnold1/spark_plotting_test/spark-tree-plotting/project
[info] loading settings for project spark-tree-plotting from build.sbt ...
[info] set current project to spark-tree-plotting (in build file:/Users/arnold1/spark_plotting_test/spark-tree-plotting/)
[info] Strategy 'discard' was applied to a file (Run the task at debug level to see details)
[success] Total time: 3 s, completed Jun 1, 2022 10:57:17 PM

than imported that jar file in Databricks (https://docs.databricks.com/libraries/workspace-libraries.html).

but still get this error: ModuleNotFoundError: No module named 'spark_tree_plotting' when I try to import:

from spark_tree_plotting import plot_tree