declarativewidgets
declarativewidgets copied to clipboard
jupyter
(feature request) upgrade to latest Toree version (with support for Spark 2.0)
As you are probably aware, Toree now supports spark 2.0, would be super-nice if DW could upgrade as well.
Thanks in advance.
@tmoerman have you tested this already. I thought that the Spark guys retained the DataFrame type around.
Also... there is no packaged version of Toree on top of Spark 2.0 yet. We are working on it.
I can confirm there's an issue I've got toree kernels for 1.6 (0.1.0.dev5) and 2.0 (0.2.0.dev1) running and the updating textbox fails to fall a function with the spark 2.0 kernel but is fine under spark 1.6. There is one conspicuous but uninformative ERROR in the jupyper's stderr.
I then downloaded the source for declarative widgets and toree 0.2.0 to compile, this fails due to a missing class: SparkIMain, messing around I found toree switched to using IMain and eventually got a hacky version of declarative widgets to compile with a dependency of toree 0.2.0 and spark 2.0.1. I'm still testing but as I need it for some dashboards I'll probably get something going.
If there's interest I can provide my detailed notes and a patch likely next week. I suspect the way it should be is that jars are published for different spark versions to Maven Central/clojars/Bintray and PyPi so one can %addjar the one needed.
@DarinJ Any chance you still have the notes and patch for getting this working with Spark 2.0, Toree 0.2?
