LightGBM on Apache Spark - support for scala 2.12

[informatica92]

Is there a reason why LightGBM on Apache Spark doesn't have a jar for scala 2.12? It would be nice to have support for both versions (2.11 and 2.12)

[welcome[bot]]

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it.

[shuDaoNan9]

can i see your example？plz

[imatiach-msft]

Agreed, I've seen some conversations around this and I believe the latest was that there are some dependencies of mmlspark which don't yet support 2.11. Otherwise, if those are excluded somehow or resolved, I believe it is straightforward.

[brunocous]

Maybe a pre-lease supporting 2.12 (excluding incompatible dependencies) could be released? Maybe spark 3.0.0 support could also be added to that one?

[informatica92]

It would be great in my opinion... I had to downgrade Scala in order to make it work

[anupam-burra-zocdoc]

We're holding up upgrading our Runtime framework, blocked on this upgrade, so this would be very helpful. Is anyone able to provide an ETA for when this will happen so we can plan around this better?

[juanpaulo]

related to #445 and #912 ?

[juanpaulo]

Looks like this (and related issues) is resolved by the recently merged PR: #970

[nhymxu]

Does anyone know how to try with the nightly build?

[imatiach-msft]

@nhymxu the nightly build is on github with a badge:

[imatiach-msft]

latest mmlspark support scala 2.12

[vinay-singh27]

I have installed the latest MML Spark version(com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc3) on databricks with scala 2.12 & spark 3.0.1 but I am unable to run MMLSpark on it. While running, it gives error "java.lang.NoClassDefFoundError: org/apache/spark/ml/util/MLWritable$class". I believe this issue is because of the non-compatibility with Scala 2.12 or is there anything additional required to run it on Scala 2.12?

[imatiach-msft]

@vinay-singh27 just from this version:

com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc3

you must have installed the 2.11 scala version, not 2.12. RC3 release does not have 2.12 version, it's only available on master. However you can install any master build, for example: coordinate: com.microsoft.ml.spark:mmlspark_2.12:1.0.0-rc3-62-25d40cff-SNAPSHOT repository: https://mmlspark.azureedge.net/maven

you can see latest master in github main page

[cosmincatalin]

Is there any chance we'll get the 2.12 version in Maven Central soon?

[imatiach-msft]

@cosmincatalin we haven't been releasing to maven central for a while, we have always been releasing to our repository: https://mmlspark.azureedge.net/maven all of the rc1-rc3 releases have been there. Do you have any issues with using the repository instead of maven central? You should be able to specify the repository when getting spark packages on all clusters/platforms.

[cosmincatalin]

That's exactly the problem I have, however, I think I have a fix for this, so we're good 😃

[mhamilton723]

Just to re-provide an answer for those looking for help

Our latest maven artifact is for 2.12 https://central.sonatype.com/artifact/com.microsoft.azure/synapseml_2.12/0.11.2