Spark 3.0 build

[w1nk]

Hello!

We're currently using mmlspark in our spark 2.4 clusters to awesome effect (training 3.4 billion rows, ~600gb of data). Thanks for all the work!

There is a desire within our organization to migrate these clusters to spark 3.0. We attempted to build mmlspark against spark 3, but enough things have been renamed/relocated that there is a pile of build errors [1].

Has anyone successfully built mmlspark against spark 3.0? If not, we may be able to get it patched up to build.

Thanks!

[1] - mmlspark/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/HTTPSourceV2.scala:26:37: object v2 is not a member of package org.apache.spark.sql.sources [error] import org.apache.spark.sql.sources.v2._

This entire v2 package has been renamed / relocated, with the bulk of it seemingly in this patch: https://github.com/apache/spark/commit/053dd858d38e6107bc71e0aa3a4954291b74f8c8

[welcome[bot]]

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it.

[w1nk]

Hello! Spark 3.0.0 was released about a week ago.

https://spark.apache.org/news/spark-3-0-0-released.html

Thanks!

[fengkehh]

Does the MMLSpark team plan to migrate to (Py)Spark 3.0 in the near future?

[brunocous]

I also would be very interested in (Py)Spark 3.0.0 and Scala 2.12 support. How can we help?

[itechbear]

Just saw a related pending PR: https://github.com/Azure/mmlspark/pull/912

[juanpaulo]

Resolved by #970, I suppose?

[saikiranvadhi]

mmlspark doesn't work on Spark 3 yet, throws this error: https://github.com/Azure/mmlspark/issues/891

[imatiach-msft]

latest mmlspark on master supports spark 3.0

[rgordon]

Is it possible to get it pushed to the maven repo so that we can install in into other clusters easily? (I created issue 1031 with this in mind but now see this one too.)

[avanunts]

hi @imatiach-msft Trying to get the latest mmlspark-master-build like you suggest, but getting an error:

Could not find com.microsoft.ml.spark:mmlspark_2.12:1.0.0-rc3-169-80889120-SNAPSHOT.
     Searched in the following locations:
     ...
       - https://mmlspark.azureedge.net/maven/com/microsoft/ml/spark/mmlspark_2.12/1.0.0-rc3-169-80889120-SNAPSHOT/maven-metadata.xml
       - https://mmlspark.azureedge.net/maven/com/microsoft/ml/spark/mmlspark_2.12/1.0.0-rc3-169-80889120-SNAPSHOT/mmlspark_2.12-1.0.0-rc3-169-80889120-SNAPSHOT.pom

Also tried the version from this comment com.microsoft.ml.spark:mmlspark_2.12:1.0.0-rc3-59-bf337941-SNAPSHOT, but got the same error.

Can you pls refine which version should i use?