Ilya Matiach

@1120172175 this looks like some issue with the cluster configuration? How are you specifying the mmlspark package? Sorry I haven't seen this issue before, I'm not sure how to resolve...

I wonder if this is helpful, but maybe it's not helpful at all: https://stackoverflow.com/questions/41112801/property-spark-yarn-jars-how-to-deal-with-it ``` I found by hit-n-trial that correct syntax of this property is spark.yarn.jars=hdfs://xx:9000/user/spark/share/lib/*.jar ```

what are the full file paths you are using to specify the jars?

@mhamilton723 may know the exact commit since he does the releases. "I am want to update this version to be compatible with Spark 3 as all our models were trained...

@DCameronMauch pyspark api: https://github.com/Azure/mmlspark/blob/master/src/main/python/mmlspark/lightgbm/mixin.py#L10 java api: https://github.com/Azure/mmlspark/blob/master/src/main/scala/com/microsoft/ml/spark/lightgbm/LightGBMRegressor.scala#L131 example notebook showing how to use save/load in pyspark: https://github.com/Azure/mmlspark/blob/663d9650d3884ece260a457d9b016088380c2cb9/notebooks/samples/LightGBM%20-%20Overview.ipynb

to install mmlspark from master (copied from another thread): please try this walkthrough with pictures on databricks: https://docs.microsoft.com/en-us/azure/cognitive-services/big-data/getting-started#azure-databricks for spark 2.4.5 you can use rc1 to rc3 releases. For latest...

@DCameronMauch for this error: java.util.NoSuchElementException: Failed to find a default value for actualNumClasses I've sent a PR here: https://github.com/Azure/mmlspark/pull/1057/files is there a stack trace for that error? Once build completes...

also did you use the saveNativeModel/loadNativeModel or the spark save/load APIs (https://stackoverflow.com/questions/33027767/save-ml-model-for-future-usage)? I would think if you used save/loadNativeModel it wouldn't generate that error, perhaps only the spark save/load.

@DCameronMauch for building I follow this guide: https://github.com/Azure/mmlspark/blob/master/docs/developer-readme.md mainly sbt setup, sbt compile - I think sbt package builds the jar after the first two steps

@DCameronMauch the build for this PR: https://github.com/Azure/mmlspark/pull/1057 is: Maven Coordinates com.microsoft.ml.spark:mmlspark_2.12:1.0.0-rc3-85-c987ad0b-SNAPSHOT Maven Resolver https://mmlspark.azureedge.net/maven please try it out, I hope it fixes your issue