Ilya Matiach
Ilya Matiach
latest mmlspark support scala 2.12
@vinay-singh27 just from this version: com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc3 you must have installed the 2.11 scala version, not 2.12. RC3 release does not have 2.12 version, it's only available on master. However you...
@cosmincatalin we haven't been releasing to maven central for a while, we have always been releasing to our repository: https://mmlspark.azureedge.net/maven all of the rc1-rc3 releases have been there. Do you...
@sebastian-janisch For missing values I don't recall right now -- I will need to take a look to validate that we have tests for missing values – and, if not,...
I believe those are the parameters to make LightGBM deterministic. I'm wondering if the issue is with how you are creating the spark dataframe. Perhaps this link might help: https://stackoverflow.com/questions/55468810/how-do-i-get-deterministic-random-ordering-in-pyspark...
I'm betting this would fix it for sure: ``` train_data = train_data.cache() # run the same code twice prediction_one = fit_model_and_get_predictions(train_data) prediction_two = fit_model_and_get_predictions(train_data) ``` will need to take a...
@kiminsigne @EthanRock yes pyspark support for custom objective function is one of the top things I am looking into adding. We just recently added custom objective function in the scala...
@andrew-arkhipov it is supported in the scala API, see param here: https://github.com/microsoft/SynapseML/blob/master/lightgbm/src/main/scala/com/microsoft/ml/spark/lightgbm/params/LightGBMParams.scala#L305 see here for param definition: https://github.com/microsoft/SynapseML/blob/master/lightgbm/src/main/scala/com/microsoft/ml/spark/lightgbm/params/FObjParam.scala see example here in scala: https://github.com/microsoft/SynapseML/blob/master/lightgbm/src/test/scala/com/microsoft/ml/spark/lightgbm/split1/VerifyLightGBMClassifier.scala#L338 It's not yet supported in pyspark...
I don't believe GPU binaries are packaged currently, so I don't think it will work out of the box without some packaging changes and possibly code changes