Ilya Matiach

@JWenBin have you tried the new single dataset mode parameter on latest master? https://github.com/Azure/mmlspark/pull/1066 In our benchmarking it resolved the low CPU utilization issue.

Sorry about the trouble you are having, not sure. Usually when I see this error it means that the lightgbm jar is not getting loaded somehow.

@zyxue sorry I will have to take a look and try to reproduce this first. "Do I understand correctly that transformed features by StringIndexer will be automatically considered categorical variables?"...

@philmassie it does extend ProbabilisticClassifier in the scala code: https://github.com/Azure/mmlspark/blob/master/src/main/scala/com/microsoft/ml/spark/lightgbm/LightGBMClassifier.scala#L27 which extends hasRawPredictionCol: https://spark.apache.org/docs/latest/api/java/org/apache/spark/ml/classification/ProbabilisticClassifier.html I think the problem is the pyspark wrapper (which calls the scala code) doesn't extend it....

the pyspark wrapper is auto-generated so maybe this is something that needs to be fixed in the autogen code... some of the autogen'ed wrapper is overloaded here: https://github.com/Azure/mmlspark/blob/master/src/main/python/mmlspark/lightgbm/LightGBMClassificationModel.py but it's...

@philmassie no please keep it open it seems like this is indeed an issue that needs to be fixed in the auto-generated pyspark wrapper

@yongliu9975 sorry about the error you are seeing. Is this dataset sparse by chance? I see: ``` LightGBM task generating dense dataset with 394097 rows and 4719 columns ``` I...

@yongliu9975 there must be some bug in this: ``` Java_com_microsoft_ml_lightgbm_lightgbmlibJNI_doubleArray_1setitem+0x0 ``` However I don't see full stack trace so I'm not completely sure which code line is calling it. But,...

I think this is a question for @mhamilton723

@janelu9 yes it should, hope that resolves your question. Is this question specifically related to one of the estimators (eg LIME, lightgbm, KNN, CNTKModel etc?)