[BUG] TypeError: 'JavaPackage' object is not callable

[yihui8776]

SynapseML version

0.9.4

System information

• Language version : python 3.9.12
• Spark Version : spark 3.3.2
• Spark Platform : Synapse
• Scala Version : 2.12.15

Describe the problem

run the py file in docker to execute LightGBMClassifier got error：

Traceback (most recent call last): File "/notebooks/features/test/test.py", line 33, in model = LightGBMClassifier( File "/opt/spark/python/pyspark/init.py", line 114, in wrapper return func(self, **kwargs) File "/usr/local/lib/python3.9/site-packages/synapse/ml/lightgbm/LightGBMClassifier.py", line 389, in init self._java_obj = self._new_java_obj("com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier", self.uid) File "/opt/spark/python/pyspark/ml/wrapper.py", line 66, in _new_java_obj return java_obj(*java_args) TypeError: 'JavaPackage' object is not callable

Code to reproduce issue

from pyspark.sql import SparkSession

Bootstrap Spark Session

spark = SparkSession.builder.getOrCreate()

#from synapse.ml.core.platform import running_on_synapse

df = ( spark.read.format("csv") .option("header", True) .option("inferSchema", True) .load( #"wasbs://[email protected]/company_bankruptcy_prediction_data.csv" "test/data.csv" ) )

print dataset size

print("records read: " + str(df.count())) print("Schema: ") df.printSchema()

train, test = df.randomSplit([0.85, 0.15], seed=1)

from pyspark.ml.feature import VectorAssembler

feature_cols = df.columns[1:] featurizer = VectorAssembler(inputCols=feature_cols, outputCol="features") train_data = featurizer.transform(train)["Bankrupt?", "features"] test_data = featurizer.transform(test)["Bankrupt?", "features"]

from synapse.ml.lightgbm import LightGBMClassifier

model = LightGBMClassifier( objective="binary", featuresCol="features", labelCol="Bankrupt?", isUnbalance=True )

model = model.fit(train_data)

Other info / logs

No response

What component(s) does this bug affect?

• [ ] area/cognitive: Cognitive project
• [ ] area/core: Core project
• [ ] area/deep-learning: DeepLearning project
• [X] area/lightgbm: Lightgbm project
• [ ] area/opencv: Opencv project
• [ ] area/vw: VW project
• [ ] area/website: Website
• [ ] area/build: Project build system
• [ ] area/notebooks: Samples under notebooks folder
• [ ] area/docker: Docker usage
• [ ] area/models: models related issue

What language(s) does this bug affect?

• [ ] language/scala: Scala source code
• [X] language/python: Pyspark APIs
• [ ] language/r: R APIs
• [ ] language/csharp: .NET APIs
• [ ] language/new: Proposals for new client languages

What integration(s) does this bug affect?

• [X] integrations/synapse: Azure Synapse integrations
• [ ] integrations/azureml: Azure ML integrations
• [ ] integrations/databricks: Databricks integrations

AB#1934663

[github-actions[bot]]

Hey @yihui8776 :wave:! Thank you so much for reporting the issue/feature request :rotating_light:. Someone from SynapseML Team will be looking to triage this issue soon. We appreciate your patience.

[mhamilton723]

Hey @yihui8776 it looks like the scala side is not installed, did you add the spark.jars.packages line to your spark session as described in the install docs for your platform? Thanks for your patience and feedback and hope this helps fix your issue!

[yihui8776]

Hey @yihui8776 it looks like the scala side is not installed, did you add the spark.jars.packages line to your spark session as described in the install docs for your platform? Thanks for your patience and feedback and hope this helps fix your issue!

ok thanks , I found the problem ,I depoly the spark in docker as the document ,but it didnt work .

[aman-solanki-kr]

@yihui8776 I noticed that you were using Docker to run the script. I am facing a similar issue. If you were able to resolve the issue, any suggestions related to the fix would be of great help. Thank you!