[BUG] java.lang.NoSuchMethodError when using synapse.ml.lightgbm

[PranavPavan]

SynapseML version

synapseml_2.12:1.0.4

System information

• Language version (e.g. python 3.8, scala 2.12):
• Spark Version (e.g. 3.4.1):
• Spark Platform ( Databricks):
• Databricks Runtime Version (13.3 LTS ML (includes Apache Spark 3.4.1, Scala 2.12))

Describe the problem

I have installed Synapse ML onto my databricks cluster from Maven using cooradinates com.microsoft.azure:synapseml_2.12:1.0.4 which should support spark 3.4.

In my notebook, I have imported the LightGBMRegressor using the code from synapse.ml.lightgbm.LightGBMRegressor import LightGBMRegressor

But when I call the regressor in the notebook, it throws me an error message

Error Message: Error encountered: An error occurred while calling None.com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor. : java.lang.NoSuchMethodError: org.apache.spark.ml.ComplexParamsWritable.$init$(Lorg/apache/spark/ml/ComplexParamsWritable;)V at com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.(LightGBMRegressor.scala:40) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:397) at py4j.Gateway.invoke(Gateway.java:257) at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80) at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:199) at py4j.ClientServerConnection.run(ClientServerConnection.java:119) at java.lang.Thread.run(Thread.java:750)

This is happening because of Spark version mismatch but SynapseML package I installed is expected to support Spark cluster has at least Spark 3.2 and Scala 2.12. This according to theinstallation page of SynapseML. https://microsoft.github.io/SynapseML/docs/Get%20Started/Install%20SynapseML/.

Kindly help me how to resolve it?

Code to reproduce issue

try: from synapse.ml.lightgbm.LightGBMRegressor import LightGBMRegressor lgbm_cv = LightGBMRegressor() except Exception as e: print(f"Error encountered: {e}")

Other info / logs

No response

What component(s) does this bug affect?

• [ ] area/cognitive: Cognitive project
• [ ] area/core: Core project
• [ ] area/deep-learning: DeepLearning project
• [ ] area/lightgbm: Lightgbm project
• [ ] area/opencv: Opencv project
• [ ] area/vw: VW project
• [ ] area/website: Website
• [ ] area/build: Project build system
• [ ] area/notebooks: Samples under notebooks folder
• [ ] area/docker: Docker usage
• [ ] area/models: models related issue

What language(s) does this bug affect?

• [ ] language/scala: Scala source code
• [x] language/python: Pyspark APIs
• [ ] language/r: R APIs
• [ ] language/csharp: .NET APIs
• [ ] language/new: Proposals for new client languages

What integration(s) does this bug affect?

• [ ] integrations/synapse: Azure Synapse integrations
• [ ] integrations/azureml: Azure ML integrations
• [ ] integrations/databricks: Databricks integrations