Production readiness of SynapseML

[vishalovercome]

I'm very interested in using this package for my office work. It perfectly complements existing suite of tools that come with Spark. I'm particularly interested in:

1. LightGBM and Deep Learaning algorithms
2. ONNX features
3. Explainability features

While I would love to get going, I'm slightly hesitant due to the nature of open bugs - job failures, job hanging, erroneous output. There seem to be a few gotchas as well - the doc mentions gpu support for lightgbm but there are tickets and pull requests indicating that that's not the case. For this reason, I want to know whether you have some sort of roadmap for fixing some of the stability related issues. I know that this project is several years old and would be getting used in production but we can't afford to have jobs failing or getting stuck. Correctness bugs and lack of gpu support are not what I'm concerned with right now.

I would also ask you to enable Discussions for this project as that's a better place to discuss this.

[github-actions[bot]]

Hey @vishalovercome :wave:! Thank you so much for reporting the issue/feature request :rotating_light:. Someone from SynapseML Team will be looking to triage this issue soon. We appreciate your patience.

[k-rush]

Hey @vishalovercome, thanks for your concern and feedback, we'd love to address these issues and encourage you to use the library.

As for LightGBM, @svotaw has recently checked in a changed which addresses some of the perf and reliability issues. He can explain more on the work he's done to support this feature.

Adding in @mhamilton723 to speak to the roadmap for fixing stability issues. In general, we try to address bugs and issues as they come up. If you have any specific concerns on open issues, please let us know so we can prioritize them appropriately.

[vishalovercome]

Thanks @k-rush! As I haven't yet used this library, I can't provide any concrete feedback on the open bugs. However when compared to other open source frameworks I use, I can say the following:

1. Documentation is very very thin
2. Lack of documentation would have been manageable if issues raised in Github were at least acknowledged and quick work arounds were suggested. I for instance have asked for help with just basic installation in #1958 and nobody has responded. Another question I asked over two weeks ago hasn't been answered as well.
3. Key pull requests seem to have been done but aren't getting merged for some reason. I can see 2 pull requests to upgrade Spark from 3.2 to 3.3 (latest Spark version is 3.4), one of which is over 6 months old and even explicitly asked about the timelines and there seems to be no response. This is typically what happens with abandoned projects.
4. Misleading documentation can have huge repercussions. Had I simply got going with this framework only to find out that GPU support for lightGBM isn't really available, then I would have had a lot of explaining to do.

My suggestion would be to greatly enhance the documentation (see https://github.com/microsoft/SynapseML/issues/1955#issue-1702544173), clarify the scope of this project and close out long open PRs

[svotaw]

Apologies for some of the slowness @vishalovercome . The same team that maintains SynapseML was also working on the major release of Microsoft Fabric, announced today at Microsoft //Build. Now that it's out for preview, some of us can get back to flushing out some SynapseML issues for our major 1.0 release. We have been working long hours to make sure that the Fabric release was a success for Data Science, which required a lot of infrastructure changes that weren't focused on features or reliability.

SynapseML will be a part of Fabric, so we are definitely not abandoning this project. We will start cleaning up the PRs and issues.

As far as LightGBM GPUs, the LightGBM native library has support for GPU but only if you build it a certain way. Currently SynapseML does not use a LightGBM build with that flag set since that was not the priority use of the library. I will make it clear in the documentation that that we don't currently support it. I will bring it up in our next SynapseML meeting if that's on the roadmap.

[vishalovercome]

Thanks @svotaw . Congratulations on your release!

You really need to improve your documentation as it might be the reason why some of us aren't able to get started. This is what your doc says:

SynapseML requires Scala 2.12, Spark 3.2+, and Python 3.8+ (https://microsoft.github.io/SynapseML/docs/about/)

However, README.md has this:

SynapseML requires Scala 2.12, Spark 3.2+, and Python 3.6+.

I created an issue #1958 where I couldn't get started and it might be because that environment has Python 3.7. Can you please clarify which one of the two statements is correct?

[JessicaXYWang]

Thanks @svotaw . Congratulations on your release!

You really need to improve your documentation as it might be the reason why some of us aren't able to get started. This is what your doc says:

SynapseML requires Scala 2.12, Spark 3.2+, and Python 3.8+ (https://microsoft.github.io/SynapseML/docs/about/)

However, README.md has this:

SynapseML requires Scala 2.12, Spark 3.2+, and Python 3.6+.

I created an issue #1958 where I couldn't get started and it might be because that environment has Python 3.7. Can you please clarify which one of the two statements is correct?

Thanks @vishalovercome, the Python version on README.md is outdated. It should be 3.8. PR: https://github.com/microsoft/SynapseML/pull/1961/files

[vishalovercome]

LightGBM is supposed to provide the plots: https://lightgbm.readthedocs.io/en/latest/Python-API.html#plotting but the one bundled with SynapseML doesn't seem to have these. If I'm wrong then can you point me to the api doc and if I'm not, can you explain what all features are missing when compared to the non-spark version of lightgbm?

[vishalovercome]

@JessicaXYWang / @svotaw can we use the spark 3.3 package in spark 3.4 also? we have to run on 3.4 as it has some critical bug fixes

[vishalovercome]

Thought I'd follow up on the two questions I had asked a couple of weeks ago:

1. LightGBM is supposed to provide the plots: lightgbm.readthedocs.io/en/latest/Python-API.html#plotting but the one bundled with SynapseML doesn't seem to have these. If I'm wrong then can you point me to the api doc and if I'm not, can you explain what all features are missing when compared to the non-spark version of lightgbm?
2. Can we use the spark 3.3 package in spark 3.4 also? we have to run on 3.4 as it has some critical bug fixes

[svotaw]

For the plots, the link you mentioned is for the LightGBM python API, which is only available if you are directly using the LightGBM library yourself. SynapseML is a wrapper around the LightGBM C library (our own custom build of LightGBM main), and does not expose any LightGBM APIs directly, or add python utilities.

For Spark 3.3 vs 3.4, I'm not sure about any special dependencies of SynapseML for 3.4. @JessicaXYWang? @mhamilton723?