Apple Silicon (ARM) Macs (arm64e) not supported yet because of missing dynamic library

[xelax]

Running lightgbm on a Mac with apple silicon (M1 chip, aarch64 architecture) fails because of missing native library for the architecture:

22/02/16 10:51:32 ERROR LightGBMRanker: {"uid":"LightGBMRanker_7c727e2d1c9e","className":"class com.microsoft.azure.synapse.ml.lightgbm.LightGBMRanker","method":"train","buildVersion":"0.9.5"}
org.apache.spark.SparkException: Job aborted due to stage failure: Could not recover from a failed barrier ResultStage. Most recent failure reason: Stage failed because barrier task ResultTask(13, 0) finished unsuccessfully.
java.lang.UnsatisfiedLinkError: /private/var/folders/31/56fwtfy17t520t96p0zwjnt40000gq/T/mml-natives618985592717473613/lib_lightgbm.dylib: dlopen(/private/var/folders/31/56fwtfy17t520t96p0zwjnt40000gq/T/mml-natives618985592717473613/lib_lightgbm.dylib, 0x0001): tried: '/private/var/folders/31/56fwtfy17t520t96p0zwjnt40000gq/T/mml-natives618985592717473613/lib_lightgbm.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/lib/lib_lightgbm.dylib' (no such file)
	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1950)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1832)
	at java.lang.Runtime.load0(Runtime.java:811)
	at java.lang.System.load(System.java:1088)
	at com.microsoft.azure.synapse.ml.core.env.NativeLoader.loadLibraryByName(NativeLoader.java:66)
	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMUtils$.initializeNativeLibrary(LightGBMUtils.scala:39)
	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.trainLightGBM(LightGBMBase.scala:356)

I think the fix should be as simple as adding the appropriate library inside lightgbmlib-3.2.110.jar. Let me know if I can help with building it.

[mhamilton723]

Thanks for bringing this up @xelax , I believe the fix would have to be added to the lightGBM repo itself as they package and publish this jar. @imatiach-msft who can point you to more info on this

[xelax]

ok. should I open an issue at https://github.com/microsoft/LightGBM/issues ?

[jcdang]

If anyone is wondering how to get it working without getting lost in all these upstream toolchains and CIs, this worked for me:

git clone --recursive [email protected]:microsoft/LightGBM.git
cd LightGBM
cmake -S . -B build -DAPPLE_OUTPUT_DYLIB=ON
cmake --build build

[nikos-kal]

Hey @jcdang - sorry to revive an old issue but unfortunately it's still unresolved.

I tried to recreate your solution but still got the same error as @xelax

Would you mind sharing your full configuration (Java version, LightGBM version, any rebuilt jars)?

[jcdang]

Hey @nikos-kal, I was setting up the pipelines for a client that I'm not working with right now so I don't have the exact details. I've tested it to work with java8 and java17. LightGBM 3.2.X. Worked with SynapseML 0.10.2 and 0.9.5.

I forgot but I think you may also need the USE_SWIG flag too.

git clone --recursive [email protected]:microsoft/LightGBM.git
cd LightGBM
cmake -S . -B build -DAPPLE_OUTPUT_DYLIB=ON -DUSE_SWIG=ON
cmake --build build

The apple+arm jar should be in the folder. You need to replace artifact jar referencing com.microsoft.ml.lightgbm:lightgbmlib in your JVM project or through your local artifact repository (ivy/mvn etc...) with the one you built.

You can test this quickly from java/scala with the following:

com.microsoft.azure.synapse.ml.lightgbm.LightGBMUtils.initializeNativeLibrary()

Sorry I can't be of more help. I couldn't figure out how to run the unit tests without using azure cli.

[nikos-kal]

Thanks @jcdang 🙌

I actually ended up on an alternate solution for now:

• use x86 brew to install libomp
• Switch to x86 Java

This combination works but obviously there is some performance hit due to emulation.

Hopefully the MS team can build native jars in the near future so we don’t have to rely on workarounds.

[mhamilton723]

Thanks @nikos-kal for tfollow up. CLosing this issue but will update if LGBM provides the right natives for this