UnsatisfiedLinkError OSX

[chris-smith-zocdoc]

Describe the bug Trying to run tests on OSX 10.14.6, I'm getting the following error indicating the shared library is failing to load.

Cause: java.lang.UnsatisfiedLinkError: /private/var/folders/bz/984j8jp91x37t980k20lx3z1gx50k5/T/mml-natives3454868880605903181/lib_lightgbm.dylib: dlopen(/private/var/folders/bz/984j8jp91x37t980k20lx3z1gx50k5/T/mml-natives3454868880605903181/lib_lightgbm.dylib, 1): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
[info]   Referenced from: /private/var/folders/bz/984j8jp91x37t980k20lx3z1gx50k5/T/mml-natives3454868880605903181/lib_lightgbm.dylib
[info]   Reason: image not found

To Reproduce sbt "testOnly com.microsoft.ml.spark.lightgbm.split2.VerifyLightGBMRanker"

Expected behavior Tests to pass

Info (please complete the following information):

• Latest master
• OSX 10.14.6

** Stacktrace**

[info] + Test Verify LightGBM Ranker on ranking dataset took 0.198s
19/08/26 13:21:24 ERROR Executor: Exception in task 3.0 in stage 16.0 (TID 49)
java.lang.UnsatisfiedLinkError: /private/var/folders/bz/984j8jp91x37t980k20lx3z1gx50k5/T/mml-natives997331245052213810/lib_lightgbm.dylib: dlopen(/private/var/folders/bz/984j8jp91x37t980k20lx3z1gx50k5/T/mml-natives997331245052213810/lib_lightgbm.dylib, 1): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
  Referenced from: /private/var/folders/bz/984j8jp91x37t980k20lx3z1gx50k5/T/mml-natives997331245052213810/lib_lightgbm.dylib
  Reason: image not found
	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
	at java.lang.Runtime.load0(Runtime.java:809)
	at java.lang.System.load(System.java:1086)
	at com.microsoft.ml.spark.core.env.NativeLoader.loadLibraryByName(NativeLoader.java:59)
	at com.microsoft.ml.spark.lightgbm.LightGBMUtils$.initializeNativeLibrary(LightGBMUtils.scala:40)
	at com.microsoft.ml.spark.lightgbm.TrainUtils$$anonfun$15.apply(TrainUtils.scala:404)
	at com.microsoft.ml.spark.lightgbm.TrainUtils$$anonfun$15.apply(TrainUtils.scala:401)
	at com.microsoft.ml.spark.core.env.StreamUtilities$.using(StreamUtilities.scala:28)
	at com.microsoft.ml.spark.lightgbm.TrainUtils$.trainLightGBM(TrainUtils.scala:400)
	at com.microsoft.ml.spark.lightgbm.LightGBMBase$$anonfun$6.apply(LightGBMBase.scala:128)
	at com.microsoft.ml.spark.lightgbm.LightGBMBase$$anonfun$6.apply(LightGBMBase.scala:128)
	at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:188)
	at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:185)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
19/08/26 13:21:24 ERROR Executor: Exception in task 0.0 in stage 16.0 (TID 46)

[chris-smith-zocdoc]

Installing OpenMP through brew resolves this. brew install libomp

A better error could be thrown with this suggestion

[eisber]

@imatiach-msft what's your take?

[shigengtian]

it seems that brew install libomp will install to path /opt/homebrew/opt/libomp/lib I link the libomp.dylib to /usr/local/opt/libomp/lib/libomp.dylib An other error occurred Reason: tried: '/usr/local/opt/libomp/lib/libomp.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/usr/lib/libomp.dylib' (no such file)

[imatiach-msft]

@shigengtian please see the issue in lightgbm repo: https://github.com/microsoft/LightGBM/issues/5035 sorry, currently there is no known workaround or fix for this yet