Java-Thread-Affinity icon indicating copy to clipboard operation
Java-Thread-Affinity copied to clipboard

Published 20 hours ago • verified publisher icon OpenHFT

Resolved java.lang.UnsatisfiedLinkError on M1 mac with Zing

Open hft-team-city opened this issue 3 years ago • 1 comments

to reproduce run all test of /Users/teamcity/projects/Chronicle-Queue-Enterprise on m1 mac

[main/queue-cluster-1-replication-event-loop] WARN net.openhft.chronicle.threads.VanillaEventLoop - MediumEventLoop has been terminated due to exception
java.lang.UnsatisfiedLinkError: /Users/teamcity/Library/Caches/JNA/temp/jna1914262682919037562.tmp: dlopen(/Users/teamcity/Library/Caches/JNA/temp/jna1914262682919037562.tmp, 1): no suitable image found.  Did find:
	/Users/teamcity/Library/Caches/JNA/temp/jna1914262682919037562.tmp: no matching architecture in universal wrapper
	/Users/teamcity/Library/Caches/JNA/temp/jna1914262682919037562.tmp: no matching architecture in universal wrapper
	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1950)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1832)
	at java.lang.Runtime.load0(Runtime.java:811)
	at java.lang.System.load(System.java:1088)
	at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1018)
	at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988)
	at com.sun.jna.Native.<clinit>(Native.java:195)
	at net.openhft.affinity.impl.OSXJNAAffinity$CLibrary.<clinit>(OSXJNAAffinity.java:76)
	at net.openhft.affinity.impl.OSXJNAAffinity.getThreadId(OSXJNAAffinity.java:66)
	at net.openhft.affinity.Affinity.getThreadId(Affinity.java:173)
	at net.openhft.affinity.AffinityLock.bind(AffinityLock.java:383)
	at net.openhft.affinity.AffinityLock.assignCurrentThread(AffinityLock.java:356)
	at net.openhft.affinity.LockInventory.updateLockForCurrentThread(LockInventory.java:73)
	at net.openhft.affinity.LockInventory.acquireLock(LockInventory.java:110)
	at net.openhft.affinity.AffinityLock.acquireLock(AffinityLock.java:297)
	at net.openhft.affinity.AffinityLock.acquireLock(AffinityLock.java:182)
	at net.openhft.affinity.AffinityLock.acquireLock(AffinityLock.java:280)
	at net.openhft.chronicle.threads.MediumEventLoop.run(MediumEventLoop.java:242)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	at net.openhft.chronicle.core.threads.CleaningThread.run(CleaningThread.java:96)
[main/queue-cluster-1-~monitor] WARN net.openhft.chronicle.threads.internal.EventLoopThreadHolder - Monitoring a task which has finished VanillaEventLoop{name='queue-cluster-1-replication-event-loop', parent=EventGroup@1, service=java.util.concurrent.Executors$FinalizableDelegatedExecutorService@753a7223, highHandler=NOOP, mediumHandlers=[software.chronicle.enterprise.queue.replication.ReplicatedQueue$Initialiser@7105f85c, software.chronicle.enterprise.queue.replication.ReplicatedQueue$Initialiser@34077818], timerHandlers=[], daemonHandlers=[], newHandler=null, pauser=net.openhft.chronicle.threads.LongPauser@2374e947}

hft-team-city avatar Jun 05 '21 07:06 hft-team-city

Currently the TeamCity mac build is not compiling.

[04:45:11] :	 [Step 1/3] javah -force -classpath ../../../target/classes -d ../../../target/classes/../jni software.chronicle.enterprise.internals.impl.NativeAffinity net.openhft.ticker.impl.JNIClock
[04:45:11] :	 [Step 1/3] g++ -O3 -Wall -shared -fPIC -L/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre/lib/amd64/server -L/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre/lib/i386/server  -L/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre/lib/amd64/jrockit/ -L/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre/lib/i386/jrockit/ -L/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre/lib/ppc64le/server -L/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre/lib/ppc64le/jrockit/ -ljvm -lrt -I /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/include -I /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/include/linux -I ../../../target/classes/../jni software_chronicle_enterprise_internals_impl_NativeAffinity.cpp net_openhft_ticker_impl_JNIClock.cpp -o ../../../target/classes/libCEInternals.so
[04:45:11]W:		 [net.openhft:affinity] In file included from software_chronicle_enterprise_internals_impl_NativeAffinity.cpp:21:
[04:45:11]W:		 [net.openhft:affinity] /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/include/jni.h:45:10: fatal error: 'jni_md.h' file not found
[04:45:11]W:		 [net.openhft:affinity] #include "jni_md.h"
[04:45:11]W:		 [net.openhft:affinity]          ^~~~~~~~~~
[04:45:11]W:		 [net.openhft:affinity] 1 error generated.
[04:45:11]W:		 [net.openhft:affinity] In file included from net_openhft_ticker_impl_JNIClock.cpp:21:
[04:45:11]W:		 [net.openhft:affinity] /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/include/jni.h:45:10: fatal error: 'jni_md.h' file not found
[04:45:11]W:		 [net.openhft:affinity] #include "jni_md.h"
[04:45:11]W:		 [net.openhft:affinity]          ^~~~~~~~~~
[04:45:11]W:		 [net.openhft:affinity] 1 error generated.
[04:45:11]W:		 [net.openhft:affinity] make: *** [../../../target/classes/libCEInternals.so] Error 1
[04:45:11]W:	 [Step 1/3] [ERROR] Command execution failed.
[04:45:11] :	 [Step 1/3] org.apache.commons.exec.ExecuteException: Process exited with an error: 2 (Exit value: 2)
[04:45:11] :	 [Step 1/3]     at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:404)
[04:45:11] :	 [Step 1/3]     at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:166)
[04:45:11] :	 [Step 1/3]     at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:982)
[04:45:11] :	 [Step 1/3]     at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:929)

JerryShea avatar Aug 25 '21 03:08 JerryShea

Im experiencing the same issue

dosier avatar Oct 31 '22 00:10 dosier

@dosier when using Zing?

JerryShea avatar Nov 01 '22 00:11 JerryShea

@JerryShea as discussed a while ago, I am also experiencing this on my m1 mac (Apple M1 Pro). I'm also using chronicle-bom 2.23.136, as it seems thats the latest BOM that has all the dependencies resolved properly (from maven central).

❯ java -version
openjdk version "17.0.4.1" 2022-08-12 LTS
OpenJDK Runtime Environment Zulu17.36+17-CA (build 17.0.4.1+1-LTS)
OpenJDK 64-Bit Server VM Zulu17.36+17-CA (build 17.0.4.1+1-LTS, mixed mode, sharing)

java.lang.UnsatisfiedLinkError: Can't load library: /Users/wibowoa/Library/Caches/JNA/temp/jna3430404444573156635.tmp
	at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2393)
	at java.base/java.lang.Runtime.load0(Runtime.java:755)
	at java.base/java.lang.System.load(System.java:1953)
	at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1018)
	at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988)
	at com.sun.jna.Native.<clinit>(Native.java:195)
	at net.openhft.affinity.impl.OSXJNAAffinity$CLibrary.<clinit>(OSXJNAAffinity.java:75)
	at net.openhft.affinity.impl.OSXJNAAffinity.getThreadId(OSXJNAAffinity.java:66)
	at net.openhft.affinity.Affinity.getThreadId(Affinity.java:173)
	at net.openhft.affinity.AffinityLock.bind(AffinityLock.java:392)
	at net.openhft.affinity.AffinityLock.assignCurrentThread(AffinityLock.java:365)
	at net.openhft.affinity.LockInventory.updateLockForCurrentThread(LockInventory.java:85)
	at net.openhft.affinity.LockInventory.acquireLock(LockInventory.java:152)
	at net.openhft.affinity.AffinityLock.acquireLock(AffinityLock.java:305)
	at net.openhft.affinity.AffinityLock.acquireLock(AffinityLock.java:178)
	at net.openhft.affinity.AffinityLock.acquireLock(AffinityLock.java:148)
	at net.openhft.affinity.AffinityLock.acquireLock(AffinityLock.java:274)
	at net.openhft.chronicle.threads.MediumEventLoop.run(MediumEventLoop.java:219)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
	at net.openhft.chronicle.core.threads.CleaningThread.run(CleaningThread.java:174)

I have also used the following JVM args, as you suggested:

--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED
--add-exports=java.base/sun.nio.ch=ALL-UNNAMED
--add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED
--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED
--add-opens=jdk.compiler/com.sun.tools.javac=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED

Let me know if you need any more information from me.

alexwibowo avatar Jan 05 '23 02:01 alexwibowo

Looks like the issue is because of dependencies from net.openhft:affinity

In my case, through chronicle bom (3.23.136), I imported net.openhft:affinity:3.23.2 which brings the following:

  • net.java.dev.jna:jna-platform:5.5.0
  • net.java.dev.jna:jna:5.5.0

If i override them to:

  • net.java.dev.jna:jna-platform:5.12.1
  • net.java.dev.jna:jna:5.12.1

Then the problem went away. Hope that helps!

FYI: https://github.com/java-native-access/jna/issues/1313

alexwibowo avatar Jan 06 '23 02:01 alexwibowo

Currently the TeamCity mac build is not compiling. I don't know why it is using zing

[04:45:11] :	 [Step 1/3] javah -force -classpath ../../../target/classes -d ../../../target/classes/../jni software.chronicle.enterprise.internals.impl.NativeAffinity net.openhft.ticker.impl.JNIClock
[04:45:11] :	 [Step 1/3] g++ -O3 -Wall -shared -fPIC -L/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre/lib/amd64/server -L/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre/lib/i386/server  -L/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre/lib/amd64/jrockit/ -L/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre/lib/i386/jrockit/ -L/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre/lib/ppc64le/server -L/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre/lib/ppc64le/jrockit/ -ljvm -lrt -I /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/include -I /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/include/linux -I ../../../target/classes/../jni software_chronicle_enterprise_internals_impl_NativeAffinity.cpp net_openhft_ticker_impl_JNIClock.cpp -o ../../../target/classes/libCEInternals.so
[04:45:11]W:		 [net.openhft:affinity] In file included from software_chronicle_enterprise_internals_impl_NativeAffinity.cpp:21:
[04:45:11]W:		 [net.openhft:affinity] /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/include/jni.h:45:10: fatal error: 'jni_md.h' file not found
[04:45:11]W:		 [net.openhft:affinity] #include "jni_md.h"
[04:45:11]W:		 [net.openhft:affinity]          ^~~~~~~~~~
[04:45:11]W:		 [net.openhft:affinity] 1 error generated.
[04:45:11]W:		 [net.openhft:affinity] In file included from net_openhft_ticker_impl_JNIClock.cpp:21:
[04:45:11]W:		 [net.openhft:affinity] /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/include/jni.h:45:10: fatal error: 'jni_md.h' file not found
[04:45:11]W:		 [net.openhft:affinity] #include "jni_md.h"
[04:45:11]W:		 [net.openhft:affinity]          ^~~~~~~~~~
[04:45:11]W:		 [net.openhft:affinity] 1 error generated.
[04:45:11]W:		 [net.openhft:affinity] make: *** [../../../target/classes/libCEInternals.so] Error 1
[04:45:11]W:	 [Step 1/3] [ERROR] Command execution failed.
[04:45:11] :	 [Step 1/3] org.apache.commons.exec.ExecuteException: Process exited with an error: 2 (Exit value: 2)
[04:45:11] :	 [Step 1/3]     at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:404)
[04:45:11] :	 [Step 1/3]     at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:166)
[04:45:11] :	 [Step 1/3]     at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:982)
[04:45:11] :	 [Step 1/3]     at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:929)

Is it zing? as far as i can tell, thats zulu, which i thought is openjdk ?

alexwibowo avatar Jan 06 '23 03:01 alexwibowo

Thanks @alexwibowo ! And yes I must have been confused when I was talking about Zing.

For internal use, we have had another issue with JNA versioning - https://github.com/ChronicleEnterprise/Chronicle-FIX-Demo/issues/326

JerryShea avatar Jan 06 '23 04:01 JerryShea

@rogersimmons is this still an issue?

peter-lawrey avatar May 24 '23 04:05 peter-lawrey

A fresh checkout of CQE today builds cleanly and all tests pass on Apple Silicon (Azul, 1.8). If you still see issues pls let us know details.

openjdk version "1.8.0_372"
OpenJDK Runtime Environment (Zulu 8.70.0.23-CA-macos-aarch64) (build 1.8.0_372-b07)
OpenJDK 64-Bit Server VM (Zulu 8.70.0.23-CA-macos-aarch64) (build 25.372-b07, mixed mode)

[INFO] 
[INFO] Results:
[INFO] 
[WARNING] Tests run: 473, Failures: 0, Errors: 0, Skipped: 22
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  06:01 min
[INFO] Finished at: 2023-05-24T10:02:32+01:00
[INFO] ------------------------------------------------------------------------

rogersimmons avatar May 24 '23 09:05 rogersimmons