JVM DLL not found. Apple m1

[orakzai]

I'm using Jaydebeapi to connect to Hive but it throws the below error. I tried with both JDK 8 and 15.

OSError: [Errno 0] JVM DLL not found: /Library/Java/JavaVirtualMachines/zulu-15.jdk/Contents/Home/lib/libjli.dylib

I've added the JAVA_HOME in .zshrc file at the end:

#export JAVA_HOME=/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home
export JAVA_HOME=/Library/Java/JavaVirtualMachines/zulu-15.jdk/Contents/Home
export PATH=${PATH}:$JAVA_HOME/bin

[Thrameos]

Is there a lib directory under Home?

[orakzai]

yes there is a lib directory. Btw I installed oracle JDK 8, and the library is able to run without problem. I guess the issue is with the zulu JDK

[vipzgy]

I had the same mistake. using mac m1 python3.9 JPype==1.3.0

Traceback (most recent call last): File "/Users/xxx/miniforge3/lib/python3.9/site-packages/jpype/_core.py", line 226, in startJVM _jpype.startup(jvmpath, tuple(args), OSError: [Errno 0] JVM DLL not found: /Library/Java/JavaVirtualMachines/jdk1.8.0_301.jdk/Contents/Home/jre/lib/jli/libjli.dylib

but i have libjli.dylib in this path.

[nils-tekampe]

I also ran into this issue. Brand new Mac Mini with BigSur and M1 CPU.

Python version: Python 3.9.6 (default, Jun 28 2021, 19:24:41) [Clang 12.0.5 (clang-1205.0.22.9)] on darwin Type "help", "copyright", "credits" or "license" for more information

Java Version openjdk version "1.8.0_292" OpenJDK Runtime Environment (Zulu 8.54.0.21-CA-macosx) (build 1.8.0_292-b10) OpenJDK 64-Bit Server VM (Zulu 8.54.0.21-CA-macosx) (build 25.292-b10, mixed mode)

[Thrameos]

What is the setting of JAVA_HOME and what is the contents of the JAVA_HOME directory?

[nils-tekampe]

Here is some more information.

1. I set the JAVA_HOME explicitely as follows export JAVA_HOME=/Users/nils/.sdkman/candidates/java/8.0.292-zulu
2. this folder contains

➜  ~ ls /Users/nils/.sdkman/candidates/java/8.0.292-zulu
ASSEMBLY_EXCEPTION       THIRD_PARTY_README       include                  readme.txt               zulu-8.jdk
CLASSPATH_EXCEPTION_NOTE Welcome.html             jre                      release
DISCLAIMER               bin                      lib                      sample
LICENSE                  demo                     man                      src.zip

3. I am using the following test.py for testing

from jpype import *
startJVM ("/Users/nils/.sdkman/candidates/java/8.0.292-zulu/jre/lib/server/libjvm.dylib")

Result as mentioned before:

Traceback (most recent call last):
  File "/Users/nils/test.py", line 2, in <module>
    startJVM ("/Users/nils/.sdkman/candidates/java/8.0.292-zulu/jre/lib/server/libjvm.dylib")
  File "/opt/homebrew/lib/python3.9/site-packages/jpype/_core.py", line 226, in startJVM
    _jpype.startup(jvmpath, tuple(args),
OSError: [Errno 0] JVM DLL not found: /Users/nils/.sdkman/candidates/java/8.0.292-zulu/jre/lib/server/libjvm.dylib

And finally:

➜  ~ ls /Users/nils/.sdkman/candidates/java/8.0.292-zulu/jre/lib/server/libjvm.dylib
/Users/nils/.sdkman/candidates/java/8.0.292-zulu/jre/lib/server/libjvm.dylib

[Thrameos]

It seems like the error message is deceiving in this case as the file is clearly present. The issue is how to figure out what the real error is.

My first thought would be making sure that the library actually exists and is a valid binary. Next I would check to see if the file is tagged with the executable flag. If it is not then it likely would fail during the load. From there is gets a bit harder as we would have to alter the code and recompile.

The error message was issued from the code in native/common/jp_exception.cpp

                else if (m_Type == JPError::_os_error_unix)
                {
                        std::stringstream ss;
                        ss << "JVM DLL not found: " << mesg;
                        PyObject* val = Py_BuildValue("(iz)", m_Error.i,
                                        ss.str().c_str());
                        if (val != NULL)
                        {
                                PyObject* exc = PyObject_Call(PyExc_OSError, val, NULL);
                                Py_DECREF(val);
                                if (exc != NULL)
                                {
                                        PyErr_SetObject(PyExc_OSError, exc);
                                        Py_DECREF(exc);
                                }
                        }

But this is just the error handler. The actual source of the error is native/common/jp_platform.cpp

        virtual void loadLibrary(const char* path) override
        {
                JP_TRACE_IN("LinuxPlatformAdapter::loadLibrary");
#if defined(_HPUX) && !defined(_IA64)
                JP_TRACE("shl_load", path);
                jvmLibrary = shl_load(path, BIND_DEFERRED | BIND_VERBOSE, 0L);
#else
                JP_TRACE("dlopen", path);
                jvmLibrary = dlopen(path, RTLD_LAZY | RTLD_GLOBAL);
#endif // HPUX
                // GCOVR_EXCL_START
                if (jvmLibrary == NULL)
                {
                        JP_TRACE("null library");
                        JP_TRACE("errno", errno);
                        if (errno == ENOEXEC)
                        {
                                JP_TRACE("dignostics", dlerror());
                        }
                        JP_RAISE_OS_ERROR_UNIX( errno, path);
                }
                // GCOVR_EXCL_STOP
                JP_TRACE_OUT; // GCOVR_EXCL_LINE
        }

The key part is the error number. We are currently getting as 0, which is not a valid value as far as I am aware. So the reason for the fail is dlopen returned NULL but did not set an errno. So to make headway we would need to find some way to print additional diagnostics as to why dlopen did not work. I know of no reason for a NULL without an errno.

[nils-tekampe]

An explicit chmod+x on the library did not change anything. Is there anything else that I can do to support debugging this?

[Thrameos]

My best guess is that it is related to this ticket. Unfortunately I don't have access to such a system so I can't really test to see if this is a solution.

https://support.azul.com/hc/en-us/articles/360039650212-On-MacOS-10-15-3-and-later-Azul-Zulu-Builds-of-OpenJDK-Java-doesn-t-load-the-JNI-Libraries-my-Application-needs

[Thrameos]

Reading some of the documentation on "notarized" shared libraries it seems very likely this is the issue. I can't find any technical documentation on how the model operates that would give me a way to catch and resolve this error from within JPype.

The apparent issue is that Python and JPype are installed at one protection level and Java was installed at another and thus calls to dlopen will fail. The solution is to have all at the same privilege level (likely by using a copy of Java outside of /Library). Why the error is silent preventing us from issuing a proper error is baffling. Without some error message that I can propagate to user there is no way this will not create endless headaches and support tickets. At best I could check if the file exists and is executable and if it still fails change the error message to something that indicates an unspecified permission error.

There is nothing about this in the online man pages that I can access. https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/dlopen.3.html

Other python libraries have similar issues in that if one piece is not notarized then the process fails.
https://stackoverflow.com/questions/67834070/error-loading-python-lib-after-notarizing-macos-application

[nils-tekampe]

@Thrameos I am not sure that this is the case here. In my Installation the Java is installed into the home directory. The “system Java” that resides in Library should not be involved at all. Or am I mistaken?

[Thrameos]

I think it has something to do with which applications on the system are signed and located in trusted paths. If Python and JPype were signed but Java was not then it would fail. The same is true in the reverse case. Unfortunately I don't know enough about the process to determine which is signed and which are not, nor do I have technical specs on how to make sure this is even the right solution. I am afraid we would need an apple expert to weigh in.

Here is the only example I found thus far.

https://docs.azul.com/prime/Native-Application

Is there a libjvm.so file as well as the dylib file? Maybe we are trying to load the wrong file? If so manually give a path to startJVM with the so file and see if that fixes it.

The next most likely problem is that the LD_LIBRARY_PATH is missing something the shared library requires. I remember a case in which I had to add the $JAVA_HOME/bin to LD_LIBRARY_PATH because some shared library was in bin for unknown reason. There should be a utility that scans a shared library and tells if all the shared library dependencies can be satisfied.

Also from that doc it appears there is a function called dlerror() perhaps printing that value (you would need to recompile JPype) to see if it can tell us what is going on. I would just add printf("%s\n", dlerror()); after JP_TRACE("errno", errno); to see if we can get some more information.

[nils-tekampe]

@Thrameos I made added the printf statements and compiled jpype locally. The printed messages are:

dlopen(/Users/nils/.sdkman/candidates/java/8.0.292-zulu/jre/lib/server/libjvm.dylib, 9): no suitable image found.  Did find:
	/Users/nils/.sdkman/candidates/java/8.0.292-zulu/jre/lib/server/libjvm.dylib: mach-o, but wrong architecture
	/Users/nils/.sdkman/candidates/java/8.0.292-zulu/zulu-8.jdk/Contents/Home/jre/lib/server/libjvm.dylib: mach-o, but wrong architecture

I guess that the problem is that the JDK that is used here, is not native for M1. When the java excutable is called directly, I guess that the Rosetta framework kicks in but this does not seem to be the case here. I will try to find a JDK that has been compiled for M1 natively next.

[Thrameos]

Okay it sounds like the issue is identified. Python and the JVM must be the same architecture. I can try to improve the error message though it won't fix the underlaying issue.

[nils-tekampe]

Making the error message more precise may help others. I guess that it would have saved me 3 hours of debugging.

[capelastegui]

I ran into this problem and I managed to fix it by installing an M1-native JVM (azul OpenJDK, .dmg version), following the instructions from this thread: https://stackoverflow.com/questions/64788005/java-jdk-for-apple-m1-chip/64881417#64881417

[TudorEsan]

Hi, i tried to download all versions of the jdk (the arm ones), it does not seem to work... But i tried to download the x86 one and it magically did work. Maybe you should try both the arm and the x86 versions

[Thrameos]

The architecture of python and the jdk must match. So if python is x86 then the jdk must be as well.

[TudorEsan]

The architecture of python and the jdk must match. So if python is x86 then the jdk must be as well.

That makes perfect sense! Thank you

[yanxg]

I got the same error. I changed the java that is used in /usr/bin to my newly-installed Zulu11 java: /Applications/zulu11.56.19-ca-jdk11.0.15-macosx_aarch64/bin/java Then it works for me. Basically, in the ~/.bash_profile or ~/.zshrc, you add export PATH="/Applications/zulu11.56.19-ca-jdk11.0.15-macosx_aarch64/bin:$PATH" I think this is caused by Jaydebeapi uses some particular java version (zulu proves java11) while the /usr/bin uses different java version

[arikg10]

The following solution worked for me: (based on: https://stackoverflow.com/questions/64788005/java-jdk-for-the-apple-m1-chip/64881417#64881417)

Use sdkmanager. vi ~/.sdkman/etc/config
sdkman_rosetta2_compatible=false should be already present in the file, make sure it is set to false. List available jdks: sdk list java Install the desired JDK, i.e: sdk install java 8.0.292-zulu Make sure your $JAVA_HOME is updated.

[yanxg]

I solved this problem by

1. Download the ARM64 jdk https://www.azul.com/downloads/?version=java-11-lts&os=macos&architecture=arm-64-bit&package=jdk
2. untar it to /Applications/
3. Make sure libel.dylib exists in /Library/Java/JavaVirtualMachines/zulu-11.jdk/Contents/Home/lib/jli/libjli.dylib (note that it in lib/jli/)
4. As pointed out by @Thrameos that you need to make sure the Python is ARM version instead of Intel version

[Ackbach]

None of the above solutions worked for me, but this did:

https://pyspi-toolkit.readthedocs.io/en/latest/faq.html

The AdoptOpenJDK appears to be the ticket for the M1 and M2 chips, for a Python-compatible library.

[johnygomez]

What worked for me recently is using the ARM64 JDK as suggested in https://stackoverflow.com/a/74398849 I.e. install https://adoptium.net/temurin/releases/?version=11 for MacOS Arm64 and make sure JAVA_HOME is set properly. After then the JPype started successfully :v:

[dulinanaaa]

就是版本的问题，之前安装的是官网下载的“jdk-8u371-macosx-x64.dmg”的x86版本，也有类似的问题， 在Azul 下载了arm版本的，就能正常使用了https://adoptium.net/zh-CN/temurin/releases/?version=11&os=mac&arch=arm

[Suleman-Tariq]

I solved this problem by

1. Download the ARM64 jdk https://www.azul.com/downloads/?version=java-11-lts&os=macos&architecture=arm-64-bit&package=jdk
2. untar it to /Applications/
3. Make sure libel.dylib exists in /Library/Java/JavaVirtualMachines/zulu-11.jdk/Contents/Home/lib/jli/libjli.dylib (note that it in lib/jli/)
4. As pointed out by @Thrameos that you need to make sure the Python is ARM version instead of Intel version

This above solution worked for me.