eclipse-openj9
Full path for System.load() is not honoured in AIX
When trying to do a System.load() on AIX, where the full path to the library is provided, a library with the same name from the builtin libraries is loaded instead (not sure of the full contents of the builtin libraries, but in this case they contain the libraries bundled with the JDK under the lib directory).
The issue was originally observed and can be reproduced by initializing the OpenJCEPlusFIPS provider, while specifying an OCK library from a location other than the lib directory of the JDK.
For example, when running ./jdk-21.0.6+2/bin/java -Djava.security.debug=jceplus -Dock.library.path=/home/jenkins/kostas/OCK/ TestLibLoad.java on AIX where TestLibLoad.java just calls new OpenJCEPlusFIPS(), I get the following stack trace:
jceplus: New OpenJCEPlusFIPS instance
jceplus: Loading ock library using value in property ock.library.path: /home/jenkins/kostas/OCK/
jceplus: Loaded : /home/jenkins/kostas/OCK/libjgsk8iccs_64.so
jceplus: Libpath not found for jgskit, use java home directory.
jceplus: Loading jgskit library using value: /home/jenkins/kostas/jdk-21.0.6+2/lib
jceplus: Loaded : /home/jenkins/kostas/jdk-21.0.6+2/lib/libjgskit.so
jceplus: Loading ock library using value in property ock.library.path: /home/jenkins/kostas/OCK/
jceplus: dependent library load path : /home/jenkins/kostas/OCK
jceplus: dependent library install path : /home/jenkins/kostas/jdk-21.0.6+2/lib/C
Exception in thread "main" java.lang.IllegalStateException: Cause already initialized
at java.base/java.lang.Throwable.initCause(Throwable.java:381)
at openjceplus/com.ibm.crypto.plus.provider.OpenJCEPlusFIPS.setOCKExceptionCause(OpenJCEPlusFIPS.java:819)
at openjceplus/com.ibm.crypto.plus.provider.OpenJCEPlusFIPS.providerException(OpenJCEPlusFIPS.java:813)
at openjceplus/com.ibm.crypto.plus.provider.OpenJCEPlusFIPS.initializeContext(OpenJCEPlusFIPS.java:756)
at openjceplus/com.ibm.crypto.plus.provider.OpenJCEPlusFIPS$1.run(OpenJCEPlusFIPS.java:86)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:692)
at openjceplus/com.ibm.crypto.plus.provider.OpenJCEPlusFIPS.<init>(OpenJCEPlusFIPS.java:78)
at TestLibLoad.main(TestLibLoad.java:11)
Caused by: java.security.ProviderException: Failed to initialize OpenJCEPlusFIPS provider
at openjceplus/com.ibm.crypto.plus.provider.OpenJCEPlusFIPS.providerException(OpenJCEPlusFIPS.java:812)
at openjceplus/com.ibm.crypto.plus.provider.OpenJCEPlusFIPS.initializeContext(OpenJCEPlusFIPS.java:756)
at openjceplus/com.ibm.crypto.plus.provider.OpenJCEPlusFIPS$1.run(OpenJCEPlusFIPS.java:86)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:692)
at openjceplus/com.ibm.crypto.plus.provider.OpenJCEPlusFIPS.<init>(OpenJCEPlusFIPS.java:78)
at TestLibLoad.main(TestLibLoad.java:11)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:586)
at jdk.compiler/com.sun.tools.javac.launcher.Main.execute(Main.java:484)
at jdk.compiler/com.sun.tools.javac.launcher.Main.run(Main.java:208)
at jdk.compiler/com.sun.tools.javac.launcher.Main.main(Main.java:135)
Caused by: java.security.ProviderException: Dependent library was loaded from /home/jenkins/kostas/jdk-21.0.6+2/lib/C
at openjceplus/com.ibm.crypto.plus.provider.ock.NativeInterface.validateLibraryLocation(NativeInterface.java:323)
at openjceplus/com.ibm.crypto.plus.provider.ock.OCKContext.createContext(OCKContext.java:43)
at openjceplus/com.ibm.crypto.plus.provider.OpenJCEPlusFIPS.initializeContext(OpenJCEPlusFIPS.java:751)
... 9 more
Notice how the dependent library load path (i.e., what I specified) is /home/jenkins/kostas/OCK, but the dependent library install path (i.e., where the library is actually loaded from) is /home/jenkins/kostas/jdk-21.0.6+2/lib/C
Running the same command (i.e., ./jdk-21.0.6+2/bin/java -Djava.security.debug=jceplus -Dock.library.path=/root/java/ock_load/OCK/ TestLibLoad.java) on Linux gives us the following:
jceplus: New OpenJCEPlusFIPS instance
jceplus: Loading ock library using value in property ock.library.path: /root/java/ock_load/OCK/
jceplus: Loaded : /root/java/ock_load/OCK/libjgsk8iccs_64.so
jceplus: Libpath not found for jgskit, use java home directory.
jceplus: Loading jgskit library using value: /root/java/ock_load/jdk-21.0.6+2/lib
jceplus: Loaded : /root/java/ock_load/jdk-21.0.6+2/lib/libjgskit.so
jceplus: Loading ock library using value in property ock.library.path: /root/java/ock_load/OCK/
jceplus: dependent library load path : /root/java/ock_load/OCK
jceplus: dependent library install path : /root/java/ock_load/OCK/C
Looking into the loading process a bit, I think that in the NativeLibraries.java class an attempt is made to find the library in the builtin ones using just the name (https://github.com/ibmruntimes/openj9-openjdk-jdk21/blob/2a7c9a4bd8d7c820698222e7d06a3655751f9564/src/java.base/share/classes/jdk/internal/loader/NativeLibraries.java#L136).
I'm not sure why the builtin libraries contain the lib directory only in AIX, but it seems problematic that there is no way to enforce using the full library paths.
Just to be clear, this behaviour is only observed on AIX where the builtin libraries contain the ones in the lib directory of the JDK.
Issue Number: 20646 Status: Open Recommended Components: comp:vm, comp:jclextensions, comp:build
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Update:
Reproduced the ProviderException in question, it was thrown because ockLoadPath doesn't match ockInstallPath.
jceplus: dependent library load path : /team/jasonf/tbd/jceplus-lib
jceplus: dependent library install path : /team/jasonf/jceplus/jdk-21.0.7+4/lib/C
The code snippets in question are https://github.com/ibmruntimes/OpenJCEPlus/blob/7d2d25a0c3c68effb418713b90b31c41fcff5a0d/src/main/java/com/ibm/crypto/plus/provider/ock/NativeInterface.java#L217-L234
String ockLoadPath = new File(getOCKLoadPath()).getCanonicalPath();
String ockInstallPath = new File(context.getOCKInstallPath()).getCanonicalPath();
if (debug != null) {
debug.println("dependent library load path : " + ockLoadPath);
debug.println("dependent library install path : " + ockInstallPath);
}
if (ockInstallPath.startsWith(ockLoadPath) == false) {
String exceptionMessage = "Dependent library was loaded from an external location";
if (debug != null) {
exceptionMessage = "Dependent library was loaded from " + ockInstallPath;
}
throw new ProviderException(exceptionMessage);
@KostasTsiounis just wondering which jceplus libraries are expected to be loaded from ockLoadPath (via -Djgskit.library.path and -Dock.library.path), how was ockInstallPath determined? It seems from https://github.com/ibmruntimes/OpenJCEPlus/blob/7d2d25a0c3c68effb418713b90b31c41fcff5a0d/src/main/native/StaticStub.c#L141, but not sure how that was associated with System.load(). I do apologize for the late response.
@KostasTsiounis just wondering which
jcepluslibraries are expected to be loaded fromockLoadPath(via-Djgskit.library.pathand-Dock.library.path), how wasockInstallPathdetermined? It seems from https://github.com/ibmruntimes/OpenJCEPlus/blob/7d2d25a0c3c68effb418713b90b31c41fcff5a0d/src/main/native/StaticStub.c#L141, but not sure how that was associated withSystem.load(). I do apologize for the late response.
The OCK (formerly known as ICC, https://github.com/IBM/OpenCryptographyKitC) library is loaded from the ockLoadPath (it is the jgsk8iccs_64.so for example in x86-64_linux). This is the native library based on OpenSSL that is FIPS certified and responsible for running the cryptographic algorithms.
The ockInstallPath is coming from the actual native OCK library after it's loaded. It has a way of determining where it was loaded from. It returns that information through an OCKContext, which is what you see in the C code that you are mentioning. At this point the library is already loaded.
The problem lies with the fact that we call System.load() here with the path defined in ockLoadPath, but when asking the library itself where it was loaded from, it returns an ockInstallPath that is different than the ockLoadPath. Keep in mind that this only occurs on AIX.
Thanks for the insights.
Run with the command line options: /team/jasonf/jceplus/jdk-21.0.7+4/bin/java -Djgskit.library.path=/team/jasonf/tbd/jceplus-lib -Dock.library.path=/team/jasonf/tbd/jceplus-lib -Djava.security.debug=jceplus
18:22:38.286 0x3001f100 j9scar.113 > JVM_LoadLibrary(name=/team/jasonf/tbd/jceplus-lib/libjgsk8iccs_64.so)
18:22:38.288 0x3001f100 j9scar.230 - JVM_LoadLibrary(name=/team/jasonf/tbd/jceplus-lib/libjgsk8iccs_64.so) j9sl_open_shared_library
18:22:38.326 0x3001f100 j9scar.113 > JVM_LoadLibrary(name=/team/jasonf/tbd/jceplus-lib/libjgskit.so)
18:22:38.329 0x3001f100 j9scar.230 - JVM_LoadLibrary(name=/team/jasonf/tbd/jceplus-lib/libjgskit.so) j9sl_open_shared_library
Both libjgsk8iccs_64.so and libjgskit.so are loaded from the directory specified via -Djgskit.library.path and -Dock.library.path just like x86-64_linux which doesn't throw ProviderException. I don't see a difference at AIX from the Xtrace output.
A couple of more questions, is there any other native library involved other than libjgsk8iccs_64.so and libjgskit.so?
The ockInstallPath is coming from the actual native OCK library after it's loaded. It has a way of determining where it was loaded from. It returns that information through an OCKContext, which is what you see in the C code that you are mentioning. At this point the library is already loaded.
Is there any debug setting that shows ockInstallPath retrieval?
Both
libjgsk8iccs_64.soandlibjgskit.soare loaded from the directory specified via-Djgskit.library.pathand-Dock.library.pathjust likex86-64_linuxwhich doesn't throwProviderException. I don't see a difference at AIX from theXtraceoutput.
Do you mean that even in AIX the libraries are loaded from the appropriate paths and the exception is avoided?
Is there any debug setting that shows
ockInstallPathretrieval?
I'm not sure since this is from the OCK (https://github.com/IBM/OpenCryptographyKitC) project that is not our code. I can check to see if there are additional options available. @jasonkatonica Are you aware of any further debugs for this?
A couple of more questions, is there any other native library involved other than
libjgsk8iccs_64.soandlibjgskit.so?
No other libraries are involved by the way.
In my example, I wasn't specifying -Djgskit.library.path since I don't really care where libjgskit.so is loaded from and that didn't seem to be an issue in other platforms. But could those two be somehow tied on AIX?
Both libjgsk8iccs_64.so and libjgskit.so are loaded from the directory specified via -Djgskit.library.path and -Dock.library.path just like x86-64_linux which doesn't throw ProviderException. I don't see a difference at AIX from the Xtrace output.
Do you mean that even in AIX the libraries are loaded from the appropriate paths and the exception is avoided?
In AIX the libraries are loaded from the appropriate paths and ProviderException is still thrown as this issue description. Adding -Djgskit.library.path doesn't solve the issue.
A couple of more questions, is there any other native library involved other than libjgsk8iccs_64.so and libjgskit.so?
No other libraries are involved by the way.
In this case, we do need to know how ockInstallPath was retrieved when JVM loaded both libraries from the expected location.
In my example, I wasn't specifying -Djgskit.library.path since I don't really care where libjgskit.so is loaded from and that didn't seem to be an issue in other platforms. But could those two be somehow tied on AIX?
No, they don't seem tied together. I tried only -Dock.library.path and got the same result, i.e., libjgsk8iccs_64.so was loaded at the location specified via the system property, but the same ProviderException was thrown. Note libjgskit.so is loaded from jdk-21.0.7+4/lib in this scenario.
The reason I added -Djgskit.library.path is to avoid the message jceplus: Libpath not found for jgskit, use java home directory..
There is a trace available from OCKC that i have used in the past. In those cases I tried touch GSKIT_CRYPT.log to create an empty file in the directory that you run your program in. Once the program runs debug traces from the OCKC library should show up in this file.
I had another look at this issue, and generated GSKIT_CRYPTO.log.
The ICC_INSTALL_PATH was /team/jasonf/jceplus/jdk-21.0.7+4/lib/C which is different from -Dock.library.path specified via command line.
240122:icclib.c :27525450:C:!GetValue ICC_INSTALL_PATH /team/jasonf/jceplus/jdk-21.0.7+4/lib/C
From the https://github.com/IBM/OpenCryptographyKitC/blob/main/icc/ICCencapsulator.java#L557-L562, it seems ICC_INSTALL_PATH is from ICC_Init which is
24:gsk_wrap2.c :27525450:S: !ICC_InitReal ICCC_Init() /team/jasonf/jceplus/jdk-21.0.7+4/lib/C
I am wondering if ockLoadPath is same as ockInstallPath when a different path is supplied via -Dock.library.path.
@KostasTsiounis Can we confirm with the author of OpenCryptographyKitC this is a valid assumption across platforms?
Additional note: I don't see this ProviderException is caused by VM unless further investigation from OpenCryptographyKitC suggests so.
I am wondering if
ockLoadPathis same asockInstallPathwhen a different path is supplied via-Dock.library.path. @KostasTsiounis Can we confirm with the author of OpenCryptographyKitC this is a valid assumption across platforms?
In all platforms except AIX, when the -Dock.library.path is supplied, the library that it points to is loaded and both ockLoadPath and ockInstallPath are set to that as well. On AIX, however, ockLoadPath is set to the library provided through the flag, but the ockInstallPath is set to the builtin one, even through we explicitly do a System.load(ockLoadPath).
So, the problem is that AIX seems to load the builtin one, even though we instruct otherwise.