snappy-java
snappy-java copied to clipboard
xerial
Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=zOS and os.arch=s390x
Hi Team, I am using version 1.1.7.3 of 'snappy-java' and getting below error:
Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=zOS and os.arch=s390x
JVM version:: IBM J9 VM (build 2.9, JRE 1.8.0 z/OS s390x-64-Bit Compressed References 20190124_408237 (JIT enabled, AOT enabled)
snappy-java currently does not have a native binary for zOS/s390x. I contributed the binary for Linux/s390x, but I do not have access to zOS now. Let me poke around....
@odaira Did you find anything or any ETA, we have install around next month that requires this change
@sunz21sunz Would you have any contact in IBM (or other company supporting zOS) on your deal? If the deadline is critical, I would reach out to the contact if any. I'm working for snappy-java on a voluntary basis, so I'll try, but I cannot commit myself to any schedule.
By the way, I've got access to a zOS instance. It has nothing installed, so I'll start with installing git.
Another idea is adding a fallback to pure-java Snappy (like aircompressor) if no native library is found.
@xerial do you have any benchmarks for a pure-java implementation vs this one?
@odaira you can get access to a free LinuxOne (Linux on Z) environment in two ways.
- The Marist College Linux One Community CLoud https://linuxone.cloud.marist.edu/cloud/#/login
- The HyperProtect Virtual Server (Free tier) on the IBM public Cloud https://cloud.ibm.com
Good luck
Added a fallback to pure-java snappy implementation in #244 and this is available since snappy-java 1.1.76
@xerial , the fallback that you implemented in #244 that is based on aircompressor doesn't support BIGENDIAN platforms
static {
ByteOrder order = ByteOrder.nativeOrder();
if (!order.equals(ByteOrder.LITTLE_ENDIAN)) {
throw new SnappyError(SnappyErrorCode.UNSUPPORTED_PLATFORM, format("pure-java snappy requires a little endian platform (found %s)", order));
}
I came here to report the bug, but @n-marion beat me :) If run on Z/OS Unix shell with snappy-java 1.1.7.6, and use pure-java option on command line :
Exception in thread "main" org.xerial.snappy.SnappyError: [UNSUPPORTED_PLATFORM] pure-java snappy requires a little endian platform (found BIG_ENDIAN)
at org.xerial.snappy.pure.UnsafeUtil.
I really want to use snappy compression within Avro, but this LITTLE_ENDIAN bug in aircompressor is preventing us. [(https://github.com/airlift/aircompressor/blob/master/src/main/java/io/airlift/compress/snappy/UnsafeUtil.java)]
Specifically, it seems that the lines identified by @xerial for LITTLE_ENDIAN should be deleted.
It's not a bug, rather big-endian architecture is not supported in the pure-java fallback version. To use zOS, s390x we need to add a native library for this.
If https://github.com/dockcross/dockcross supports cross-compilation for zOS/s390s, it's the ideal way to support zOS in snappy java because the build will be reproducible, and we don't need to ask IBM team members like @odaira to rebuild native libraries.
I only referred to it as a bug in pure-java, as it seems the only reason why big endian isn't supported is because there was no effort to reverse the byte order to allow it to work. ie It doesn't seem to be something that is completely unresolvable
This link has examples of methods that could be leveraged to provide the big endian functionality []http://cr.openjdk.java.net/~jmelvin/7130404/jdk/webrev.04/raw_files/new/src/share/classes/sun/security/provider/ByteArrayAccess.java
Personally I don't have time for working on big-endian support by checking every detail of the compressor/decompressor implementation (e.g., where byte order matters).
If there is a docker image of g++ cross compiler for zOS, I'd be happy to included it as a part of the build process of snappy-java. Otherwise, I need to wait for contributions from open source community.
This link seems to mention a docker image for cross compile of gcc to Z/OS platform (at bottom of blog). [https://blog.the-leviathan.ch/?p=1344] My company blocks the link, so I am unable to validate.
In the meantime, I might play around with Pure Java and try to get a local version that for Big Endian, does the reversebytes alternative that seems to be needed around every UNSAFE.getXXX() or UNSAFE.putxxx invocation
You should be able to test this on Linux on Z... which you can get a VM on the Marist College LInuxOne Community Cloud here: https://developer.ibm.com/components/ibm-linuxone/gettingstarted/
On Tue, Aug 25, 2020 at 12:26 PM kiwi1969 [email protected] wrote:
This link seems to mention a docker image for cross compile of gcc to Z/OS platform (at bottom of blog). [https://blog.the-leviathan.ch/?p=1344] My company blocks the link, so I am unable to validate.
In the meantime, I might play around with Pure Java and try to get a local version that for Big Endian, does the reversebytes alternative that seems to be needed around every UNSAFE.getXXX() or UNSAFE.putxxx invocation
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/xerial/snappy-java/issues/232#issuecomment-680130875, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHJIVXR2AVESFQLKY55OEDSCPQZTANCNFSM4I7CYR3A .
-- I/T Architect
@fosslou - I am using Z/OS native, not ZLinux. So far I have a version of libsnappy.so natively compiled on z/os using c89, without any GNU tools, based upon snappy.c implementation here: [https://github.com/yitzikc/snappy-c]. I am yet to add the JNI stubs to allow JNI to work. I am not sure if this actually works properly yet, or if it is 100% compatible with dll that snappy-java uses, so I am thinking I may be better off (and perhaps easier) to attempting an update to pure-java for big endian.
libsnappy.so won't work as snappy-java needs its own extension (libsnappyjava.so). Running make native is the basic step to create snappy-java that should work at your target machine.
https://github.com/xerial/snappy-java/blob/master/Makefile#L143
I have successfully updated snappy.purejava to work on both little and Big Endian, by providing new methods, instead of directly using UNSAFE.getXXX methods for short, int,long datatypes. ie Initial testing looks good.
I would like to note, that the 'fallback' doesn't actually occur when JNI DLL isn't being found. I can only get the pure java implementation to run by forcing it on the command line See below
java -cp "SnappyTest.jar:lib/*" com.equifax.usis.acro.Main
Exception in thread "main" org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=zOS and os.arch=s390x
at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:361)
at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:195)
at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:167)
at org.xerial.snappy.Snappy.init(Snappy.java:69)
at org.xerial.snappy.Snappy.<clinit>(Snappy.java:46)
at com.equifax.usis.acro.Main.main(Main.java:12)
java -Dorg.xerial.snappy.purejava=true -cp "SnappyTest.jar:lib/*" com.equifax.usis.acro.Main
Hello snappy-java! Snappy-java is a JNI-based wrapper of Snappy, a fast compresser/decompresser.
Original size = 96
Compressed size = 87
Uncompressed size = 96
My work won't let me attach to your Git repository from this PC, so I will have just pasted the 3 changed modules and my Test driver mainline here below snappytest.zip If someone could incorporate these into a later release, that would solve my issues.
I managed to create own fork of this project, apply my changes, and submit a pull request for this project to merge in changes from my fork : https://github.com/xerial/snappy-java/pull/254
Hoping this will solve this issue for systems without DLL for native Java (and usually do not have GNU toolsets etc in order to produce it themselves). They will just have to specify "-Dorg.xerial.snappy.purejava=true" to select pure Java.