vowpal_wabbit
vowpal_wabbit copied to clipboard
VowpalWabbit
JNI Build
I am trying to build the java JNI wrapper for the library, but when I try running make java i get the following error:
-- Build files have been written to: /Users/clementeocejo/Documents/java_vw/build
cd build; make -j4 vw_jni
[ 4%] Built target allreduce
[ 4%] Built target vw_io
[ 85%] Built target vw
[ 87%] Built target vw-bin
[ 88%] Linking CXX shared library libvw_jni.dylib
ld: unknown option: --enable-new-dtags
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[4]: *** [java/libvw_jni.dylib] Error 1
make[3]: *** [java/CMakeFiles/vw_jni.dir/all] Error 2
make[2]: *** [java/CMakeFiles/vw_jni.dir/rule] Error 2
make[1]: *** [vw_jni] Error 2
make: *** [java_build] Error 2
I have made sure that my JAVA_HOME is set appropriately and that Maven is installed. I also have all other dependencies. What could be causing this?
Thank you.
Looking further into it, it looks like some of the options used by the library link in the java wrapper CMakeLists.txt are not recognized in clang. Is the only option to use gcc as the compiler instead?
@eisber do you know if the Java bindings need GCC? It looks like the spark bindings added the --enable-new-dtags flag
the flag is used to make sure the dynamic library can be found (e.g. in the same directory).
the flag is used to make sure the dynamic library can be found (e.g. in the same directory).
does this mean I have to use gcc? I tried finding a way to port the command over with options usable by clang but it is more difficult than I thought and building boost with gcc is giving me trouble too.
Let's try a mitigation:
@ClementeOcejo can you try just removing the flag here: https://github.com/VowpalWabbit/vowpal_wabbit/blob/2a4317daa2806ee4830eb278215f001c86febdc7/java/CMakeLists.txt#L58 So the line should become:
target_link_libraries(vw_jni PUBLIC -fPIC -Wl,-rpath,\"\$ORIGIN\" vw)
Thoughts:
It seems that --enable-new-dtags is only supported on ELF targets, which MacOS is not. (https://stackoverflow.com/questions/22428356/unrecognized-option-enable-new-dtags-error-during-building-c-library)
The requirement of using this flag in the build in general needs further investigation. --enable-new-dtags causes DT_RUNPATH instead of DT_RPATH to be used.
DT_RUNPATH has lower precedence than LD_LIBRARY_PATH, but DT_RPATH has higher precendence. Therefore, a user can't override DT_RPATH if that's how the binary was distributed. DT_RPATH is deprecated and DT_RUNPATH is the recommended way to specify runtime dependencies.
https://stackoverflow.com/questions/52018092/how-to-set-rpath-and-runpath-with-gcc-ld https://stackoverflow.com/questions/7967848/use-rpath-but-not-runpath
Yeah I was moreso curious if this is a valid fix. Will this still build correctly? Thanks for the detailed response.
Given that on MacOS --enable-new-dtags isn't supported I think it is a valid fix.
--enable-new-dtags is used to ensure the binary uses the newer DT_RUNPATH, and the point of this is to allow users more freedom about specifying dependencies on their system over the hardcoded search path in the binary. But if the platform doesn't support it then we're out of luck.
We 100% need to update the build scripts to not fail on MacOS though!
Results :
Tests in error:
VowpalWabbitNativeIT.testAudit:211 NoClassDefFound Could not initialize class ...
VowpalWabbitNativeIT.testBFGS:161 NoClassDefFound Could not initialize class o...
VowpalWabbitNativeIT.testHashing:29 » NoClassDefFound Could not initialize cla...
VowpalWabbitNativeIT.testPrediction:102 NoClassDefFound Could not initialize c...
VowpalWabbitNativeIT.testWrappedVsCommandLine:53 » UnsatisfiedLink Can't load ...
Tests run: 5, Failures: 0, Errors: 5, Skipped: 0
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent! The file encoding for reports output files should be provided by the POM property ${project.reporting.outputEncoding}.
[INFO]
[INFO] --- maven-failsafe-plugin:2.19.1:verify (default) @ vw-jni ---
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent! The file encoding for reports output files should be provided by the POM property ${project.reporting.outputEncoding}.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 20.104 s
[INFO] Finished at: 2020-07-22T13:55:23-04:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:2.19.1:verify (default) on project vw-jni: There are test failures.
[ERROR]
[ERROR] Please refer to /Users/clementeocejo/Documents/javavw/java/target/failsafe-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
make[4]: *** [java/libvw_jni.dylib] Error 1
make[4]: *** Deleting file `java/libvw_jni.dylib'
make[3]: *** [java/CMakeFiles/vw_jni.dir/all] Error 2
make[2]: *** [java/CMakeFiles/vw_jni.dir/rule] Error 2
make[1]: *** [vw_jni] Error 2
make: *** [java_build] Error 2
Looks like from the logs it thinks I am on a linux machine or something along those line, not totally sure.
Tests run: 5, Failures: 0, Errors: 5, Skipped: 0, Time elapsed: 0.249 sec <<< FAILURE! - in org.vowpalwabbit.spark.VowpalWabbitNativeIT
testWrappedVsCommandLine(org.vowpalwabbit.spark.VowpalWabbitNativeIT) Time elapsed: 0.209 sec <<< ERROR!
java.lang.UnsatisfiedLinkError: Can't load library: /var/folders/yw/38vjslx90td1x5mwcc7j23g00000gn/T/tmplibvw14838778802605794497/natives/linux_64/libvw_jni.so
at org.vowpalwabbit.spark.VowpalWabbitNativeIT.testWrappedVsCommandLine(VowpalWabbitNativeIT.java:53)
testAudit(org.vowpalwabbit.spark.VowpalWabbitNativeIT) Time elapsed: 0 sec <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.vowpalwabbit.spark.VowpalWabbitNative
at
I'm getting these errors now which seem to have to do with Maven. Any thoughts? Thanks again.
Yes, it looks like there is an assumption around the target platform being Linux. It looks like it was introduced with the Spark bindings but is affecting use for the orignal bindings too. @eisber how can we resolve this so that it can be loaded on all platforms?
See this file: https://github.com/VowpalWabbit/vowpal_wabbit/blob/master/java/src/main/java/org/vowpalwabbit/spark/Native.java
Hi,
org.scijava.nativelib.NativeLoader this project does it almost right, but doesn't have the dependency extraction code. The main issue I didn't tackle it is that ideally one would build for all platforms (linux, mac, windows) and then package the corresponding binaries into a single jar.
If you want to just build your self it's probably easier. https://github.com/scijava/native-lib-loader/blob/master/src/main/java/org/scijava/nativelib/MxSysInfo.java and classes around might have a good guidance on how to detect the OS.
Markus
I am not really sure where to go from here. Thank you for the responses though.
It's possible. Quickest path is to build yourself and update the loading code. Integrating this into a public build is probably a bit more complicated. Either cross-compiled to OSX (not sure if that's possible) or setup multiple builds (Linux, Windows, Mac) and a final release one that collects all the artifacts and packages into a single jar.
Made the change suggested above about removing the dtags flag. Able to build a Jar along with an OSX dylib successfully. Noticed that the resultant jar had natives/linux_64/libvw_jni.dylib the lib placed inside natives/linux_64. Is this correct location?
One issue with the current root level Makefile command make java is that it runs cmake without the enable java flags. I manually ran the cmake step like this cmake ../ -DBUILD_JAVA=ON from the build folder
linux_64 sounds like the hack I was alluding too. ideally you'd drop it into natives/mac . https://stackoverflow.com/questions/228477/how-do-i-programmatically-determine-operating-system-in-java has a suggestion on how to detect the OS
Followed suggested steps and able to build a Jar on OSX successfully. Bunch of tests failed during the build though. While trying to load, Jar fails with an error no main manifest attribute, in vw-jni.jar
Does this mean that although the jar is built successfully, it is corrupted?