datafusion-comet
datafusion-comet copied to clipboard
chore: Remove NativeBase static initializer (to improve error handling when native lib fails to load)
Which issue does this PR close?
Closes https://github.com/apache/datafusion-comet/issues/999
Rationale for this change
If the static initialization block failed to load the library then we did not see the reason why (in some cases).
Before:
│ 24/10/07 22:42:11 WARN CometSparkSessionExtensions: Comet extension is disabled because of error when loading native lib. Falling back to Spark │
│ java.lang.NoClassDefFoundError: Could not initialize class org.apache.comet.NativeBase
After:
│ 24/10/07 22:54:09 WARN CometSparkSessionExtensions: Comet extension is disabled because of error when loading native lib. Falling back to Spark │
│ java.lang.UnsatisfiedLinkError: /tmp/libcomet-14517097013463249870.so: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.35' not found (required by /tmp/libcomet-14517097013463249870.so) │
│ at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method) │
│ at java.base/java.lang.ClassLoader$NativeLibrary.load(Unknown Source) │
│ at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(Unknown Source) │
│ at java.base/java.lang.ClassLoader.loadLibrary0(Unknown Source) │
│ at java.base/java.lang.ClassLoader.loadLibrary(Unknown Source) │
│ at java.base/java.lang.Runtime.load0(Unknown Source) │
│ at java.base/java.lang.System.load(Unknown Source) │
│ at org.apache.comet.NativeBase.bundleLoadLibrary(NativeBase.java:110) │
│ at org.apache.comet.NativeBase.load(NativeBase.java:78) │
│ at org.apache.comet.CometSparkSessionExtensions$.isCometEnabled(CometSparkSessionExtensions.scala:1108)
What changes are included in this PR?
Improve error handling by removing the static init code, which was triggered by the plugin calling isLoaded. The plugin now just calls load directly.
How are these changes tested?
Manually.
@parthchandra could you review?
@parthchandra @viirya I reimplemented this with a much simpler approach. PTAL when you can.