datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard

chore: Remove NativeBase static initializer (to improve error handling when native lib fails to load)

Open andygrove opened this issue 1 year ago • 1 comments

Which issue does this PR close?

Closes https://github.com/apache/datafusion-comet/issues/999

Rationale for this change

If the static initialization block failed to load the library then we did not see the reason why (in some cases).

Before:

│ 24/10/07 22:42:11 WARN CometSparkSessionExtensions: Comet extension is disabled because of error when loading native lib. Falling back to Spark                                                                                                   │
│ java.lang.NoClassDefFoundError: Could not initialize class org.apache.comet.NativeBase  

After:

│ 24/10/07 22:54:09 WARN CometSparkSessionExtensions: Comet extension is disabled because of error when loading native lib. Falling back to Spark                                                                                                   │
│ java.lang.UnsatisfiedLinkError: /tmp/libcomet-14517097013463249870.so: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.35' not found (required by /tmp/libcomet-14517097013463249870.so)                                                        │
│     at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)                                                                                                                                                                         │
│     at java.base/java.lang.ClassLoader$NativeLibrary.load(Unknown Source)                                                                                                                                                                         │
│     at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(Unknown Source)                                                                                                                                                                  │
│     at java.base/java.lang.ClassLoader.loadLibrary0(Unknown Source)                                                                                                                                                                               │
│     at java.base/java.lang.ClassLoader.loadLibrary(Unknown Source)                                                                                                                                                                                │
│     at java.base/java.lang.Runtime.load0(Unknown Source)                                                                                                                                                                                          │
│     at java.base/java.lang.System.load(Unknown Source)                                                                                                                                                                                            │
│     at org.apache.comet.NativeBase.bundleLoadLibrary(NativeBase.java:110)                                                                                                                                                                         │
│     at org.apache.comet.NativeBase.load(NativeBase.java:78)                                                                                                                                                                                       │
│     at org.apache.comet.CometSparkSessionExtensions$.isCometEnabled(CometSparkSessionExtensions.scala:1108)       

What changes are included in this PR?

Improve error handling by removing the static init code, which was triggered by the plugin calling isLoaded. The plugin now just calls load directly.

How are these changes tested?

Manually.

andygrove avatar Oct 07 '24 22:10 andygrove

@parthchandra could you review?

andygrove avatar Oct 07 '24 22:10 andygrove

@parthchandra @viirya I reimplemented this with a much simpler approach. PTAL when you can.

andygrove avatar Oct 22 '24 15:10 andygrove