sqlite-jdbc
sqlite-jdbc copied to clipboard
xerial
Support the Foreign API from project Panama
I've been playing with the Foreign API - the replacement for JNI that's been incubating since Java 14. I have managed to hack a version of the sqlite-jdbc code to use the Foreign API directly against the sqlite3 library without the need for any extra C code. The code I have is only capable of returning Strings, ints and doubles form a query.
The test I ran was
- a table with just under 7000 rows and 27 columns.
- query "select * from ...".
- Run 1000 times, throw away timing from first 100
In my test, the Foreign API is about 2x faster than the existing JNI implementation. (mean of 44ms vs 23ms to retrieve all rows and columns).
My experience with the Foreign API is that when passing primitive values it's noticeably faster than JNI. When passing arrays of data the Foreign API is faster than JNI, but the gap is smaller. In other testing I've done, callback methods that call from C into Java have a similar or better speed increase (about 3.75x).
The Foreign API is only in incubator still, so it's subject to change. To support this API would require a multi-release jar. Using a tool like JExtract on sqlite3.h would do a large chunk of the translation required.
That sounds very promising. I've heard about Panama but never gave it a try.
However I don't see Panama replacing JNI in this repo. The API is still incubating, and will not land as stable before Java 19 or 20 probably. The current version supports Java 8.
Most of the recent work done in this repo is around the compilation of native libs (sqlite + JNI code) to multiple platforms, or some fixes in the JNI code or around it.
I don't think the 2 can coexist in the same repo easily.
I think however that a Panama fork would be most interesting. It would target higher JDKs, and replace JNI with Panama. Without the need to have a custom JNI code, you would only require the native SQlite library for your platform, which in itself could be complicated, given the variety of options to compile it, and making sure you have the right version. I'm thinking there could be 2 modules: the main one that would use the sqlite found on your system (or which you point at), and another module that would do what we do here, which is compiling the sqlite lib for multiple OSes and architectures, and make it available as resources inside the jar. That way if you want a similar behaviour as the current version, you can have it. But if you are happy to have sqlite as a dependency, using a package manager or something, you can too.
Similar has been done recently for tomcat-native library. There the perf is on par with JNI code.
I like the idea of a separate fork to target more recent JDK's. Getting the existing repo to build using Java 17 and use the foreign API was not a simple drop in. The code has to be modularized to compile and, you need to provide the VM --enable-native-access [module names] in order to get anything to run.
I just started looking at the Java 18 early-access release and it looks like I already have to rewrite quite a bit of my code. I'm curious if something slowed down in 18 such that tomcat native didn't see much improvement, or if the jextract generated code needed optimizing.
I finally got around to making a fork with Panama. Not all of the unit tests are passing, but I think most of the failures are pretty benign. SQLite with Panama. The changes were all pretty minimal. There's still quite a bit of polish required, but it works.
I've only got the binaries compiled for Windows 64 bit and Linux 64 bit. Every other platform falls back to JNI.
For me the point of Panama is to not ship the lib with the jar, and use what's on the system already. That becomes easier because you don't need the small JNI part to be compiled, and you can rely on package managers to install the libs needed.
That's what I'm doing on my imageio project, NightMonkeys.
For me the point of Panama is to not ship the lib with the jar, and use what's on the system already. That becomes easier because you don't need the small JNI part to be compiled, and you can rely on package managers to install the libs needed.
That's what I'm doing on my imageio project, NightMonkeys.
But what about macros and runtime checks?
But what about macros and runtime checks?
Not sure to understand what you mean, can you provide same details / example ?
But what about macros and runtime checks?
Not sure to understand what you mean, can you provide same details / example ?
How are macros evaluated if you run jextract once, but all platforms are different.
But what about macros and runtime checks?
Not sure to understand what you mean, can you provide same details / example ?
How are macros evaluated if you run jextract once, but all platforms are different.
Still don't understand, sorry.
But what about macros and runtime checks?
Not sure to understand what you mean, can you provide same details / example ?
How are macros evaluated if you run jextract once, but all platforms are different.
Still don't understand, sorry.
If the target header file contains: #ifdef __platform ...#endif and you run `jextract one platform, but not on the target one, how is this supported to properly detect everything?
But what about macros and runtime checks?
Not sure to understand what you mean, can you provide same details / example ?
How are macros evaluated if you run jextract once, but all platforms are different.
Still don't understand, sorry.
If the target header file contains:
#ifdef __platform ...#endifand you run `jextract one platform, but not on the target one, how is this supported to properly detect everything?
Without concrete examples i don't really see what could be the impact. jextract is also optional.
But what about macros and runtime checks?
Not sure to understand what you mean, can you provide same details / example ?
How are macros evaluated if you run jextract once, but all platforms are different.
Still don't understand, sorry.
If the target header file contains:
#ifdef __platform ...#endifand you run `jextract one platform, but not on the target one, how is this supported to properly detect everything?Without concrete examples i don't really see what could be the impact.
jextractis also optional.
I'll try to provide an example the upcoming weeks. Maybe this is just an inconception from my side.