Add support for Kafka Streams

[pdebuitlear]

I am trying to get a Spring Kafka application using Kafka Streams working. I'm getting a runtime exception when the code is trying to create a state store.

Caused by: java.lang.RuntimeException: librocksdbjni-linux64.so was not found inside JAR.
	at org.rocksdb.NativeLibraryLoader.loadLibraryFromJarToTemp(NativeLibraryLoader.java:110)
	at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)
	at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)
	at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:64)
	at org.rocksdb.RocksDB.<clinit>(RocksDB.java:35)
	at com.oracle.svm.core.classinitialization.ClassInitializationInfo.invokeClassInitializer(ClassInitializationInfo.java:375)

Sample code here. Here are the hints I've added to get things working to this point.

From looking at the quarkus implementation, it looks like they are registering the default key/value serdes, as well as the configured exception handler also. It would be good if the support here also did similar.

[sdeleuze]

I am discussing that one with the GraalVM team.

[sdeleuze]

Based on GraalVM team feedback, you can try to extract the native library from the jar and put it in the same folder as native-image, and if that doesn't work, add the current directory to the java.library.path. Please let me know how it goes.

[pdebuitlear]

Ok, I've just been using the 'bootBuildImage' buildpack build up to now. I'm guessing I'll not be able to do the above using that approach?

[pdebuitlear]

I've extracted the librocksdbjni-linux64.so lib from the org.rocksdb jar and placed it in resources/META-INF/native-image. I've done a quick test and I'm not reproducing the above issue. I'm not sure if this is a good solution though. From looking at the quarkus code base, it looks like this type of issue will crop up in a few different areas for this project.

[sdeleuze]

We should probably add dedicated support to do that automatically I agree.

[pdebuitlear]

I've extracted the librocksdbjni-linux64.so lib from the org.rocksdb jar and placed it in resources/META-INF/native-image. I've done a quick test and I'm not reproducing the above issue. I'm not sure if this is a good solution though. From looking at the quarkus code base, it looks like this type of issue will crop up in a few different areas for this project.

Sorry, I spoke too soon. I took another look at this, just extracting the .so didn't work

[sobychacko]

@sdeleuze We added the native support only for the regular Kafka binder, but not for Kafka Streams binder. There is this issue. I've seen this error from Kafka Streams binder while looking at it initially. I guess these are all related, do you know the status of the error on that rocksdb/jni issue?

[sdeleuze]

@sobychacko Not 100% sure, but you can try to put librocksdbjni-linux64.so in resources/META-INF/native-image manually for now and see how it goes.

[sshemirani]

Hello all, Did anyone manage to solve this issue?

[pdebuitlear]

I didn't investigate it further anyway, sorry

[sdeleuze]

This issue is now tracked by https://github.com/spring-cloud/spring-cloud-stream/issues/2323 and should target resolution leveraging Spring Boot 3.