Runtime compilation fails when executing within a fat jar (e.g. jar produced by spring boot)

[k-abram]

Expected behavior and actual behavior:

Expected behavior: runtime class compiles. Actual behavior: compilation fails if compiled code references a class that is in a jar within a fat jar.

Steps to reproduce the problem:

Create a class called Foo.java. Create a jar with this. Create a class called Bar.java that compiles a class that derives from Foo. Create a super-jar with Bar.java and the jar containing Foo.java. Run bar.java using Spring boot created super-jar (fat jar).

Versions:

• jOOR: 0.9.9
• Java: 8

The core issue is that the classpath entry created by spring-boot class loader looks like this: //super-jar.jar!/lib/foo.jar

It doesn't look like the Tool compiler can handle such a classpath reference (alas, there even seems to be a request going back to 2002 asking to add support for such a thing).

[lukaseder]

Thank you very much for your report.

For the avoidance of doubt, would it be possible to provide a project setup that helps reproduce these jar files? Since you seem to have already gone through the trouble of creating an MCVE, providing that would definitely help speed up things.

Also, could you provide a link to that request you've found?

[tioricardo]

@k-abram it seems like javax.tools.JavaCompiler can't handle inner jars on classpath.

You may try to use spring-boot-maven-plugin with requiresUnpack (https://docs.spring.io/spring-boot/docs/current/reference/html/howto-build.html#howto-extract-specific-libraries-when-an-executable-jar-runs), and then set java.class.path property pointing to the extracted jar before calling org.joor.Reflect::compile.

[ushulau]

same issue here ... using spring boot

[k-abram]

This is a problem with the underlying tool being used - javax.tools.JavaCompiler and the requiresUnpack option doesn't work since the user of my framework will have to figure out all the classes that need to be unpacked, figure out the packaging and do so. Besides, unpack itself may not be an option.

I've reverted and gone back to using Javassist given that they are keeping up with java releases.

[lukaseder]

I've reverted and gone back to using Javassist

Yeah, sorry - jOOR really scratches a very minor itch. Surely, tools like javassist are much more thoroughly solving these sets of problems.

given that they are keeping up with java releases.

I understand that this isn't strictly related to java releases, though

[k-abram]

given that they are keeping up with java releases.

Sorry, I must clarify what I meant by that - my purpose in looking at jOOR as a replacement for Javassist was due to uncertainty regarding javassist support for future java releases. I didn't mean the remark against jOOR. The JOOR technique provides a superior syntax support (javassist requires use of jdk 4 syntax - no generics, nothing - not even import statements). But my perception of javassist support for future releases was incorrect and so I was able to switch back.

As far as the issue with JavaCompiler goes, yeah, I don't think any of us can do anything about it. Where applicable, jOOR runtime compilation works, and works very well.

[lukaseder]

I didn't mean the remark against jOOR

Oh, no offense taken :) I was just curious what could be improved here in addition to this particular problem.

javassist requires use of jdk 4 syntax - no generics, nothing - not even import statements

Oh, true. I kinda remember that from the last time I used it. It was really only of very limited use.

But my perception of javassist support for future releases was incorrect and so I was able to switch back.

I see. By the way, bytebuddy is also very good. Rafael Winterhalter is maintaining it very actively. I'm not sure how far you can get from using Java source code, but maybe in your case that's an option. I'm actually often looking at its sources for inspiration in this area.

As far as the issue with JavaCompiler goes, yeah, I don't think any of us can do anything about it. Where applicable, jOOR runtime compilation works, and works very well.

There might be a hack to be discovered... :-)

[Delwing]

For anyone getting here while searching for solution. It's true that javax.tools.JavaCompiler will have difficulties with reading inner jar (at least I didn't find any workaround to make it read those directly).

The workaround I've found was to create shadow jar out of my project (can be as addition to regular jar).

List<String> optionList = List.of("-classpath", "/path/to/shadow.jar");
Supplier<String> mapper = Reflect.compile("className", "code", new CompileOptions().options(optionList)).create().get();

Might work for you like it sure did a job for me.

[davsclaus]

Maybe for spring boot as fat jar, then jOOQ Compile.java could have an option that would then unpack the spring-boot fat-jar into some temporary folder location, and then build a regular classpath from that the javax compiler can use, and then after that it can cleanup the temporary folder.

Just a heads up, that I reproduced this issue with an Apache Camel Spring Boot Example where I am using our new camel-joor language. It works when you run spring boot via its mvn spring-boot:run but then its not a fat jar.

[lukaseder]

Maybe for spring boot as fat jar, then jOOQ Compile.java could have an option that would then unpack the spring-boot fat-jar into some temporary folder location, and then build a regular classpath from that the javax compiler can use, and then after that it can cleanup the temporary folder.

That sounds like an awful lot of knowledge for jOOR to have about other things it shouldn't know about. If this is fixable at all, I really prefer a less laborious fix (in jOOR).

[davsclaus]

Maybe for spring boot as fat jar, then jOOQ Compile.java could have an option that would then unpack the spring-boot fat-jar into some temporary folder location, and then build a regular classpath from that the javax compiler can use, and then after that it can cleanup the temporary folder.

That sounds like an awful lot of knowledge for jOOR to have about other things it shouldn't know about. If this is fixable at all, I really prefer a less laborious fix (in jOOR).

Yeah I can see that point too. I wonder if there could be a plugin interface (spi) for 3rd party to plugin which gives some hook/control of this compilation process, so you can do some custom logic before | after compilation, and also to compute the classpath and whatnot.

Then we at Apache Camel could implement a custom spi in our camel-joor-starter module for spring boot. And if Spring Boot itself pickup jOOR then they can implement a custom plugin and provide as part of Spring Boot.

Another runtime that is gaining popularity is Quarkus. I have not checked whether its a similar issue if you use fat jar with Quarkus.

[lukaseder]

Yeah I can see that point too. I wonder if there could be a plugin interface (spi) for 3rd party to plugin which gives some hook/control of this compilation process, so you can do some custom logic before | after compilation

You're aware that we're talking about roughly 30 lines of code? 😁 Let's not overengineer these 30 lines of code.

and also to compute the classpath and whatnot.

You can already pass that via CompileOptions.options

Another runtime that is gaining popularity is Quarkus. I have not checked whether its a similar issue if you use fat jar with Quarkus.

Thinking about it, maybe, this is just a bug in JavaCompiler? Why do clients of JavaCompiler (including jOOR) have to worry about these things?