Missing .class files?

[sogaiu]

Since https://github.com/taylorwood/clj.native-image/commit/b3823a48be75122b9671c86ce5353a85589ef15f I'm getting errors like:

Error: Unsupported features in 12 methods
Detailed message:
Error: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved type during parsing: clojure.tools.reader.reader_types.IPushbackReader. To diagnose the issue you can use the --allow-incomplete-classpath option. The missing type is then reported at run time when it is accessed the first time.
Trace: 
        at parsing rewrite_clj.reader$unread.invokeStatic(reader.clj:115)
Call path from entry point to rewrite_clj.reader$unread.invokeStatic(Object, Object): 
        at rewrite_clj.reader$unread.invokeStatic(reader.clj:112)
        at rewrite_clj.reader$unread.invoke(reader.clj:112)
        at clojure.tools.reader.default_data_readers.proxy$java.lang.ThreadLocal$ff19274a.equals(Unknown Source)
        at java.util.HashMap.getNode(HashMap.java:579)
        at java.util.HashMap.get(HashMap.java:557)
        at com.oracle.svm.jni.access.JNIReflectionDictionary.getFieldNameByID(JNIReflectionDictionary.java:278)
        at com.oracle.svm.jni.functions.JNIFunctions.ToReflectedField(JNIFunctions.java:836)
        at com.oracle.svm.core.code.IsolateEnterStub.JNIFunctions_ToReflectedField_80d8233579d5215df0227b770e5c01228a0de9b9(generated:0)

When I try to build the same project by first building an uberjar and handing that to native-image, I don't get those errors.

Upon close inspection of the classes directory that clj.native-image creates for compilation of class files, I noticed that the set of .class files contained differs from what's contained in the uberjar.

One difference is that while the uberjar contains class files for tools.reader.reader_types, the clj.native-image's classes directory hierarchy does not (though it does have other tools.reader-related class files).

When I do (compile 'script) from the project (without the clj.native-image alias enabled), I do get those class files.

On a possibly related note, it appears that the use of clj.native-image (since the aforementioned commit), affects the version of tools.reader on the classpath, which happens to be something used by the project in question.

(I wonder if using something like mranderson to "vendor" clj.native-image's dependencies might be desirable to not affect the classpath.)

[taylorwood]

Thanks for the detailed notes! Do you have an example project I can use to reproduce this?

[sogaiu]

I have one, but I don't know if it will successfully build for other folks (might be a broken dependency). If you don't mind trying, the missing-file-branch of https://github.com/sogaiu/adorn might be able to reproduce.

https://github.com/sogaiu/adorn/tree/missing-class-files

FWIW, I tried with a test account and reproduced, so may be it should work for other folks.

[sogaiu]

Experiments suggest that depending on what has been loaded already in a process, the exact class files that will get written to disk for (compile 'script) may differ.

It turns out that adorn and clj.native-image share at least one transitive dependency: tools.reader.

Invoking clj -A:native-image for adorn leads to (at least) parts of tools.reader being loaded (and thus compiled, but cached in memory, not written to disk). At this point, an invocation of (compile 'script) may not load (some?) pieces shared by clj.native-image and adorn, and thus they are not compiled again, so they are not written to disk in class file form either.

This can be observed by using code like (thanks @noisesmith):

(in-ns 'clojure.core)
(binding [*loading-verbosely* true] (compile 'script))

This should produce output as to which namespaces are loaded during compilation.

I compared the results of running the above code right after a process has started to run, to after having invoked (require 'clj.native-image) or (require 'clojure.tools.namespace.find), and noticed differences. Specifically, the two cases of requiring something explicitly before the compile didn't show that reader-types was being loaded, and observing the classes directory confirmed that no corresponding class files were produced. (I was careful to appropriately remove and recreate the classes directory during the tests).

I think it's possible that prior to https://github.com/taylorwood/clj.native-image/commit/b3823a48be75122b9671c86ce5353a85589ef15f, loading clojure.tools.namespace.find didn't end up pulling in (some?) parts of tools.reader that are used by adorn. If that's true, it might explain why (compile 'script) used to produce class files for (some of?) the reader-types portion of tools.reader (which don't get produced in commits including and beyond https://github.com/taylorwood/clj.native-image/commit/b3823a48be75122b9671c86ce5353a85589ef15f).

It appears that explicitly compiling clojure.tools.reader.reader-types does produce class files whether clojure.tools.namespace.find has been loaded or not.

FWIW, @borkdude mentioned that explicitly compiling each namespace in a project seemed to resolve this type of issue for him.

IIUC, there is at least one potential issue with only following that approach -- the overlap of dependencies between clj.native-image and one's project. This may affect what specific versions are actually compiled.

Some method to preserve a project's specific dependencies might be nicer. Possibly using something like mranderson to inline (vendor?) clj.native-image's dependencies might be one approach.

[sogaiu]

Regarding the issue of clj.native-image's dependencies possibly influencing a project's dependencies...

Would launching an external clj / clojure process from clj.native-image to execute (compile 'script) (and thus generate class files) be worth considering?

[sogaiu]

FYI, found this comment in Compile.java:

// Compiles libs and generates class files stored within the directory
// named by the Java System property "clojure.compile.path". Arguments are
// strings naming the libs to be compiled. The libs and compile-path must
// all be within CLASSPATH.

https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compile.java#L18_L21