delay starting executors for graal compilation

[josh-7t]

Graal does not like it when Thread objects are initialized at build time (statically) e.g. error:

Error: Detected a started Thread in the image heap. Threads running in the image generator are no longer running at image runtime.  To see how this object got instantiated use --trace-object-instantiation=java.lang.Thread. The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image runtime by using the option --initialize-at-run-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Trace: Object was reached by
        reading field java.util.concurrent.SynchronousQueue$TransferStack$SNode.waiter of
                constant java.util.concurrent.SynchronousQueue$TransferStack$SNode@20bf1f43 reached by
        reading field java.util.concurrent.SynchronousQueue$TransferStack.head of
                constant java.util.concurrent.SynchronousQueue$TransferStack@36676ee4 reached by
        reading field java.util.concurrent.SynchronousQueue.transferer of
                constant java.util.concurrent.SynchronousQueue@74f498f2 reached by
        reading field io.aleph.dirigiste.Executor._queue of
                constant io.aleph.dirigiste.Executor@2aa705b1 reached by
        reading field clojure.lang.Var.root of
                constant clojure.lang.Var@1375a85a reached by
        indexing into array
                constant java.lang.Object[]@6084b3c0 reached by
        reading field clojure.lang.PersistentHashMap$BitmapIndexedNode.array of
                constant clojure.lang.PersistentHashMap$BitmapIndexedNode@6423ded3 reached by
        indexing into array
                constant clojure.lang.PersistentHashMap$INode[]@60c39780 reached by
        reading field clojure.lang.PersistentHashMap$ArrayNode.array of
                constant clojure.lang.PersistentHashMap$ArrayNode@2f013229 reached by
        reading field clojure.lang.PersistentHashMap.root of
                constant clojure.lang.PersistentHashMap@60600939 reached by
        reading field java.util.concurrent.atomic.AtomicReference.value of
                constant java.util.concurrent.atomic.AtomicReference@72e17dd8 reached by
        reading field clojure.lang.Namespace.mappings of
                constant clojure.lang.Namespace@56703b62 reached by
        indexing into array
                constant java.lang.Object[]@757237be reached by
        reading field clojure.lang.PersistentHashMap$BitmapIndexedNode.array of
                constant clojure.lang.PersistentHashMap$BitmapIndexedNode@6536badd reached by
        reading field clojure.lang.PersistentHashMap.root of
                constant clojure.lang.PersistentHashMap@7fe0398d reached by
        reading field java.util.concurrent.atomic.AtomicReference.value of
                constant java.util.concurrent.atomic.AtomicReference@c5658fe reached by
        reading field clojure.lang.Namespace.aliases of
                constant clojure.lang.Namespace@1d526bdd reached by
        reading field clojure.lang.Var.ns of
                constant clojure.lang.Var@53f5d68d reached by
        scanning method via.adapters.aleph$websocket_adapter$reify__15706.disconnect(aleph.clj:32)
Call path from entry point to via.adapters.aleph$websocket_adapter$reify__15706.disconnect(Object):
        at via.adapters.aleph$websocket_adapter$reify__15706.disconnect(aleph.clj:32)
        at via.adapter$fn__667$G__609__670.invoke(adapter.cljc:15)
        at clojure.core.proxy$clojure.lang.APersistentMap$ff19274a.applyTo(Unknown Source)
        at graal_test.main.main(Unknown Source)
        at com.oracle.svm.core.JavaMainWrapper.runCore(JavaMainWrapper.java:146)
        at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:182)
        at com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)

Wrapping the statically defined connection-pools and executors with delay ensures that the threads are not started until runtime.

Also upgrading netty resolves this object initialization error when compiling with graal:

Fatal error:com.oracle.svm.core.util.VMError$HostedError: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Detected an instance of Random/SplittableRandom class in the image heap. Instances created during image generation have cached seed values and don't behave as expected.  To see how this object got instantiated use --trace-object-instantiation=java.util.Random. The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image runtime by using the option --initialize-at-run-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.

And finally when https://github.com/clj-commons/byte-streams/pull/50 gets merged I would like to include up-versioning byte-streams in this pr.

[josh-7t]

Any questions? concerns?

[KingMob]

Hi Joshua, you might get faster responses if you add me as a reviewer. Erik (slipset) is the overall clj-commons maintainer, but doesn't really review aleph/manifold/etc code.

I bumped up the byte-streams version, so you don't need to do that in this PR.

As for the rest, I need to take a deeper look. Unfortunately, I don't know much about Graal. However, I can say this is a breaking change for anyone who currently uses the pool and executor outside of Aleph.

When it says:

Error: Detected a started Thread in the image heap. Threads running in the image generator are no longer running at image runtime. To see how this object got instantiated use --trace-object-instantiation=java.lang.Thread. The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image runtime by using the option --initialize-at-run-time=. Or you can write your own initialization methods and call them explicitly from your main entry point."

does that mean you could bypass the issue by using --initialize-at-run-time=java.lang.Thread or something similar?

[arnaudgeiser]

I just did a simple HTTP server using Aleph and Graal native-image without any issues (expect the usual ones - plus the Netty ones).

Here is the dummy project: https://github.com/arnaudgeiser/aleph-graal

And the native-image flags I'm using:

:opts ["--no-fallback"
       "--report-unsupported-elements-at-runtime"
       "--initialize-at-build-time"
       "--allow-incomplete-classpath"
       "--initialize-at-run-time=io.netty.channel.DefaultFileRegion"
       "--initialize-at-run-time=io.netty.channel.epoll.Epoll"
       "--initialize-at-run-time=io.netty.channel.epoll.Native"
       "--initialize-at-run-time=io.netty.channel.epoll.EpollEventLoop"
       "--initialize-at-run-time=io.netty.channel.epoll.EpollEventArray"
       "--initialize-at-run-time=io.netty.channel.unix.Errors"]}

[arnaudgeiser]

As it's working on me side, the PR is one year old and we have no news from the requester, I will close.

Do not hesitate to re-open another one @josh-7t if needed.