I got JWM working with GraalVM to produce native binaries =)

[GavinRay97]

Tonsky, you held up your end of the deal -- now I get to hold up mine 😃

Video of fresh run, showing starting from nothing, compiling the JWM app, and then launching it (0:55 seconds in, compilation takes a while lol):

https://user-images.githubusercontent.com/26604994/134990377-b8da88f7-40bf-46ec-9007-2081294e09ae.mp4

Reproduction

Here's a zipped copy of the whole project, including the built files that contain the required GraalVM configuration:

• jwm-test.zip

This was done with GraalVM 21.3.0-dev, JDK 17:

λ java --version
openjdk 17 2021-09-14
OpenJDK Runtime Environment GraalVM CE 21.3.0-dev (build 17+35-jvmci-21.3-b02)
OpenJDK 64-Bit Server VM GraalVM CE 21.3.0-dev (build 17+35-jvmci-21.3-b02, mixed mode, sharing)

What I did was set up a basic Gradle Java app, using the GettingStarted.java example code:

• https://github.com/HumbleUI/JWM/blob/main/docs/GettingStarted.java

plugins {
    id 'application'
    id 'org.graalvm.buildtools.native' version '0.9.5'
}

repositories {
    mavenCentral()
    // GraalVM Native plugin, not on Maven yet
    // See: https://graalvm.github.io/native-build-tools/0.9.5/gradle-plugin.html
    gradlePluginPortal()
    // JWM local jar file, at "app/lib/jwm-main.jar"
    flatDir {
        dirs 'lib'
    }
}

dependencies {
    implementation name: 'jwm-main'
}

application {
    mainClass = 'jwm.test.Application'
}

graalvmNative {
    binaries {
      main {
          verbose = true
      }
    }
}

Trying to build the JWM app and run it normally will throw a segfault.

To fix this, you have to first run the "agent" to profile the GraalVM app and determine what configuration it needs for things like JNI, dynamic behavior, etc:

• https://graalvm.github.io/native-build-tools/0.9.5/gradle-plugin.html#_reflection_support_and_running_with_the_native_agent

./gradlew -Pagent run # Runs on JVM with native-image-agent.
./gradlew -Pagent nativeCompile # Builds image using configuration acquired by agent.

This is broken though, at least for me. It was generating the configuration in: app\build\native\agent-output

But I could see when it was compiling that it was only sourcing config from: -H:ConfigurationFileDirectories=app\build\native\generated\generateResourcesConfigFile

So I just copied all the files from app\build\native\agent-output to app\build\native\generated\generateResourcesConfigFile and then it worked:

If it's useful at all, below is an output of the compilation task with verbose logging:

Click to view

  C:\Users\rayga\Projects\tmp\jwm-test
  λ gradlew.bat -Pagent nativeCompile
  
  > Task :app:nativeCompile
  [native-image-plugin] Args are: [-cp, C:\Users\rayga\Projects\tmp\jwm-test\app\build\libs\nativecompile-classpath.jar, --no-fallback, --verbose, -H:Path=C:\Users\rayga\Projects\tmp\jwm-test\app\build\native\nativeCompile, -H:Name=app, -H:ConfigurationFileDirectories=C:\Users\rayga\Projects\tmp\jwm-test\app\build\native\generated\generateResourcesConfigFile, --allow-incomplete-classpath, -H:Class=jwm.test.Application]
  Executing [
  'C:\GraalVM\graalvm-ce-java17-21.3.0-dev\bin\java.exe' \
  -XX:+UseJVMCINativeLibrary \
  -XX:+UseParallelGC \
  -XX:+UnlockExperimentalVMOptions \
  -XX:+EnableJVMCI \
  -Dtruffle.TrustAllTruffleRuntimeProviders=true \
  -Dtruffle.TruffleRuntime=com.oracle.truffle.api.impl.DefaultTruffleRuntime \
  -Dgraalvm.ForcePolyglotInvalid=true \
  -Dgraalvm.locatorDisabled=true \
  -Dsubstratevm.IgnoreGraalVersionCheck=true \
  --add-exports=java.base/com.sun.crypto.provider=ALL-UNNAMED \
  --add-exports=java.base/jdk.internal.access.foreign=ALL-UNNAMED \
  --add-exports=java.base/jdk.internal.event=ALL-UNNAMED \
  --add-exports=java.base/jdk.internal.loader=ALL-UNNAMED \
  --add-exports=java.base/jdk.internal.logger=ALL-UNNAMED \
  --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED \
  --add-exports=java.base/jdk.internal.module=ALL-UNNAMED \
  --add-exports=java.base/jdk.internal.org.objectweb.asm=ALL-UNNAMED \
  --add-exports=java.base/jdk.internal.org.xml.sax.helpers=ALL-UNNAMED \
  --add-exports=java.base/jdk.internal.perf=ALL-UNNAMED \
  --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED \
  --add-exports=java.base/jdk.internal.util.xml.impl=ALL-UNNAMED \
  --add-exports=java.base/jdk.internal.util.xml=ALL-UNNAMED \
  --add-exports=java.base/sun.invoke.util=ALL-UNNAMED \
  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED \
  --add-exports=java.base/sun.reflect.annotation=ALL-UNNAMED \
  --add-exports=java.base/sun.reflect.generics.reflectiveObjects=ALL-UNNAMED \
  --add-exports=java.base/sun.reflect.generics.repository=ALL-UNNAMED \
  --add-exports=java.base/sun.reflect.generics.tree=ALL-UNNAMED \
  --add-exports=java.base/sun.security.jca=ALL-UNNAMED \
  --add-exports=java.base/sun.security.provider=ALL-UNNAMED \
  --add-exports=java.base/sun.security.util=ALL-UNNAMED \
  --add-exports=java.base/sun.text.spi=ALL-UNNAMED \
  --add-exports=java.base/sun.util.calendar=ALL-UNNAMED \
  --add-exports=java.base/sun.util.locale.provider=ALL-UNNAMED \
  --add-exports=java.base/sun.util.resources=ALL-UNNAMED \
  --add-exports=java.xml.crypto/org.jcp.xml.dsig.internal.dom=ALL-UNNAMED \
  --add-exports=jdk.internal.vm.ci/jdk.vm.ci.aarch64=ALL-UNNAMED \
  --add-exports=jdk.internal.vm.ci/jdk.vm.ci.amd64=ALL-UNNAMED \
  --add-exports=jdk.internal.vm.ci/jdk.vm.ci.code.site=ALL-UNNAMED \
  --add-exports=jdk.internal.vm.ci/jdk.vm.ci.code.stack=ALL-UNNAMED \
  --add-exports=jdk.internal.vm.ci/jdk.vm.ci.code=ALL-UNNAMED \
  --add-exports=jdk.internal.vm.ci/jdk.vm.ci.common=ALL-UNNAMED \
  --add-exports=jdk.internal.vm.ci/jdk.vm.ci.hotspot.aarch64=ALL-UNNAMED \
  --add-exports=jdk.internal.vm.ci/jdk.vm.ci.hotspot.amd64=ALL-UNNAMED \
  --add-exports=jdk.internal.vm.ci/jdk.vm.ci.hotspot=ALL-UNNAMED \
  --add-exports=jdk.internal.vm.ci/jdk.vm.ci.meta=ALL-UNNAMED \
  --add-exports=jdk.internal.vm.ci/jdk.vm.ci.runtime=ALL-UNNAMED \
  --add-exports=jdk.internal.vm.ci/jdk.vm.ci.services=ALL-UNNAMED \
  --add-exports=jdk.jfr/jdk.jfr.events=ALL-UNNAMED \
  --add-exports=jdk.jfr/jdk.jfr.internal.consumer=ALL-UNNAMED \
  --add-exports=jdk.jfr/jdk.jfr.internal.handlers=ALL-UNNAMED \
  --add-exports=jdk.jfr/jdk.jfr.internal.jfc=ALL-UNNAMED \
  --add-exports=jdk.jfr/jdk.jfr.internal=ALL-UNNAMED \
  -Xss10m \
  -Xms1g \
  -Xmx14g \
  -Duser.country=US \
  -Duser.language=en \
  -Djava.awt.headless=true \
  -Dorg.graalvm.version=21.3.0-dev \
  -Dorg.graalvm.config=CE \
  -Dcom.oracle.graalvm.isaot=true \
  -Djava.system.class.loader=com.oracle.svm.hosted.NativeImageSystemClassLoader \
  -Xshare:off \
  -Djdk.internal.lambda.disableEagerInitialization=true \
  -Djdk.internal.lambda.eagerlyInitialize=false \
  -Djava.lang.invoke.InnerClassLambdaMetafactory.initializeLambdas=false \
  '-javaagent:C:\GraalVM\graalvm-ce-java17-21.3.0-dev\lib\svm\builder\svm.jar' \
  -cp \
  'C:\GraalVM\graalvm-ce-java17-21.3.0-dev\lib\svm\builder\objectfile.jar;C:\GraalVM\graalvm-ce-java17-21.3.0-dev\lib\svm\builder\pointsto.jar;C:\GraalVM\graalvm-ce-java17-21.3.0-dev\lib\svm\builder\svm.jar' \
  --module-path \
  'C:\GraalVM\graalvm-ce-java17-21.3.0-dev\lib\truffle\truffle-api.jar' \
  'com.oracle.svm.hosted.NativeImageGeneratorRunner$JDK9Plus' \
  -imagecp \
  'C:\Users\rayga\Projects\tmp\jwm-test\app\build\libs\nativecompile-classpath.jar;C:\GraalVM\graalvm-ce-java17-21.3.0-dev\lib\svm\library-support.jar' \
  '-H:Path=C:\Users\rayga\Projects\tmp\jwm-test\app\build\native\nativeCompile' \
  -H:FallbackThreshold=0 \
  '-H:Path=C:\Users\rayga\Projects\tmp\jwm-test\app\build\native\nativeCompile' \
  -H:Name=app \
  '-H:ConfigurationFileDirectories=C:\Users\rayga\Projects\tmp\jwm-test\app\build\native\generated\generateResourcesConfigFile' \
  -H:+AllowIncompleteClasspath \
  -H:Class=jwm.test.Application \
  '-H:CLibraryPath=C:\GraalVM\graalvm-ce-java17-21.3.0-dev\lib\svm\clibraries\windows-amd64'
  ]
  [app:28380]    classlist:   2,552.43 ms,  0.96 GB
  [app:28380]        (cap):   5,273.75 ms,  0.96 GB
  [app:28380]        setup:   7,129.85 ms,  0.96 GB
  [app:28380]     (clinit):     221.34 ms,  2.37 GB
  [app:28380]   (typeflow):   3,175.61 ms,  2.37 GB
  [app:28380]    (objects):   5,393.20 ms,  2.37 GB
  [app:28380]   (features):     652.02 ms,  2.37 GB
  [app:28380]     analysis:  10,019.95 ms,  2.37 GB
  [app:28380]     universe:     973.36 ms,  2.37 GB
  [app:28380]      (parse):     560.62 ms,  2.37 GB
  [app:28380]     (inline):     815.50 ms,  2.37 GB
  [app:28380]    (compile):   7,547.62 ms,  4.67 GB
  [app:28380]      compile:   9,666.75 ms,  4.67 GB
  [app:28380]        image:   1,291.57 ms,  4.67 GB
  [app:28380]        write:   2,160.76 ms,  4.67 GB
  [app:28380]      [total]:  34,818.08 ms,  4.67 GB
  # Printing build artifacts to: C:\Users\rayga\Projects\tmp\jwm-test\app\build\native\nativeCompile\app.build_artifacts.txt
  [native-image-plugin] Native Image written to: C:\Users\rayga\Projects\tmp\jwm-test\app\build\native\nativeCompile

[GavinRay97]

I was also able to export a C ABI function from Java, and compile my JWM app as a .dll then call it from C++!! 😲 😍

graalvmNative {
    binaries {
        main {
            sharedLibrary = true
        }
    }
}

See gist for Application.java code:

• https://gist.github.com/GavinRay97/3b62127e1683b846d42a490fc73bc908

    @CEntryPoint(name = "displayEntrypoint")
    @CEntryPointOptions(prologue = CEntryPointSetup.EnterCreateIsolatePrologue.class, epilogue = CEntryPointSetup.LeaveTearDownIsolateEpilogue.class)
    static void display(long hwnd) {
        App.init();

        WindowWin32 window = new WindowWin32();
        window.setTitle("JWM Window");
        window.setWindowSize(250, 250);
        window.setEventListener(new EventHandler(window));
        window.setVisible(true);
        window.requestFrame();
        // Try reparent
        window.winSetParent(hwnd);

        App.start();
    }

    interface DisplayFunctionPointer extends CFunctionPointer {
        @InvokeCFunctionPointer
        void display(long hwnd);
    }

    private static final CEntryPointLiteral<DisplayFunctionPointer> displayCallback =
        CEntryPointLiteral.create(Application.class, "display");

Compiling this exports a header and a .dll:

[SHARED_LIB]
app.dll

[HEADER]
graal_isolate.h
app.h
graal_isolate_dynamic.h
app_dynamic.h

[IMPORT_LIB]
app.lib

The generated app.h contains the C definition for the exported @CEntryPoint() function:

// app.h
#ifndef __APP_H
#define __APP_H

#include <graal_isolate.h>


#if defined(__cplusplus)
extern "C" {
#endif

void displayEntrypoint(long long int);

int run_main(int argc, char** argv);

void vmLocatorSymbol(graal_isolatethread_t* thread);

#if defined(__cplusplus)
}
#endif
#endif

And if we look at the app.dll in Dependencies.exe, we can see the export here:

Writing a small test which uses a hardcoded window HWND:

// test.cpp
#include "app.h"
#pragma comment(lib, "app") // hacky -Lapp because I'm lazy

int main()
{
    long testHWND = 0x3530ae8;
    displayEntrypoint(testHWND);
    return 1;
}

Then running it -- the Graal-compiled .dll containing our JWM app is successfully started, and the exported function that attaches to another window is able to be called from our C++ program!!

We have liftoff! With all of this in place -- it confirms it is possible to write VST plugins in JVM languages using JWM + Graal! 💯 🔥 🙌

https://user-images.githubusercontent.com/26604994/135004266-5744dae5-32e6-423c-b542-c04baaafb26e.mp4

[i10416]

@GavinRay97

Great! I succeeds to run native image on both Windows and Linux thanks to information in this issue! https://github.com/HumbleUI/JWM/pull/160

[Hilal-Anwar]

That is so cool.

[GavinRay97]

@i10416 Awesome!! I saw you made a great PR! 🙌

Maybe soon the GraalVM checkboxes here will be green? 😅

[tonsky]

Looks great! I wasn’t able to run it on macOS, but Windows worked!

[GavinRay97]

@tonsky 🤔 I assume you probably checked this -- but there I think should be only one difference in the config files between OS'es and that would be the resource-config.json (plus manually copying the shared-lib into the same directory as the built binary)

// app\build\native\agent-output\run\resource-config.json
{
  "resources":{
  "includes":[
    {"pattern":"\\Qjwm.version\\E"}, 
    {"pattern":"\\Qjwm_x64.dll\\E"}
  ]},
  "bundles":[]
}

I suppose this should include the .dylib for Mac and the .so for Linux

This gets generated by running the "profiling agent" (native-image-agent) that instruments and records what the app accesses during it's lifetime:

• https://docs.oracle.com/en/graalvm/enterprise/19/guide/reference/native-image/tracing-agent.html

[tonsky]

Yeah, I did that. I think the problem lies in the fact that macOS requires all windows to be manipulated from the main thread of the app, so I have to write a C program that call JVM program etc. Probably doable, but a lot of stuff to figure out, and I am currently focused elsewhere.

I hope pick it up one day for sure, for now, I am glad you are unblocked. Let me know if anything else is missing in JWM

[GavinRay97]

Ah. Was this with the attempted re-parenting or just a basically empty app? Curious if it runs on Mac at all. And thanks a ton -- really appreciate it! 🙏

There is a LOT of neat stuff that can be built out with this now, for native apps where you would have before used C++ or Rust etc.

Maybe it's even useful for this?

• https://tonsky.me/blog/clojure-ui/

👀

[tonsky]

Without reparenting. Yes I plan to leverage this eventually for Clojure UI