graal
graal copied to clipboard
oracle
[GR-59263][Native Image] JDK23+ jdk.internal.org.jline* does not work for Windows
Describe the Issue
Hello,
As I have been debugging build time issues:
- https://github.com/quarkusio/quarkus/issues/43777
- originally reported as https://github.com/oracle/graal/issues/9727
I noticed this runtime issue where terminal escape sequences are ignored in a terminal that cannot handle those and would require Kernel32 API native API calls to e.g. set colors.
Consider this Java code:
public class Main {
public static void main(String[] args) {
final char[] redLineBytes = new char[] {
0x1B, 0x5B, 0x33, 0x31, 0x6D, // ESC[31m Set to red
0x52, 0x45, 0x44, // "RED"
0x1B, 0x5B, 0x30, 0x6D // ESC[0m Reset
};
try (final java.io.PrintWriter w = System.console().writer()) {
w.write(redLineBytes);
w.println(System.lineSeparator() + "NOT RED");
w.flush();
}
}
}
CMDER https://cmder.app/
Cmder terminal understands ESC sequences, so it interprets them and it works nicely both in HotSpot and native:
C:\tmp
λ native-image --version
native-image 23 2024-09-17
OpenJDK Runtime Environment Mandrel-24.1.0.0-Final (build 23+37)
OpenJDK 64-Bit Server VM Mandrel-24.1.0.0-Final (build 23+37, mixed mode)
C:\tmp
λ javac Main.java
C:\tmp
λ java -agentlib:native-image-agent=config-output-dir=AGENT Main
RED
NOT RED
C:\tmp
λ native-image --no-fallback --link-at-build-time -march=native -H:ConfigurationFileDirectories=./AGENT -H:+ForeignAPISupport Main
Warning: The option '-H:+ForeignAPISupport' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: Please re-evaluate whether any experimental option is required, and either remove or unlock it. The build output lists all active experimental options, including where they come from and possible alternatives. If you think an experimental option should be considered as stable, please file an issue.
========================================================================================================================
GraalVM Native Image: Generating 'main' (executable)...
========================================================================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
------------------------------------------------------------------------------------------------------------------------
[1/8] Initializing... (12.1s @ 0.06GB)
Java version: 23+37, vendor version: Mandrel-24.1.0.0-Final
Graal compiler: optimization level: 2, target machine: native
C compiler: cl.exe (microsoft, x64, 19.36.32535)
Garbage collector: Serial GC (max heap size: 80% of RAM)
1 user-specific feature(s):
- com.oracle.svm.thirdparty.gson.GsonFeature
------------------------------------------------------------------------------------------------------------------------
1 experimental option(s) unlocked:
- '-H:+ForeignAPISupport' (origin(s): command line)
------------------------------------------------------------------------------------------------------------------------
Build resources:
- 8.81GB of memory (55.1% of 16.00GB system memory, determined at start)
- 8 thread(s) (100.0% of 8 available processor(s), determined at start)
[2/8] Performing analysis... [*****] (10.8s @ 0.31GB)
3,283 reachable types (71.4% of 4,599 total)
3,852 reachable fields (43.5% of 8,848 total)
15,510 reachable methods (44.7% of 34,727 total)
1,021 types, 13 fields, and 140 methods registered for reflection
61 types, 51 fields, and 52 methods registered for JNI access
0 downcalls and 0 upcalls registered for foreign access
1 native library: version
[3/8] Building universe... (1.8s @ 0.36GB)
[4/8] Parsing methods... [*] (1.3s @ 0.40GB)
[5/8] Inlining methods... [***] (1.1s @ 0.44GB)
[6/8] Compiling methods... [***] (10.1s @ 0.38GB)
[7/8] Laying out methods... [*] (1.7s @ 0.42GB)
[8/8] Creating image... [**] (2.4s @ 0.50GB)
5.45MB (40.77%) for code area: 8,885 compilation units
7.75MB (57.96%) for image heap: 106,145 objects and 1,979 resources
173.48kB ( 1.27%) for other data
13.37MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area: Top 10 object types in image heap:
4.04MB java.base 1.43MB byte[] for java.lang.String
1.01MB svm.jar (Native Image) 1.40MB byte[] for code metadata
114.06kB java.logging 999.47kB java.lang.String
70.33kB org.graalvm.nativeimage.base 782.64kB java.lang.Class
50.67kB jdk.proxy2 304.25kB byte[] for general heap data
40.44kB jdk.proxy1 282.13kB com.oracle.svm.core.hub.DynamicHubCompanion
26.96kB jdk.internal.vm.ci 271.08kB java.util.HashMap$Node
22.05kB org.graalvm.collections 262.34kB java.lang.Object[]
12.43kB jdk.proxy3 183.54kB java.lang.String[]
8.13kB jdk.graal.compiler 164.83kB byte[] for embedded resources
3.28kB for 5 more packages 1.75MB for 935 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
HEAP: Set max heap for improved and more predictable memory usage.
------------------------------------------------------------------------------------------------------------------------
2.1s (4.5% of total time) in 364 GCs | Peak RSS: 0.92GB | CPU load: 5.14
------------------------------------------------------------------------------------------------------------------------
Build artifacts:
C:\tmp\main.exe (executable)
========================================================================================================================
Finished generating 'main' in 43.9s.
C:\tmp
λ main
RED
NOT RED
CMD
Good old cmd terminal you find on all Windows does not understand ESC sequences and it requires new JDK internal JLine (formerly standalone lib) to interpret ESC codes (e.g. WindowsAnsiWriter.java) into Kernel32 API calls with which one can control the terminal.
The bug is that while HotSpot does this, native-image does not go this FFM code path, does not call to Windows Kernel32 API and just dumps the bytes to the terminal:
C:\tmp>native-image --version
native-image 23 2024-09-17
OpenJDK Runtime Environment Mandrel-24.1.0.0-Final (build 23+37)
OpenJDK 64-Bit Server VM Mandrel-24.1.0.0-Final (build 23+37, mixed mode)
C:\tmp>javac Main.java
C:\tmp> java -agentlib:native-image-agent=config-output-dir=AGENT Main
RED
NOT RED
C:\tmp> native-image --no-fallback --link-at-build-time -march=native -H:ConfigurationFileDirectories=./AGENT -H:+ForeignAPISupport Main
Warning: The option '-H:+ForeignAPISupport' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: Please re-evaluate whether any experimental option is required, and either remove or unlock it. The build output lists all active experimental options, including where they come from and possible alternatives. If you think an experimental option should be considered as stable, please file an issue.
========================================================================================================================
GraalVM Native Image: Generating 'main' (executable)...
========================================================================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
------------------------------------------------------------------------------------------------------------------------
[1/8] Initializing... (13.1s @ 0.06GB)
Java version: 23+37, vendor version: Mandrel-24.1.0.0-Final
Graal compiler: optimization level: 2, target machine: native
C compiler: cl.exe (microsoft, x64, 19.36.32535)
Garbage collector: Serial GC (max heap size: 80% of RAM)
1 user-specific feature(s):
- com.oracle.svm.thirdparty.gson.GsonFeature
------------------------------------------------------------------------------------------------------------------------
1 experimental option(s) unlocked:
- '-H:+ForeignAPISupport' (origin(s): command line)
------------------------------------------------------------------------------------------------------------------------
Build resources:
- 8.81GB of memory (55.1% of 16.00GB system memory, determined at start)
- 8 thread(s) (100.0% of 8 available processor(s), determined at start)
[2/8] Performing analysis... [*****] (12.3s @ 0.32GB)
3,283 reachable types (71.4% of 4,599 total)
3,852 reachable fields (43.5% of 8,848 total)
15,510 reachable methods (44.7% of 34,726 total)
1,021 types, 13 fields, and 140 methods registered for reflection
61 types, 51 fields, and 52 methods registered for JNI access
0 downcalls and 0 upcalls registered for foreign access
1 native library: version
[3/8] Building universe... (1.9s @ 0.37GB)
[4/8] Parsing methods... [*] (1.4s @ 0.39GB)
[5/8] Inlining methods... [***] (1.1s @ 0.43GB)
[6/8] Compiling methods... [***] (10.4s @ 0.35GB)
[7/8] Laying out methods... [*] (1.6s @ 0.43GB)
[8/8] Creating image... [**] (2.3s @ 0.50GB)
5.45MB (40.77%) for code area: 8,885 compilation units
7.75MB (57.96%) for image heap: 106,150 objects and 1,980 resources
173.50kB ( 1.27%) for other data
13.37MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area: Top 10 object types in image heap:
4.04MB java.base 1.43MB byte[] for java.lang.String
1.01MB svm.jar (Native Image) 1.40MB byte[] for code metadata
114.06kB java.logging 999.50kB java.lang.String
70.33kB org.graalvm.nativeimage.base 782.64kB java.lang.Class
50.67kB jdk.proxy2 304.25kB byte[] for general heap data
40.44kB jdk.proxy1 282.13kB com.oracle.svm.core.hub.DynamicHubCompanion
26.96kB jdk.internal.vm.ci 271.08kB java.util.HashMap$Node
22.05kB org.graalvm.collections 262.34kB java.lang.Object[]
12.43kB jdk.proxy3 183.54kB java.lang.String[]
8.13kB jdk.graal.compiler 165.82kB byte[] for embedded resources
3.28kB for 5 more packages 1.74MB for 935 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
HEAP: Set max heap for improved and more predictable memory usage.
------------------------------------------------------------------------------------------------------------------------
2.3s (4.7% of total time) in 367 GCs | Peak RSS: 0.88GB | CPU load: 5.20
------------------------------------------------------------------------------------------------------------------------
Build artifacts:
C:\tmp\main.exe (executable)
========================================================================================================================
Finished generating 'main' in 46.8s.
C:\tmp>java Main
RED
NOT RED
C:\tmp>main
[31mRED[0m
NOT RED
JDK21 Optional
For JDK 21 HotSpot, you can switch the jline capability on like this:
C:\tmp> java -Djdk.console=jdk.internal.le Main
Agent
The difference between what the native agent generates as reachability json is quite subtle:
CMDER
{
"resources": [
{
"glob": "META-INF/services/java.lang.System$LoggerFinder"
},
{
"glob": "META-INF/services/org/jline/terminal/provider/jansi"
},
{
"glob": "META-INF/services/org/jline/terminal/provider/jna"
},
{
"glob": "META-INF/services/org/jline/terminal/provider/jni"
},
{
"module": "jdk.internal.le",
"glob": "jdk/internal/org/jline/utils/capabilities.txt"
}
],
"bundles": []
}
CMD
{
"resources": [
{
"glob": "META-INF/services/org/jline/terminal/provider/jansi"
},
{
"glob": "META-INF/services/org/jline/terminal/provider/jna"
},
{
"glob": "META-INF/services/org/jline/terminal/provider/jni"
},
{
"module": "jdk.internal.le",
"glob": "jdk/internal/org/jline/utils/capabilities.txt"
},
{
"module": "jdk.internal.le",
"glob": "jdk/internal/org/jline/utils/windows.caps"
}
],
"bundles": []
}
Reproduce it
SET JAVA_HOME="C:\tmp\graalvm-community-openjdk-24+17.1\"
SET GRAALVM_HOME=%JAVA_HOME%
SET PATH=%JAVA_HOME%\bin;%PATH%
vcvars64
native-image --version
javac Main.java
java -agentlib:native-image-agent=config-output-dir=AGENT Main
native-image --no-fallback --link-at-build-time -march=native -H:ConfigurationFileDirectories=AGENT -H:+ForeignAPISupport Main
CI
You need an actual terminal. If you merely run it in a CI headless, there is no terminal to talk to.
I will keep digging...
WIP
Using the latest version of GraalVM can resolve many issues.
- [X] I tried with the latest version of GraalVM.
GraalVM Version
C:\tmp>native-image --version
native-image 23 2024-09-17
OpenJDK Runtime Environment Mandrel-24.1.0.0-Final (build 23+37)
OpenJDK 64-Bit Server VM Mandrel-24.1.0.0-Final (build 23+37, mixed mode)
and
C:\tmp>native-image --version
native-image 24 2025-03-18
GraalVM Runtime Environment GraalVM CE 24-dev+17.1 (build 24+17-jvmci-b01)
Substrate VM GraalVM CE 24-dev+17.1 (build 24+17, serial gc)
Operating System and Version
Windows 2019 Server
Hi @Karm,
Thank you for reaching out to us about this! I tested the issue you reported and I confirm it. We'll take a closer look into this and I'll make sure to keep you updated.
Hi @Karm,
Yes for sure! Basically that is just an identifier for the ticket that our dev team is working on. Once they have updates regarding this I'll make sure to keep you informed.
Hi @selhagani,
the comment in https://github.com/oracle/graal/blob/f51c7c3049d705c3e53e81f2bc3a29a63f9119a8/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/ServiceLoaderFeature.java#L116-L117
indicates that GR-44085 tracks some console-providers-related issue which might be related to this issue.
Yes for sure! Basically that is just an identifier for the ticket that our dev team is working on. Once they have updates regarding this I'll make sure to keep you informed.
We understand that this is the internal tracker, but we would really appreciate if you could share any non-confidential parts of it. What we would like to know is what is this issue about, what have been tried, and whether there is any info that could potenially help us resolve related issues.
Thank you
Hi @Karm and @zakkak,
Thank you for clarifying your question. From my understanding, these issues do not appear to be related. However, I’ve reached out to our development team to confirm. I’ll follow up with you as soon as I have any updates.