graaljs icon indicating copy to clipboard operation
graaljs copied to clipboard

Published 20 hours ago • verified publisher icon oracle

Random JVM crashes with GraalJS

Open pragatisureka opened this issue 2 years ago • 2 comments

Recently we are getting sporadic JVM crashes when using GraalJS.

Environment

OpenJDK Runtime Environment (Red_Hat-11.0.16.0.8-1.el8_6) (11.0.16+8) (build 11.0.16+8-LTS) GraalVM/JS 21.3.3 built on 11.0.16 JVM options -XX:+UnlockExperimentalVMOptions -XX:+EnableJVMCI --upgrade-module-path=/home/cplace/software/WEB-INF/lib/graaljs Jars available on upgrade-module-path - compiler-21.3.3.jar, compiler-management-21.3.3.jar, graal-sdk-21.3.3.jar, truffle-api-21.3.3.jar

Problem

Since we updated to JDK 11.0.15+ we are seeing random JVM crashes when running some javascript code. Initially we assumed it might be because of incompatible JDK version, we were using GraalJS v21.3.1(built on 11.0.14) with JDK 11.0.15 but after updating graal libraries the crashes still occur.

We observed that the crash only happens when an exception is thrown by java code called from javascript. Another thing we noticed was that most of the crashes happened when scripts ran for long time 10 minutes - 2 hrs

We are unbale to reliably reproduce this since the crashes are really random. If we disbale optimization then there are no crashes but then the performance is really bad.

Below is an example thread dump after the crash. All the crashes are similar and fails when filling stacktrace.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fc18d3cd9bb, pid=8959, tid=537175
#
# JRE version: OpenJDK Runtime Environment (Red_Hat-11.0.16.0.8-1.el8_6) (11.0.16+8) (build 11.0.16+8-LTS)
# Java VM: OpenJDK 64-Bit Server VM (Red_Hat-11.0.16.0.8-1.el8_6) (11.0.16+8-LTS, mixed mode, tiered, jvmci, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x8b89bb]  java_lang_Throwable::fill_in_stack_trace(Handle, methodHandle const&, Thread*)+0x66b
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" (or dumping to /home/cplace/scripts/core.8959)
#
# If you would like to submit a bug report, please visit:
#   https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%208&component=java-11-openjdk
#

---------------  S U M M A R Y ------------

Command Line: -Dcplace.workspaceDirectory=/home/cplace -Dcplace.platform.systemOut.withTime=false -XX:+PrintCommandLineFlags -XX:+CrashOnOutOfMemoryError -Xms8g -Xmx8g -Xss2m -XX:HeapDumpPath=/home/cplace/data/log -XX:ErrorFile=/home/cplace/data/log/hs_err_pid%p.log -Dfile.encoding=UTF-8 -Djava.locale.providers=COMPAT,CLDR -Xlog:gc*,gc+age=trace,safepoint:file=/home/cplace/data/log/cplace_gc.log:utctime,pid,level,tags:filecount=1,filesize=100m -XX:+UnlockExperimentalVMOptions -XX:+EnableJVMCI --upgrade-module-path=/home/cplace/software/WEB-INF/lib/graaljs -Duser.timezone=Europe/Berlin cf.cplace.platform.server.Main

Host: AMD Ryzen 9 5950X 16-Core Processor, 32 cores, 125G, Rocky Linux release 8.6 (Green Obsidian)
Time: Wed Aug 31 19:20:41 2022 CEST elapsed time: 1167478.700490 seconds (13d 12h 17m 58s)

---------------  T H R E A D  ---------------

Current thread (0x00007fbf70206800):  JavaThread "qtp1597545871-238950" [_thread_in_vm, id=537175, stack(0x00007fbdaadf7000,0x00007fbdaaff8000)]

Stack: [0x00007fbdaadf7000,0x00007fbdaaff8000],  sp=0x00007fbdaaff0df0,  free space=2023k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x8b89bb]  java_lang_Throwable::fill_in_stack_trace(Handle, methodHandle const&, Thread*)+0x66b
V  [libjvm.so+0x8b8e02]  java_lang_Throwable::fill_in_stack_trace(Handle, methodHandle const&)+0x82
V  [libjvm.so+0x95cb20]  JVM_FillInStackTrace+0x80
C  [libjava.so+0x17fd5]  Java_java_lang_Throwable_fillInStackTrace+0x15
J 2866  java.lang.Throwable.fillInStackTrace(I)Ljava/lang/Throwable; [email protected] (0 bytes) @ 0x00007fc17455746b [0x00007fc1745573c0+0x00000000000000ab]
J 92874 c2 java.lang.Throwable.fillInStackTrace()Ljava/lang/Throwable; [email protected] (29 bytes) @ 0x00007fc178f12a68 [0x00007fc178f129c0+0x00000000000000a8]
J 2478915 c2 java.lang.Exception.<init>(Ljava/lang/String;)V [email protected] (6 bytes) @ 0x00007fc1759fc91c [0x00007fc1759fc740+0x00000000000001dc]
J 279750 c2 cf.cplace.platform.scripting.api.wrapper.entity.WrappedPage.getIncomingPages(Ljava/lang/String;Ljava/lang/String;Lcf/cplace/platform/assets/Space;)Ljava/util/Set; (171 bytes) @ 0x00007fc178db030c [0x00007fc178dae440+0x0000000000001ecc]
J 2497326 c2 cf.cplace.platform.scripting.api.wrapper.entity.WrappedPage.getIncomingPages(Ljava/lang/String;Ljava/lang/String;Ljava/lang/Iterable;)Ljava/util/Set; (66 bytes) @ 0x00007fc178a6840c [0x00007fc178a67dc0+0x000000000000064c]
j  cf.cplace.platform.scripting.api.wrapper.entity.WrappedPage.getIncomingPagesFromAllSpaces(Ljava/lang/String;Ljava/lang/String;)Ljava/util/Set;+9
J 2150881 c2 java.lang.invoke.LambdaForm$DMH.invokeVirtual(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; [email protected] (18 bytes) @ 0x00007fc1746a1890 [0x00007fc1746a1840+0x0000000000000050]
j  java.lang.invoke.LambdaForm$MH.invoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+93 [email protected]
J 2470315 c2 com.oracle.truffle.host.HostMethodDesc$SingleMethod$MHBase.invokeHandle(Ljava/lang/invoke/MethodHandle;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; org.graalvm.truffle (7 bytes) @ 0x00007fc1757ced88 [0x00007fc1757ced40+0x0000000000000048]
J 2497515 jvmci org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.profiledPERoot([Ljava/lang/Object;)Ljava/lang/Object; jdk.internal.vm.compiler (51 bytes) @ 0x00007fc170e82b4c [0x00007fc170e7af40+0x0000000000007c0c] (:anonymous#2)
J 2474344 jvmci org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.profiledPERoot([Ljava/lang/Object;)Ljava/lang/Object; jdk.internal.vm.compiler (51 bytes) @ 0x00007fc1790388f8 [0x00007fc1790385e0+0x0000000000000318] (Apply@b325d68b#2)
J 2264945 c2 org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.call([Ljava/lang/Object;)Ljava/lang/Object; jdk.internal.vm.compiler (57 bytes) @ 0x00007fc174ce615c [0x00007fc174ce6040+0x000000000000011c]
J 2469010 c2 java.lang.invoke.LambdaForm$MH.invoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; [email protected] (86 bytes) @ 0x00007fc174a9bf74 [0x00007fc174a9bbe0+0x0000000000000394]
J 2470315 c2 com.oracle.truffle.host.HostMethodDesc$SingleMethod$MHBase.invokeHandle(Ljava/lang/invoke/MethodHandle;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; org.graalvm.truffle (7 bytes) @ 0x00007fc1757ced88 [0x00007fc1757ced40+0x0000000000000048]
J 2497456 jvmci org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.profiledPERoot([Ljava/lang/Object;)Ljava/lang/Object; jdk.internal.vm.compiler (51 bytes) @ 0x00007fc17aa6545c [0x00007fc17aa64460+0x0000000000000ffc] (asArray#1)
J 2272186 c2 com.oracle.truffle.js.nodes.function.JSFunctionCallNode$UnboundJSFunctionCacheNode.executeCall([Ljava/lang/Object;)Ljava/lang/Object; (9 bytes) @ 0x00007fc1745b1338 [0x00007fc1745b1280+0x00000000000000b8]
J 2325479 c2 com.oracle.truffle.js.nodes.function.JSFunctionCallNode$CallNode.execute(Lcom/oracle/truffle/api/frame/VirtualFrame;)Ljava/lang/Object; (36 bytes) @ 0x00007fc176dbe5b0 [0x00007fc176dbe4c0+0x00000000000000f0]
J 2439145 c2 com.oracle.truffle.js.nodes.access.WritePropertyNode.executeVoid(Lcom/oracle/truffle/api/frame/VirtualFrame;)V (178 bytes) @ 0x00007fc17a66f0d4 [0x00007fc17a66f020+0x00000000000000b4]
J 2311405 c2 com.oracle.truffle.js.nodes.control.VoidBlockNode.execute(Lcom/oracle/truffle/api/frame/VirtualFrame;)Ljava/lang/Object; (9 bytes) @ 0x00007fc17a2b7ff8 [0x00007fc17a2b7ee0+0x0000000000000118]
J 1806295 c2 com.oracle.truffle.js.nodes.control.AbstractBlockNode.execute(Lcom/oracle/truffle/api/frame/VirtualFrame;)Ljava/lang/Object; (10 bytes) @ 0x00007fc17521aab0 [0x00007fc17521a8e0+0x00000000000001d0]
J 1773966 c2 com.oracle.truffle.js.nodes.function.FunctionBodyNode.execute(Lcom/oracle/truffle/api/frame/VirtualFrame;)Ljava/lang/Object; (9 bytes) @ 0x00007fc17ae0c5e8 [0x00007fc17ae0c5a0+0x0000000000000048]
J 1757752 c2 com.oracle.truffle.js.runtime.JavaScriptRealmBoundaryRootNode.execute(Lcom/oracle/truffle/api/frame/VirtualFrame;)Ljava/lang/Object; (205 bytes) @ 0x00007fc1752cec5c [0x00007fc1752ceb80+0x00000000000000dc]
J 2304581 c2 org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.executeRootNode(Lcom/oracle/truffle/api/frame/VirtualFrame;Lorg/graalvm/compiler/truffle/runtime/CompilationState;)Ljava/lang/Object; jdk.internal.vm.compiler (150 bytes) @ 0x00007fc17aee3230 [0x00007fc17aee31c0+0x0000000000000070]
J 1505499 c1 org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.profiledPERoot([Ljava/lang/Object;)Ljava/lang/Object; jdk.internal.vm.compiler (51 bytes) @ 0x00007fc174d97fa4 [0x00007fc174d97ee0+0x00000000000000c4]
J 2258480 jvmci org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callBoundary([Ljava/lang/Object;)Ljava/lang/Object; jdk.internal.vm.compiler (19 bytes) @ 0x00007fc17462446c [0x00007fc1746243c0+0x00000000000000ac]
J 1616862 c2 com.oracle.truffle.js.nodes.function.ConstructorRootNode.execute(Lcom/oracle/truffle/api/frame/VirtualFrame;)Ljava/lang/Object; (60 bytes) @ 0x00007fc17b02f868 [0x00007fc17b02f5e0+0x0000000000000288]
J 2304581 c2 org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.executeRootNode(Lcom/oracle/truffle/api/frame/VirtualFrame;Lorg/graalvm/compiler/truffle/runtime/CompilationState;)Ljava/lang/Object; jdk.internal.vm.compiler (150 bytes) @ 0x00007fc17aee3230 [0x00007fc17aee31c0+0x0000000000000070]
J 1505499 c1 org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.profiledPERoot([Ljava/lang/Object;)Ljava/lang/Object; jdk.internal.vm.compiler (51 bytes) @ 0x00007fc174d97fa4 [0x00007fc174d97ee0+0x00000000000000c4]
J 2258480 jvmci org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callBoundary([Ljava/lang/Object;)Ljava/lang/Object; jdk.internal.vm.compiler (19 bytes) @ 0x00007fc17462446c [0x00007fc1746243c0+0x00000000000000ac]
J  0x00007fc1766005e8

pragatisureka avatar Sep 01 '22 11:09 pragatisureka

Hi, Thank you for reporting this, we will take a look into it and get back to you

oubidar-Abderrahim avatar Sep 05 '22 16:09 oubidar-Abderrahim

Unfortunately, there's not enough information in this stack trace to figure out what's going on. It would be great if you could try with the latest GraalVM release to see if the problem persists.

woess avatar Feb 28 '23 00:02 woess