Segfaults JVM, in libc via native code, trying to run Node app

[jamesrgrinter]

trafficstars

Does anyone recognise this crash? Or have some thoughts on how to investigate further?

I'm getting it when trying to run a particular Node application (Node-red, https://github.com/node-red/node-red), with com.eclipsesource.j2v8:j2v8_linux_x86_64:jar:4.8.0

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f2a5deefe48, pid=37, tid=0x00007f29f4cfb700
#
# JRE version: OpenJDK Runtime Environment (8.0_181-b13) (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
# Java VM: OpenJDK 64-Bit Server VM (25.181-b13 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libc.so.6+0x128e48]

Here's what's available in the stack trace, taken from the hs_err_pid.log file:

Stack: [0x00007f29f4bfb000,0x00007f29f4cfc000],  sp=0x00007f29f4cf5a78,  free space=1002k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.6+0x128e48]
C  [libj2v8_linux_x86_64.so+0x55d125]
C  [libj2v8_linux_x86_64.so+0x8fe157]  v8::internal::PropertyCallbackArguments::Call(void (*)(v8::Local<v8::Name>, v8::PropertyCallbackInfo<v8::Value> const&), v8::internal::Handle<v8::internal::Name>)+0x127
C  [libj2v8_linux_x86_64.so+0x905743]  v8::internal::Object::GetPropertyWithAccessor(v8::internal::LookupIterator*)+0x213
C  [libj2v8_linux_x86_64.so+0x930eab]  v8::internal::Object::GetProperty(v8::internal::LookupIterator*)+0x13b
C  [libj2v8_linux_x86_64.so+0x88ee4e]  v8::internal::LoadIC::Load(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Name>)+0x13e
C  [libj2v8_linux_x86_64.so+0x8946ce]  v8::internal::Runtime_LoadIC_Miss(int, v8::internal::Object**, v8::internal::Isolate*)+0x2ce
...
   [ native stack frames, with no info ]
...
C  [libj2v8_linux_x86_64.so+0x7e2bc0]  v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*)+0x1c0
C  [libj2v8_linux_x86_64.so+0x5f4639]  v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*)+0x219
C  [libj2v8_linux_x86_64.so+0x60c971]  v8::Function::Call(v8::Local<v8::Value>, int, v8::Local<v8::Value>*)+0x41
C  [libj2v8_linux_x86_64.so+0x54d301]  node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*)+0x241
C  [libj2v8_linux_x86_64.so+0x581c5f]
C  [libj2v8_linux_x86_64.so+0xe18789]
C  [libj2v8_linux_x86_64.so+0xe1a9eb]
C  [libj2v8_linux_x86_64.so+0xe1aac3]
C  [libj2v8_linux_x86_64.so+0xe2b090]
C  [libj2v8_linux_x86_64.so+0xe1b563]  uv_run+0x123
C  [libj2v8_linux_x86_64.so+0x52c541]  Java_com_eclipsesource_v8_V8__1pumpMessageLoop+0x19c
j  com.eclipsesource.v8.V8._pumpMessageLoop(J)Z+0
j  com.eclipsesource.v8.V8.pumpMessageLoop()Z+4
j  com.eclipsesource.v8.NodeJS.handleMessage()Z+11

(I've looked at the core file, but its frames seem to be corrupted and contain less clues than above).

(gdb) where
Python Exception <class 'gdb.error'> Frame is invalid.: 
#-1 __GI_raise (sig=6) at ../sysdeps/unix/sysv/linux/raise.c:51
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Here's a bit more about my environment (this is executing inside of a Docker container, based upon openjdk:8u181-jre-slim)

OS:PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian

uname:Linux 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018 x86_64
libc:glibc 2.24 NPTL 2.24 
rlimit: STACK 8192k, CORE infinity, NPROC infinity, NOFILE 1048576, AS infinity
load average:1.71 1.95 2.10

# node -v
v7.10.1
# npm -v
4.2.0

(Node and NPM are installed separately in order to build/install the modules that Node-RED requires. I'm currently assuming that it being a slightly later v7 is not the cause of my problem.)

Here's my NodeJS initialisation/loop, which I'm actually executing inside of its own Thread (there's only this one thread interacting with the NodeJS object, and it's the one that crashes):

    protected void runNodeJs() {
      NodeJS nodeJs  = NodeJS.createNodeJS();
      try {
        nodeJs.exec(new File(scriptFile));
        while (nodeJs.isRunning()) {
          nodeJs.handleMessage();
        }
        nodeJs.release();
      } catch (Exception e) {
        logger.error("NodeExecutor encountered error", e);
      } finally {
        if (nodeJs != null) {
          nodeJs.release();
        }
      }
    }

and here's the output from Node-RED at start-up, from within the JVM:

Welcome to Node-RED
===================

27 Nov 10:11:11 - [info] Node-RED version: v0.19.5
27 Nov 10:11:11 - [info] Node.js  version: v7.4.0
27 Nov 10:11:11 - [info] Linux 4.9.125-linuxkit x64 LE
27 Nov 10:11:11 - [info] Loading palette nodes
27 Nov 10:11:12 - [warn] rpi-gpio : Raspberry Pi specific node set inactive
27 Nov 10:11:12 - [warn] rpi-gpio : Cannot find Pi RPi.GPIO python library
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007ff486706e48, pid=41, tid=0x00007ff460dd9700

I could investigate whether it's a mismatch between the run-time libc, and the one that the libj2v8_linux_x86_64.so native code shared library has been linked against, if there are no other ideas.

Here's how the run-time environment is currently resolving libj2v8's dependent libraries, and the package/version they come from:

# ldd libj2v8_linux_x86_64.so 
	linux-vdso.so.1 (0x00007ffe94efa000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f4c3256f000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f4c321ed000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f4c31ee9000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f4c31cd2000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4c31933000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4c31716000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f4c33de2000)

# dpkg -S $(ldd /home/dexda-agent/libj2v8_linux_x86_64.so  | awk '/=>/ { print $3 }') | awk -F: '{ printf "%s:%s\n", $1, $2 }' | sort -u  | xargs -n1 dpkg-query --showformat='${Package} ${Version}\n' --show 
libc6 2.24-11+deb9u3
libgcc1 1:6.3.0-18+deb9u1
libstdc++6 6.3.0-18+deb9u1

[jamesrgrinter]

Had a chance to do a bit more digging. In the Node-RED package removing their "MQTT" node avoids the crash at initialisation.

Notably, that uses the mqtt node module, and one called https-proxy-agent.

Hopefully I'll get the chance to do a bit more investigation later in the week.

[JessHolle]

@irbull Isn't this a good issue to close if Node is no longer in scope? The counter-argument would be if the thought was that J2V8 was doing something improper with memory that this particular combination just happens to shed light on.