Test failures in current master

[rhuanjl]

• Version: node-chakracore - latest master branch on 28/04/2018
• Platform: macOS 10.13.4
• Subsystem: Tests (issue may relate to something else)

Picked this up whilst working on https://github.com/nodejs/node-chakracore/pull/525

The following tests fail (hard crash) in the latest master with no changes when run via make test: test/parallel/test-console-no-swallow-stack-overflow.js test/parallel/test-console-sync-write-error.js test/message/stack_overflow.js

(they do not crash in the same way when run individually)

Steps to reproduce:

1. clone master: git clone https://github.com/nodejs/node-chakracore
2. copy the 3 CC headers (chakracore.h, chakradebug.h and chakracommon.h) from deps/chakrashim/core/lib/jsrt to deps/chakrashim/include)
3. ./configure
4. make -j4
5. make test J4

(step 2 is necessary because of issue https://github.com/nodejs/node-chakracore/issues/512 )

[kfarnung]

I'm not seeing any failures in the CI for #525: https://ci.nodejs.org/job/chakracore-test-osx/330/nodes=osx1010/

If you have the time, could you try the same thing with upstream node?

[rhuanjl]

All passed running with upstream node master.

[rhuanjl]

So to confirm these tests:

1. pass if I run them individually with node-chakracore
2. pass when run via the online CI
3. pass when I run them with upstream-node with offline make test
4. reliably segfault when I run them offline with make-test using node-chakracore

Could be some kind of issue with the test running mechanism and macOS 10.13? (I'm guessing the online CI may uses an older version of macOS?) Or perhaps it's just something up with my computer.

Perhaps just not worth investigating?

[kfarnung]

Definitely worth investigating. The CI does indeed use an older version (I believe it's 10.10.x), so it's possible that there is something in newer macOS versions.

[rhuanjl]

Note I can't think of any way to follow up on/further assess this issue.

[jrasanen]

I can confirm that these tests to throw an error on the latest macOS (10.13.4).

» gcc --version                                                                                                                                                                                        130 ↵
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include/c++/4.2.1
Apple LLVM version 9.1.0 (clang-902.0.39.2)
Target: x86_64-apple-darwin17.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

=== release test-console-no-swallow-stack-overflow === Path: parallel/test-console-no-swallow-stack-overflow Command: out/Release/node /Users/jrasanen/Development/oss/node-chakracore/test/parallel/test-console-no-swallow-stack-overflow.js --- CRASHED (Signal: 11) --- === release test-console-sync-write-error === Path: parallel/test-console-sync-write-error Command: out/Release/node /Users/jrasanen/Development/oss/node-chakracore/test/parallel/test-console-sync-write-error.js --- CRASHED (Signal: 11) --- === release stack_overflow === Path: message/stack_overflow before Command: out/Release/node /Users/jrasanen/Development/oss/node-chakracore/test/message/stack_overflow.js --- CRASHED (Signal: 11) --- [08:09|% 100|+ 2093|- 3]: Done make[1]: *** [jstest] Error 1 make: *** [test] Error 2

Edit: huh, but I can run those just fine one by one:

python tools/test.py -J --mode=release -v test/message/stack_overflow.js test/parallel/test-console-no-swallow-stack-overflow.js test/parallel/test-console-sync-write-error.js
# out/Release/node -p process.arch
# out/Release/node -p process.jsEngine
# out/Release/node -p process.versions.openssl
# out/Release/node -p process.arch
# out/Release/node -p process.jsEngine
# out/Release/node -p process.versions.openssl
# out/Release/node -p process.arch
# out/Release/node -p process.jsEngine
# out/Release/node -p process.versions.openssl
# out/Release/node -p process.config.variables.v8_enable_inspector
[00:00|%   0|+   0|-   0]: release stack_overflow # out/Release/node /Users/jrasanen/Development/oss/node-chakracore/test/parallel/test-console-no-swallow-stack-overflow.js
 # out/Release/node /Users/jrasanen/Development/oss/node-chakracore/test/message/stack_overflow.js
# out/Release/node /Users/jrasanen/Development/oss/node-chakracore/test/parallel/test-console-sync-write-error.js
[00:05|% 100|+   3|-   0]: Done

Running python tools/test.py -J --mode=release -v parallel does not give any crashes, but running make test -j4 will always crash on those tests.

[rhuanjl]

Exactly the same as I observed - glad it's not just an oddity with my computer but no closer to a solution :(

[kfarnung]

@jrasanen That's an interesting observation, what version of make are you running? We've had previous issues where the environment that make was providing was causing crashes, so it's not unheard of.

[jrasanen]

GNU Make 3.81 Copyright (C) 2006 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0

[jrasanen]

Yes! Very interesting! Running these tests through make makes make (or Python?) to crash.

I tested a lot of various combinations. Running this crashes these tests:

.PHONY: jstest4
jstest4:
	python tools/test.py -J --mode=release -v \
		test/message/stack_overflow.js \
		test/parallel/test-console-no-swallow-stack-overflow.js \
		test/parallel/test-console-sync-write-error.js

Results:

$ make jstest4                                                                                                                                                                                      130 ↵
python tools/test.py -J --mode=release -v test/message/stack_overflow.js test/parallel/test-console-no-swallow-stack-overflow.js test/parallel/test-console-sync-write-error.js
# out/Release/node -p process.arch
# out/Release/node -p process.jsEngine
# out/Release/node -p process.versions.openssl
# out/Release/node -p process.arch
# out/Release/node -p process.jsEngine
# out/Release/node -p process.versions.openssl
# out/Release/node -p process.arch
# out/Release/node -p process.jsEngine
# out/Release/node -p process.versions.openssl
# out/Release/node -p process.config.variables.v8_enable_inspector
[00:00|%   0|+   0|-   0]: release stack_overflow ## # out/Release/node /Users/jrasanen/Development/oss/node-chakracore/test/parallel/test-console-no-swallow-stack-overflow.js out/Release/node /Users/jrasanen/Development/oss/node-chakracore/test/parallel/test-console-sync-write-error.jsout/Release/node /Users/jrasanen/Development/oss/node-chakracore/test/message/stack_overflow.js


=== release test-console-no-swallow-stack-overflow ===
Path: parallel/test-console-no-swallow-stack-overflow
Command: out/Release/node /Users/jrasanen/Development/oss/node-chakracore/test/parallel/test-console-no-swallow-stack-overflow.js
--- CRASHED (Signal: 11) ---
=== release test-console-sync-write-error ===
Path: parallel/test-console-sync-write-error
Command: out/Release/node /Users/jrasanen/Development/oss/node-chakracore/test/parallel/test-console-sync-write-error.js
--- CRASHED (Signal: 11) ---
=== release stack_overflow ===
Path: message/stack_overflow
before
Command: out/Release/node /Users/jrasanen/Development/oss/node-chakracore/test/message/stack_overflow.js
--- CRASHED (Signal: 11) ---
[00:06|% 100|+   0|-   3]: Done
make: *** [jstest4] Error 1

But running python tools/test.py -J --mode=release -v test/message/stack_overflow.js test/parallel/test-console-no-swallow-stack-overflow.js test/parallel/test-console-sync-write-error.js from CLI does not throw the signal 11 and these pass gracefully.

[jrasanen]

Makefile and command line, both use the same Python 2.7.15 version of Python, which is from /usr/local/bin/python installed via Homebrew.

[jrasanen]

Everything works well on GNU Make 4.2.1, it does not segfault, unlike macOS' own make is the fault.

This is solved by brew install make (installs GNU Make). Then running commands with gmake instead of make.

Odd thing:

the default make reports that it was built with i386-apple-darwin11.3.0, while gmake from homebrew says x86_64-apple-darwin17.0.0. Wikipedia says that Darwin 11 equals Mac OS X Lion (2011) and Darwin 17 means High Sierra.

[kfarnung]

Thanks for the investigation @jrasanen! I'll try to debug the crash and see what the cause is.