Package sometimes causes segfaults on Node v14

[davidmurdoch]

I'm sorry my reproduction steps are so horrible... but I just haven't been able to figure out how to reliably trigger the error.

To reproduce:

0. git clone https://github.com/trufflesuite/ganache-core.git
1. cd ganache-core
2. git checkout 8a4fb09c970d2ded061c1b1e1fd912cd7ae0229a
3. nvm use 14
4. npm ci
5. npm test (you may have to run this many times until you trigger the segfault)

You should see something like:

Segmentation fault (core dumped)
npm ERR! code 139
npm ERR! path /home/david/work/ganache-core
npm ERR! command failed
npm ERR! command sh -c mocha --check-leaks --recursive "./test/local"

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/david/.npm/_logs/2020-10-07T15_21_43_948Z-debug.log

(The debug log isn't helpful)

I've somewhat "fixed" this in ganache-core by patching keccak so it never loads the native modules.

To reproduce my workaround:

1. git checkout c26ba24ebf24d1c09b328a76baf0cf43e82fba04
2. npm ci (if you are using npm 7.0.0-rc.0 you'll need to also run npm run postinstall)
3. npm test (run it as many times as you'd like; it doesn't crash anymore... I ran the tests for over an hour via while npm test; do :; done without a crash)

Here is the patch, if you're curious: https://github.com/trufflesuite/ganache-core/blob/c26ba24ebf24d1c09b328a76baf0cf43e82fba04/patches/keccak%2B3.0.1.patch#L1-L11

The biggest issue with my workaround is that patch-package isn't safe to use in published npm modules. :-(

I've also opened this issue in ethereum-cryptography because I figured it might be faster/easier to update that package to do something like const keccak = require("keccack/js"); (untested) than it would be to figure out the fix in this package's native code.

Note: removing nyc from the test command reduces the likelihood of the crash, however, it does still occur when running with mocha only.

[fanatid]

:disappointed:

Is there a way to get a simpler code for reproducing error? Maybe a specified function call with specified data?

[davidmurdoch]

I can't narrow it down further right now, as I've got some other important issues to work on (updating ganache-core to work with Node v14 has taken me well over a week now!). I haven't been able to find a minimal reproduction, as when I reduce complexity the the segfault error becomes less likely to occur.

The thought has occurred to me that it is possible keccak isn't even the problem; another native package may be misbehaving by attempting to read/write memory, and having keccak native loaded simply triggers a bug in the other native library -- something that causes its own internal memory references to be misaddressed/aligned :sweat_smile:

I think right now, the best we can hope for is that others run into the same problem and find this issue and comment below :-).

I've found a temporary workaround in ganache (bundling keccak into ganache-core without the native modules), so it's not a super high priority for ganache-core specifically.

Here are some reproductions of the issue on Travis CI:

Node.js: 14.13.0 Ubuntu 16.04: https://travis-ci.org/github/trufflesuite/ganache-core/jobs/733700191#L1324

Node.js: 14.13.0 OS X 10.12.6: https://travis-ci.org/github/trufflesuite/ganache-core/jobs/733700194#L1101

[davidmurdoch]

Oh wait... I just found this failure on Node v10 on OS X 10.12.6!

https://travis-ci.org/github/trufflesuite/ganache-core/jobs/733700192#L1149

Hmmmm, this is the first I've noticed it on anything but Node 14. Not sure what to make of this discovery...