Header extraction is unreliable and results in truncated files

[tmm1]

I frequently see corrupted node/electron headers. Sounds similar to #1054, but I'm using npm. And I see this on an isolate github actions runner, so it doesn't seem like a concurrency issue either?

npm error gyp info using [email protected]
npm error gyp info using [email protected] | linux | x64
npm error gyp info find Python using Python version 3.12.3 found at "/usr/bin/python3"
npm error gyp http GET https://electronjs.org/headers/v34.5.1/node-v34.5.1-headers.tar.gz
npm error gyp http 200 https://electronjs.org/headers/v34.5.1/node-v34.5.1-headers.tar.gz
npm error gyp http GET https://electronjs.org/headers/v34.5.1/SHASUMS256.txt
npm error gyp http 200 https://electronjs.org/headers/v34.5.1/SHASUMS256.txt
...
npm error   File "/home/runner/.cache/node-gyp/34.5.1/include/node/common.gypi", line 177
npm error     'msvs_settings': {
npm error                      ^
npm error SyntaxError: '{' was never closed

more examples:

npm error /home/runner/.cache/node-gyp/34.3.4/include/node/v8-object.h:5: error: unterminated #ifndef
npm error     5 | #ifndef INCLUDE_V8_OBJECT_H_

npm error gyp: /Users/ec2-user/Library/Caches/node-gyp/20.18.2/common.gypi not found while reading includes of binding.gyp while trying to load binding.gyp

npm error gyp: /home/AzDevOps/.cache/node-gyp/30.5.1/common.gypi not found while reading includes of binding.gyp while trying to load binding.gyp

We see this on both linux and macos, but interestingly never on windows. Perhaps due to this?

https://github.com/nodejs/node-gyp/blob/7d883b5cf4c26e76065201f85b0be36d5ebdcc0e/lib/install.js#L203-L206

I would propose making the separate directory download step the default on all platforms.

[tmm1]

I tried setting the env var npm_config_devdir=${{ runner.temp }}/node-gyp, in order to fully isolate the headers extraction destination from everything else.

It still fails randomly with corrupted incomplete files:

npm error   File "/home/runner/work/_temp/node-gyp/34.5.1/include/node/common.gypi", line 55
npm error     'v8_enable_disassembler
npm error     ^
npm error SyntaxError: unterminated string literal (detected at line 55)

[tmm1]

It's almost as if it's not letting the tar.extract() finish until the end...

npm error   File "/home/runner/work/_temp/node-gyp/34.5.1/include/node/common.gypi", line 489
npm error     [ 'OS in "linux freebsd openbsd 
npm error       ^
npm error SyntaxError: unterminated string literal (detected at line 489)

[cclauss]

Can you please test the changes in

• #3167

If they help then we can consider merging that.

[tmm1]

Hmm, my hunch was this is on the nodejs side? The common.gypi file that is frequently corrupted is inside the headers tarball:

❯ curl -v https://electronjs.org/headers/v34.5.1/node-v34.5.1-headers.tar.gz -O
❯ tar tvf node-v34.5.1-headers.tar.gz node_headers/include/node/common.gypi
-rw-rw-rw-  0 0      0       26137 Apr  2 05:11 node_headers/include/node/common.gypi

Do you think sleeps in the update-gyp.py path will help here?

[cclauss]

Do you think sleeps in the update-gyp.py path will help here?

I do not know ~because I have no access to Windows.~

If the header files are corrupted at their source then the changes in #3167 will not help.

If the files are corrupted in the download or move then perhaps the changes in #3167 will help. They need to be tested by someone who ~has access to Windows and~ knows the steps to reproduce the problem.

[tmm1]

Okay thanks. To be clear, the issue happens on Mac and Linux but not Windows.

I will do some testing to see if this helps. The issue is intermittent so it is hard to be certain.

[Jarred-Sumner]

I've seen this happen as well, and I assumed it was a bug in bun install but given it happens in yarn and npm that seems unlikely.

[cclauss]

Can someone please create a GitHub Action that replicates this failure so that we all can see it?