Node-gyp fails on npx but not npm install

[sunsheeppoplar]

In our Jenkins job, we have a script that uploads sourcemaps. In other applications with reasonably similar setup we've been successful with calling npx @datadog/datadog ci sourcemaps upload ...

Not totally sure if this enough information and could totally be user error on my part, but wanted to raise it in any case. Thanks ahead of time for any insight you might be able to provide!

Expected behaviour: When we update to install the lib globally, it works as expected. Note that even though there's still the same Permission denied log from buildcheck.gypi, there's a rollback and it exits cleanly.

Given running this script:

npx @datadog/datadog-ci sourcemaps upload ./dist --minified-path-prefix / --release-version ${DD_VERSION} --service service-name --dry-run

We see:

[2022-07-01T19:56:17.883Z] + npm install -g @datadog/datadog-ci
[2022-07-01T19:56:17.883Z] npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
[2022-07-01T19:56:17.883Z] npm WARN deprecated [email protected]: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
[2022-07-01T19:56:28.238Z] /usr/local/bin/datadog-ci -> /usr/local/lib/node_modules/@datadog/datadog-ci/dist/cli.js
[2022-07-01T19:56:28.238Z] 
[2022-07-01T19:56:28.238Z] > [email protected] install /usr/local/lib/node_modules/@datadog/datadog-ci/node_modules/cpu-features
[2022-07-01T19:56:28.238Z] > node buildcheck.js > buildcheck.gypi && node-gyp rebuild
[2022-07-01T19:56:28.238Z] 
[2022-07-01T19:56:28.238Z] sh: 1: cannot create buildcheck.gypi: Permission denied
[2022-07-01T19:56:28.238Z] 
[2022-07-01T19:56:28.238Z] > [email protected] install /usr/local/lib/node_modules/@datadog/datadog-ci/node_modules/ssh2
[2022-07-01T19:56:28.238Z] > node install.js
[2022-07-01T19:56:28.239Z] 
[2022-07-01T19:56:28.239Z] gyp WARN EACCES current user ("nobody") does not have permission to access the dev dir "/root/.cache/node-gyp/14.19.0"
[2022-07-01T19:56:28.239Z] gyp WARN EACCES attempting to reinstall using temporary dev dir "/usr/local/lib/node_modules/@datadog/datadog-ci/node_modules/ssh2/.node-gyp"
[2022-07-01T19:56:28.239Z] gyp WARN install got an error, rolling back install
[2022-07-01T19:56:28.239Z] gyp WARN install got an error, rolling back install
[2022-07-01T19:56:28.239Z] gyp ERR! configure error 
[2022-07-01T19:56:28.239Z] gyp ERR! stack Error: EACCES: permission denied, mkdir '/usr/local/lib/node_modules/@datadog/datadog-ci/node_modules/ssh2/.node-gyp'
[2022-07-01T19:56:28.239Z] gyp ERR! System Linux 5.4.149-73.259.amzn2.x86_64
[2022-07-01T19:56:28.239Z] gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "--target=v14.19.0" "--real_openssl_major=1" "rebuild"
[2022-07-01T19:56:28.239Z] gyp ERR! cwd /usr/local/lib/node_modules/@datadog/datadog-ci/node_modules/ssh2/lib/protocol/crypto
[2022-07-01T19:56:28.239Z] gyp ERR! node -v v14.19.0
[2022-07-01T19:56:28.239Z] gyp ERR! node-gyp -v v5.1.0
[2022-07-01T19:56:28.239Z] gyp ERR! not ok 
[2022-07-01T19:56:28.239Z] Failed to build optional crypto binding
[2022-07-01T19:56:28.239Z] 
[2022-07-01T19:56:28.240Z] > [email protected] postinstall /usr/local/lib/node_modules/@datadog/datadog-ci/node_modules/aws-sdk
[2022-07-01T19:56:28.240Z] > node scripts/check-node-version.js
[2022-07-01T19:56:28.240Z] 
[2022-07-01T19:56:28.240Z] npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/@datadog/datadog-ci/node_modules/cpu-features):
[2022-07-01T19:56:28.240Z] npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] install: `node buildcheck.js > buildcheck.gypi && node-gyp rebuild`
[2022-07-01T19:56:28.240Z] npm WARN optional SKIPPING OPTIONAL DEPENDENCY: Exit status 2
[2022-07-01T19:56:28.240Z] 
[2022-07-01T19:56:28.240Z] + @datadog/[email protected]
[2022-07-01T19:56:28.240Z] added 228 packages from 241 contributors in 12.015s
[2022-07-01T19:56:28.240Z] + datadog-ci sourcemaps upload ./dist --minified-path-prefix / --release-version 4.9.0-dev1-25ffdfcb --service service-name --dry-run
[2022-07-01T19:56:28.240Z] ⚠️ DRY-RUN MODE ENABLED. WILL NOT UPLOAD SOURCEMAPS
[2022-07-01T19:56:28.240Z] Starting upload with concurrency 20. 
[2022-07-01T19:56:28.240Z] Will look for sourcemaps in dist
[2022-07-01T19:56:28.240Z] Will match JS files for errors on files starting with /
[2022-07-01T19:56:28.240Z] version: 4.9.0-dev1-25ffdfcb service: service-name project path: 
[2022-07-01T19:56:28.240Z] [DRYRUN] Uploading sourcemap dist/main.91df454922e43431b1b0.js.map for JS file available at main.91df454922e43431b1b0.js
[2022-07-01T19:56:28.241Z] [DRYRUN] Uploading sourcemap dist/normalize.53ea77c73475c439d735.js.map for JS file available at normalize.53ea77c73475c439d735.js
[2022-07-01T19:56:28.241Z] [DRYRUN] Uploading sourcemap dist/vendor.7bc6d2cf376bc9132bce.js.map for JS file available at vendor.7bc6d2cf376bc9132bce.js
[2022-07-01T19:56:28.241Z] 
[2022-07-01T19:56:28.241Z] Command summary:
[2022-07-01T19:56:28.241Z] ✅ [DRYRUN] Handled 3 sourcemaps with success in 0.103 seconds.
[2022-07-01T19:56:28.514Z] + echo Datadog sourcemap uploaded
[2022-07-01T19:56:28.514Z] Datadog sourcemap uploaded

Actual behaviour: Given running this script:

npm install -g @datadog/datadog-ci
datadog-ci sourcemaps upload ./dist --minified-path-prefix / --release-version ${DD_VERSION} --service service-name --dry-run

[2022-07-01T19:47:54.814Z] + npx @datadog/datadog-ci sourcemaps upload ./dist --minified-path-prefix / --release-version 4.9.0-dev1-c06d4b89 --service service-name --dry-run
[2022-07-01T19:48:05.107Z] sh: 1: cannot create buildcheck.gypi: Permission denied
[2022-07-01T19:48:05.367Z] internal/modules/cjs/loader.js:905
[2022-07-01T19:48:05.367Z]   throw err;
[2022-07-01T19:48:05.367Z]   ^
[2022-07-01T19:48:05.367Z] 
[2022-07-01T19:48:05.367Z] Error: Cannot find module '/root/.npm/_npx/120/lib/node_modules/@datadog/datadog-ci/node_modules/ssh2/install.js'
[2022-07-01T19:48:05.367Z]     at Function.Module._resolveFilename (internal/modules/cjs/loader.js:902:15)
[2022-07-01T19:48:05.367Z]     at Function.Module._load (internal/modules/cjs/loader.js:746:27)
[2022-07-01T19:48:05.367Z]     at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:75:12)
[2022-07-01T19:48:05.367Z]     at internal/main/run_main_module.js:17:47 {
[2022-07-01T19:48:05.367Z]   code: 'MODULE_NOT_FOUND',
[2022-07-01T19:48:05.367Z]   requireStack: []
[2022-07-01T19:48:05.367Z] }
[2022-07-01T19:48:05.936Z] npm ERR! code ELIFECYCLE
[2022-07-01T19:48:05.936Z] npm ERR! errno 1
[2022-07-01T19:48:05.936Z] npm ERR! [email protected] install: `node install.js`
[2022-07-01T19:48:05.936Z] npm ERR! Exit status 1
[2022-07-01T19:48:05.936Z] npm ERR! 
[2022-07-01T19:48:05.936Z] npm ERR! Failed at the [email protected] install script.
[2022-07-01T19:48:05.936Z] npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
[2022-07-01T19:48:05.936Z] 
[2022-07-01T19:48:05.936Z] npm ERR! A complete log of this run can be found in:
[2022-07-01T19:48:05.936Z] npm ERR!     /root/.npm/_logs/2022-07-01T19_48_05_827Z-debug.log
[2022-07-01T19:48:05.936Z] Install for [ '@datadog/datadog-ci@latest' ] failed with code 1

Additional context Node version in the Docker image is 14.19.0