[BUG] Tar is very slow on npm 6.14.15

[Tallyb]

What / Why

npm 6.14.15 is horribly slow, and it seems to result from upgrading tar from 4.4.14 to 4.4.19.

To reproduce you can try and install a package that needs to extract data like geo-tz(6.0.1). On GHA we saw an increase in npm install from 1 minute to 7 minutes!!!

Check out this repo github actions timing: https://github.com/Tallyb/slow-npm/actions/runs/1228854056

[theseion]

I can confirm that. However, its not the node version but NPM, 6.14.14 was fine 6.14.15 is not. The change of the tar version is from 4.4.15 to 4.4.19.

[theseion]

I've narrowed it down to 4.4.16. Tested with material-design-icons 3.0.1. NPM version: 6.14.14

I manually changed the tar dependency in the npm package.json to 4.4.16, then npm i (for npm). With that the processing time goes up from ~120 seconds to 300+ seconds.

[theseion]

Oh, just for future reference: I got here because my builds stopped working after the node:lts-buster image was upgraded last to use NPM 6.14.15. NPM would start timing out with "cb() never called".

[alerighi]

I have the same problem. It seems related to the directory caching mechanism in mkdir: by disabling the cache, that can simply done by changing the functions cGet and cSet in lib/mkdir.js to empty functions, I improved installation time of geo-tz from 1.33 minutes to just 10s on Linux. It seems faster to try to make the directory, and fail because the file exist, than to write and lookup in the cache, when you have a lot of directories. On Windows it took ages (10s of minutes), with this simple fix only 46 seconds.

I made other tests, and it seems that the cause is the function pruneCache that iterates on every item in the cache. By bypassing it the performance is even better but it should have implications. If I instead change the cache data structure from a Map to a Set the performance seems to improve significantly.

[kuzmeo]

I have similar performance issues on 7.x and 8.x npm versions as well. npm install runs 3-4 times slowly than on 6.14.14 npm version.

[psxpa3]

Hi @Tallyb I am trying to upgrade tar version from 4.4.8 to 6.1.9/6.1.10, however getting the following error - TypeError: Cannot destructure property isAbsolute of 'undefined' or 'null'.

5 | import * as tmp from "tmp"; 6 | // import * as tar from "tar";

7 | const tar = require("tar");

Could you provide any direction on this?. It seems to be breaking at tar.extract(), but everything seems intact to me.

[theseion]

For reference, this appears to be the issue where people noticed that something was going on with NPM: https://github.com/npm/cli/issues/3676