yarn
yarn copied to clipboard
yarnpkg
Spurious "Extracting tar content of undefined failed"
Followup on #6312. Potentially related issues: #7087, #6755, #6312
Details
-
yarn --version: 1.15.2 - yarn.lock
- command-line:
rm ~/.cache/yarn ~/.npm ./node_modules -rf && yarn install --production --frozen-lockfile - Arch Linux. Also reproduced on
node:latestdocker image.
I can still reproduce the issue on the latest yarn version. I do have a git dependency ("@magicfind/wf-build-engine": "https://gitlab.com/magicfind/warframe/wf-build-engine.git") and I'm fairly certain it's related to that, because I've been messing with it, though I don't know what exactly caused this to start happening.
I've been hitting #5235 and trying to work around it, and while doing that started hitting #6312.
Locally I see three different behaviours:
- I hit the error and yarn aborts
- I hit the error and yarn continues
- I do not hit the error and the install succeeds.
This points to a race condition of some type?
Example outputs
All outputs below run the exact same command line, removing all the cache I can find. I've not found a reliable way to reproduce the issue 100% of the time.
yarn install v1.15.2
[1/4] Resolving packages...
[2/4] Fetching packages...
[1/4] Resolving packages...
[2/4] Fetching packages...
error https://registry.yarnpkg.com/require-main-filename/-/require-main-filename-1.0.1.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, chmod '/home/adys/.cache/yarn/v4/npm-require-main-filename-1.0.1-97f717b69d48784f5f526a6c5aa8ffdda055a4d1/node_modules/require-main-filename/LICENSE.txt'"
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
yarn install v1.15.2
[1/4] Resolving packages...
[2/4] Fetching packages...
[1/4] Resolving packages...
[2/4] Fetching packages...
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
$ npm run build
npm WARN lifecycle The node binary used for scripts is /tmp/yarn--1555780098541-0.2687961721890739/node but npm is using /usr/bin/node itself. Use the `--scripts-prepend-node-path` option to include the path for the node binary npm was executed with.
> [email protected] build /home/adys/.cache/yarn/v4/.tmp/83dda1d0ded23f5a9d7b82676a04465a.c67a312e2664e1fa7b0ae1176d36d03b6e40e079.prepare
> rollup -c
src/index.ts → dist/index.js, dist/index.es.js...
created dist/index.js, dist/index.es.js in 1.1s
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning " > [email protected]" has unmet peer dependency "prop-types@^15.6.0".
warning " > [email protected]" has unmet peer dependency "draft-js@^0.10.5".
warning "react-scripts > [email protected]" has incorrect peer dependency "[email protected]".
warning "react-scripts > [email protected]" has incorrect peer dependency "[email protected]".
warning " > [email protected]" has unmet peer dependency "prop-types@^15.5.0".
warning " > [email protected]" has unmet peer dependency "remark-parse@>=3.0.0".
warning " > [email protected]" has unmet peer dependency "webpack@^3.0.0 || ^4.0.0".
[4/4] Building fresh packages...
Done in 23.64s.
yarn install v1.15.2
[1/4] Resolving packages...
[2/4] Fetching packages...
[1/4] Resolving packages...
[2/4] Fetching packages...
error https://registry.yarnpkg.com/is-stream/-/is-stream-1.1.0.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, open '/home/adys/.cache/yarn/v4/npm-is-stream-1.1.0-12d4a3dd4e68e0b79ceb8dbc84173ae80d91ca44/node_modules/is-stream/package.json'"
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
yarn install v1.15.2
[1/4] Resolving packages...
[2/4] Fetching packages...
[1/4] Resolving packages...
[2/4] Fetching packages...
error https://registry.yarnpkg.com/iconv-lite/-/iconv-lite-0.4.24.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, chmod '/home/adys/.cache/yarn/v4/npm-iconv-lite-0.4.24-2022b4b25fbddc21d2f524974a474aafe733908b/node_modules/iconv-lite/encodings/utf16.js'"
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
$ npm run build
npm WARN lifecycle The node binary used for scripts is /tmp/yarn--1555780287695-0.048757454583545856/node but npm is using /usr/bin/node itself. Use the `--scripts-prepend-node-path` option to include the path for the node binary npm was executed with.
> [email protected] build /home/adys/.cache/yarn/v4/.tmp/83dda1d0ded23f5a9d7b82676a04465a.c67a312e2664e1fa7b0ae1176d36d03b6e40e079.prepare
> rollup -c
src/index.ts → dist/index.js, dist/index.es.js...
created dist/index.js, dist/index.es.js in 1.1s
yarn install v1.15.2
[1/4] Resolving packages...
[2/4] Fetching packages...
[1/4] Resolving packages...
[2/4] Fetching packages...
error https://registry.yarnpkg.com/fast-json-stable-stringify/-/fast-json-stable-stringify-2.0.0.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "EEXIST: file already exists, mkdir '/home/adys/.cache/yarn/v4/npm-fast-json-stable-stringify-2.0.0-d5142c0caee6b1189f87d3a76111064f86c8bbf2/node_modules/fast-json-stable-stringify/test'"
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
Weird:
$ yarn cache list
yarn cache v1.15.2
error An unexpected error occurred: "There should only be one folder in a package cache (got in /home/adys/.cache/yarn/v4/npm-@magicfind-wf-build-engine-0.1.0-c67a312e2664e1fa7b0ae1176d36d03b6e40e079/node_modules/@magicfind)".
info If you think this is a bug, please open a bug report with the information provided in "/home/adys/src/overframe/frontend/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/cache for documentation about this command.
tree ~/.cache/yarn/v4/npm-@magicfind-wf-build-engine-0.1.0-c67a312e2664e1fa7b0ae1176d36d03b6e40e079/
/home/adys/.cache/yarn/v4/npm-@magicfind-wf-build-engine-0.1.0-c67a312e2664e1fa7b0ae1176d36d03b6e40e079/
└── node_modules
└── @magicfind
2 directories, 0 files
Edit: Relevant commit: b1f21a80275294afd4c8a2cb244676345868ff52
The undefined showing up in the error message made me wonder if there was another issue at play here but it seems to be a missing parameter which is only used for the error message.
https://github.com/yarnpkg/yarn/blob/4ad88165fdf607a70c75db556a3af95864b2189a/src/fetchers/tarball-fetcher.js#L244-L246
Passing tarballCachePath as a third parameter here will make the error message make more sense:
error https://registry.yarnpkg.com/glob/-/glob-7.1.3.tgz: Extracting tar content of "/home/adys/.cache/yarn/v4/npm-glob-7.1.3-3960832d3f1574108342dafd3a67b332c0969df1/node_modules/glob/.yarn-tarball.tgz" failed, the file appears to be corrupt: "ENOENT: no such file or directory, open '/home/adys/.cache/yarn/v4/npm-glob-7.1.3-3960832d3f1574108342dafd3a67b332c0969df1/node_modules/glob/changelog.md'"
It does nothing to actually help the error though.
I ran rm ~/.cache/yarn ~/.npm node_modules yarn.lock && yarn install --verbose, to see if anything else can be gleaned from there:
verbose 13.540595951 Error: https://registry.yarnpkg.com/fast-json-stable-stringify/-/fast-json-stable-stringify-2.0.0.tgz: Extracting tar content of "/home/adys/.cache/yarn/v4/npm-fast-json-stable-stringify-2.0.0-d5142c0caee6b1189f87d3a76111064f86c8bbf2/node_modules/fast-json-stable-stringify/.yarn-tarball.tgz" failed, the file appears to be corrupt: "ENOENT: no such file or directory, stat '/home/adys/.cache/yarn/v4/npm-fast-json-stable-stringify-2.0.0-d5142c0caee6b1189f87d3a76111064f86c8bbf2/node_modules/fast-json-stable-stringify/test/to-json.js'"
at MessageError.ExtendableBuiltin (/usr/lib/node_modules/yarn/lib/cli.js:721:66)
at new MessageError (/usr/lib/node_modules/yarn/lib/cli.js:750:123)
at Extract.<anonymous> (/usr/lib/node_modules/yarn/lib/cli.js:62682:14)
at Extract.emit (events.js:198:15)
at Extract.module.exports.Extract.destroy (/usr/lib/node_modules/yarn/lib/cli.js:139624:17)
at onunlock (/usr/lib/node_modules/yarn/lib/cli.js:139501:26)
at /usr/lib/node_modules/yarn/lib/cli.js:45391:25
at /usr/lib/node_modules/yarn/lib/cli.js:45357:23
at /usr/lib/node_modules/yarn/lib/cli.js:62632:13
at FSReqCallback.oncomplete (fs.js:158:21)
error https://registry.yarnpkg.com/fast-json-stable-stringify/-/fast-json-stable-stringify-2.0.0.tgz: Extracting tar content of "/home/adys/.cache/yarn/v4/npm-fast-json-stable-stringify-2.0.0-d5142c0caee6b1189f87d3a76111064f86c8bbf2/node_modules/fast-json-stable-stringify/.yarn-tarball.tgz" failed, the file appears to be corrupt: "ENOENT: no such file or directory, stat '/home/adys/.cache/yarn/v4/npm-fast-json-stable-stringify-2.0.0-d5142c0caee6b1189f87d3a76111064f86c8bbf2/node_modules/fast-json-stable-stringify/test/to-json.js'"
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
Full log: verbose.txt.gz
This is on 1.15.2.
Yes, my bad. "yarn install --network-concurrency 1" is working for me, a temporary, slow workaround.
I used the 1.7 version before today, it never happened, and today I upgraded to the latest version (1.15.2), it will happen: Extracting tar content of undefined failed, the file appears to be corrupt
I was seeing this fairly consistently on 1.16.0, and was finally able to track this down.
Our setup is as follows:
- A
/package.jsonwhich defines a bunch of workspaces - A
/workspace/package.jsonwhich references a git modules - A custom build of a node module in git, with a
package.jsonwith ascripts -> preparekey.
This was causing an extra instance of yarn to run that script, which then tried to fetch packages needed by this lib outside of the expected order. Since the lib in question is mostly dead, and barely needed in our case I was able to resolve the problem by removing the prepare script.
For everyone else having these problems I recommend removing any git packages you have referenced, and failing that to grep all the package.json files in your project for these reserved scripts since they may be the cause. This is almost certainly the case if you see this sort of duplication:
[1/4] Resolving packages...
[2/4] Fetching packages...
[1/4] Resolving packages...
[2/4] Fetching packages...
Hope this helps root cause this in a more reliable fashion.
@arcanis You mentioned in #6312 that it was closed because there was no further details. I believe my post above this one may explain what's happening here. It look like (as an educated guess) that the instance of yarn being called to service the special scripts like install, postinstall, prepublish, and prepare is not respecting any sort of mutex config.
I was seeing this fairly consistently on 1.16.0, and was finally able to track this down.
Our setup is as follows:
- A
/package.jsonwhich defines a bunch of workspaces- A
/workspace/package.jsonwhich references a git modules- A custom build of a node module in git, with a
package.jsonwith ascripts -> preparekey.This was causing an extra instance of yarn to run that script, which then tried to fetch packages needed by this lib outside of the expected order. Since the lib in question is mostly dead, and barely needed in our case I was able to resolve the problem by removing the prepare script.
For everyone else having these problems I recommend removing any git packages you have referenced, and failing that to grep all the package.json files in your project for these reserved scripts since they may be the cause. This is almost certainly the case if you see this sort of duplication:
[1/4] Resolving packages... [2/4] Fetching packages... [1/4] Resolving packages... [2/4] Fetching packages...Hope this helps root cause this in a more reliable fashion.
THANK YOU! I had recently added .yarnrc with --install.mutex network to fix another issue (multiple simultaneous yarn in different repo on the same build server) and having this similar-but-shouldn't-be-possible-anymore error popped up really freaked me out. I noticed the repeated [1/4]... stuff though and turns out packages recursively calling yarn was indeed the problem!
I started by piping yarn into a file e.g. yarn --verbose 2>&1 | tee yarn-output and looking for clues around the area where I saw [1/4] repeat itself:
verbose 34.141 Performing "GET" request to "https://registry.yarnpkg.com/synchronous-promise/-/synchronous-promise-2.0.7.tgz".
verbose 34.15 Performing "GET" request to "https://registry.yarnpkg.com/toposort/-/toposort-2.0.2.tgz".
verbose 34.592 Checking for configuration file "/Users/xxxxx/Library/Caches/Yarn/v4/.tmp/e237f3006359452a9561519898e180fb.5025ebf2427d709adfb10ee07f7dc36517d03694.prepare/.npmrc".
verbose 34.592 Checking for configuration file "/Users/xxxxx/.npmrc".
verbose 34.592 Checking for configuration file "/Users/xxxxx/.nvm/versions/node/v10.15.0/etc/npmrc".
verbose 34.592 Checking for configuration file "/Users/xxxxx/Library/Caches/Yarn/v4/.tmp/e237f3006359452a9561519898e180fb.5025ebf2427d709adfb10ee07f7dc36517d03694.prepare/.npmrc".
verbose 34.592 Checking for configuration file "/Users/xxxxx/Library/Caches/Yarn/v4/.tmp/.npmrc".
verbose 34.592 Checking for configuration file "/Users/xxxxx/Library/Caches/Yarn/v4/.npmrc".
verbose 34.593 Checking for configuration file "/Users/xxxxx/Library/Caches/Yarn/.npmrc".
verbose 34.593 Checking for configuration file "/Users/xxxxx/Library/Caches/.npmrc".
verbose 34.593 Checking for configuration file "/Users/xxxxx/Library/.npmrc".
verbose 34.593 Checking for configuration file "/Users/xxxxx/.npmrc".
verbose 34.593 Checking for configuration file "/Users/.npmrc".
verbose 34.593 Checking for configuration file "/Users/xxxxx/Library/Caches/Yarn/v4/.tmp/e237f3006359452a9561519898e180fb.5025ebf2427d709adfb10ee07f7dc36517d03694.prepare/.yarnrc".
verbose 34.593 Checking for configuration file "/Users/xxxxx/.yarnrc".
verbose 34.593 Found configuration file "/Users/xxxxx/.yarnrc".
verbose 34.594 Checking for configuration file "/Users/xxxxx/.nvm/versions/node/v10.15.0/etc/yarnrc".
verbose 34.594 Checking for configuration file "/Users/xxxxx/Library/Caches/Yarn/v4/.tmp/e237f3006359452a9561519898e180fb.5025ebf2427d709adfb10ee07f7dc36517d03694.prepare/.yarnrc".
verbose 34.594 Checking for configuration file "/Users/xxxxx/Library/Caches/Yarn/v4/.tmp/.yarnrc".
verbose 34.594 Checking for configuration file "/Users/xxxxx/Library/Caches/Yarn/v4/.yarnrc".
verbose 34.594 Checking for configuration file "/Users/xxxxx/Library/Caches/Yarn/.yarnrc".
verbose 34.594 Checking for configuration file "/Users/xxxxx/Library/Caches/.yarnrc".
verbose 34.594 Checking for configuration file "/Users/xxxxx/Library/.yarnrc".
verbose 34.594 Checking for configuration file "/Users/xxxxx/.yarnrc".
verbose 34.595 Found configuration file "/Users/xxxxx/.yarnrc".
verbose 34.595 Checking for configuration file "/Users/.yarnrc".
warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
[1/4] Resolving packages...
[2/4] Fetching packages...
verbose 34.869 Performing "GET" request to "https://registry.yarnpkg.com/core-js/-/core-js-2.5.7.tgz".
verbose 34.87 Performing "GET" request to "https://registry.yarnpkg.com/css-box-model/-/css-box-model-1.0.0.tgz".
luckily, I decided to check out /Users/xxxxx/Library/Caches/Yarn/v4/.tmp/e237f3006359452a9561519898e180fb.5025ebf2427d709adfb10ee07f7dc36517d03694.prepare/.yarnrc and that folder was still around - checked out the package.json and realized that it was a library that I had forked and included as my dependancy recently. I'm guessing that the prepare script isn't called when the package is included "naturally" through npm, but it is when it is included as a github repo. For the record, the package I forked was react-beautiful-dnd. Once I updated my fork to remove the prepare script, the phantom pipeline issues stopped appearing. THANK YOU!
try this way in your Dockerfile:
RUN mkdir .yarncache
RUN yarn install --cache-folder ./.yarncache
RUN rm -rf .yarncache
or in your workspace:
mkdir .yarncache
yarn install --cache-folder ./.yarncache
rm -rf .yarncache
- Seeing this on my CI machine but not my dev machine with yarn 1.16.0
- I have seen problem with both yarn 1.16.0 and 1.17.0
#6312 suggested adding '--network-concurrency 1' which ** seems ** to work.
# https://github.com/yarnpkg/yarn/issues/6312
# Added '--network-concurrency 1'
#
# Example Error:
# error confusing-browser-globals-1.0.7.tgz:
# Integrity check failed for "confusing-browser-globals" (computed integrity doesn't match our records
yarn install --network-concurrency 1
@esutton Check my first comment in this thread here. You can probably get away without having to use the network-concurrency key if you find the offending lib.
I'm very disappointed with this issue and had to switch to pure npm. I wonder why yarn community can't fix the issue so long.
- For now, I will stay with
yarn install --network-concurrency 1 - Uninstalling yarn and reverting to npm will be my next step.
@bitsal yarn has caused way too much time debugging build failures. I do not recall having to do the same with npm which just seemed to work.
@TikiTDO Thanks for advice, however I am using react native which has too many packages and dependencies, to sort through which ones are problem packages. While I enjoy a good puzzle, it is far less time consuming to revert to using npm and uninstall yarn.
@esutton yarn install --network-concurrency 1 doesn't fully help me
I still have errors like this:
yarn install v1.16.0
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "ENOENT: no such file or directory, open '.../.cache/yarn/v4/npm-spdx-exceptions-2.2.0-2ea450aee74f2a89bfb94519c07fcd6f41322977/node_modules/spdx-exceptions/.yarn-metadata.json'".
info If you think this is a bug, please open a bug report with the information provided in ".../yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
script returned exit code 1
https://github.com/yarnpkg/yarn/issues/7212#issuecomment-493720324
I can confirm the issue seems to be related to a git dependency. But a workspace is not needed in my case to repro the problem.
Also, in my case it happens on a Docker node:12.7-alpine (I tried even with 11.x) image, but on on a a standard [email protected] travis worker.
The --ignore-scripts suggestion may be relevant to the issue at hand, though it honestly looks like that issue is really a few problems rolled into one discussion
Update found a workaround that does work.... a git submodule and then have the location of the package be the local folder. Not ideal but it works.
Think we're having this problem too details here https://github.com/yarnpkg/yarn/issues/2629#issuecomment-522871612 seems related to having a dependency hosted on github.com
When you have more than 1 packages that install over Git (that have prepare scripts), this error happens almost every time and is not spurious anymore.
I confirm that this is a functional workaround:
yarn install --network-concurrency 1
This issue appears to have resurfaced in yarn version 1.22.0 (MacOS). The issue is persistent regardless of installation method (via brew, npm, or install script). The offending package is a git repository with no scripts.
--network-concurrency and --ignore-scripts flags did not work for me.
Downgrading yarn to 1.21.1 did resolve this issue, however.
yarn policies set-version 1.21.1
That fixed the problem for me, thanks @dlio! (--network-concurrency 1 did not.)
Based on the errors I'm seeing in https://github.com/pierrec/node-lz4/issues/92, it seems that yarn is extracting the tgz files in an incorrect order, causing failures when they contain hard links?
Hi there.
I found a way how to reproduce this issue. The error is caused when you run a parallel yarn install at the same project. We faced this issue on our Jenkins server.
- delete your node_modules folder
- execute
yarn cache clean - open 2 console tabs
- ensure that you run
yarn installparallel in both tabs
This will cause the following error.
error https://registry.yarnpkg.com/cssstyle/-/cssstyle-1.2.2.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, chmod '/home/amoeller/.cache/yarn/v6/npm-cssstyle-1.2.2-427ea4d585b18624f6fdbf9de7a2a1a3ba713077-integrity/node_modules/cssstyle/.eslintrc.js'"
Haven't found a solution yet.
@andrmoel If you're actually running two instances on yarn concurrently, there is are a couple ways to have yarn not interfere with itself. Take a look at the --mutex options for yarn.
However, I believe most people here are not intentionally running two concurrent instances of yarn and the --mutex options do not solve the problem.
Provided workarounds failed for me. This change to the offending dependency in package.json worked in my case:
- "require-css": "guybedford/require-css#<hash>",
+ "require-css": "https://github.com/guybedford/require-css#<hash>",
Other dependencies also had the shorter github references, but I did not have to change those.
I arrived at that solution through obscure hints on Github, shotgun programming, and mysticism. No clue why it works. I am a newbie all over again and question my professionalism. HTH someone, or I'll question my sanity also.
