Yarn install does not install missing modules in node_modules

[classicspam]

Do you want to request a feature or report a bug?

Bug

What is the current behavior?

Missing module directories in node_modules are not installed when running yarn install after you get the message "success Already up-to-date". Please note that a work around is to always run yarn install --force, however that slows down yarn install if no new modules are needed. Also everything seems to work with yarn install if the node_modules directory is deleted.

If the current behavior is a bug, please provide the steps to reproduce.

1. Add a module (i.e. gulp-concat) through yarn -> "yarn add gulp-concat"
2. Run "yarn install" until the quick return with "Already up-to-date"
3. rm -rf node_modules/gulp-concat
4. Run "yarn install"
5. It will quick return with the message "Already up-to-date" without reinstalling gulp-concat

What is the expected behavior?

If a module is missing from node_modules (i.e. directory is not there) yarn should install the missing module and not report everything is up to date.

Think of a multi developer environment where one developer installs a module and updates the source control with the updated yarn.lock file and/or offline cache but not the node_modules directory. Another developer updates and then runs yarn install, which should not report back that everything is up to date.

Please mention your node.js, yarn and operating system version.

yarn 0.17.10 node.js 6.9.2 os CentOS 6.7 (I also tested on Windows)

[ValYouW]

Any news on that? Thx.

[joshma]

Interestingly, I found that this doesn't affect new packages:

# in a git repo with node_modules tracked (just to undo install)
yarn add left-pad
git add package.json yarn.lock
git commit -m save
git clean -df && git checkout -- node_modules/
# this will install left-pad
yarn install

My guess is that node_modules/.yarn-integrity is doing something here, such that if you update yarn.lock it'll notice things are out of sync. It does mean you can't be fidgeting around with node_modules, though.

[BjornRombaut]

Same issue here.

I would expect that yarn installs packages missing from my node_modules folder when I use the yarn install command.

[glebec]

Yep, this is definitely something to do with the integrity check. If you run yarn check it correctly notes the missing dependency. But when you simply yarn / yarn install, it assumes all is well. Delete yarn.integrity and it obviously rebuilds.

This is potentially problematic also if you switch a package from dependencies to devDependencies and attempt to re-yarn install. It claims everything is already up to date. You can yarn --force, but that ignores caches, which isn't quite what we need.

What we need is for yarn install to automatically do an integrity check. If the check fails, it should rebuild.

[mattyork]

Any updates on this? This is causing lots of wasted time here at Tableau.

[chlab]

To add to this, if you rm -rf node_modules/*, then try to yarn install again, it says success Already up-to-date. If you run yarn check it notices all the missing modules, but another yarn install will still not install them. You have to actually delete the node_modules folder itself to reinstall.

[omarkhan]

Thanks for the tip @chlab, that was driving me nuts.

[steven-pribilinskiy]

Looks like yarn install --check-files is the answer, but why it's not documented.

[raido]

@pribilinskiy https://github.com/yarnpkg/website/pull/497

[luchillo17]

@pribilinskiy @raido I've found yarn --force or yarn install --force does the trick as well, what's the difference?

[steven-pribilinskiy]

@luchillo17 the --force flag probably should be used as the last resort. It will also make requests to the server, ignoring the cache, all packages will be downloaded again.

[luchillo17]

@pribilinskiy Indeed its a nuke solution.

[ricklove]

Tried everything here, then realized I was in the wrong directory in my cli... 30 mins later 😭 😆

[palfrey]

yarn install --skip-integrity-check seems to actually do the trick

[jasonmorita]

I had what seemed like the same problem, but my issue ended up being the NODE_ENV var was accidentally set to production on my system and I could not install devDependencies. This is not a bug and is not specific to Yarn, but just in case this is your issue you can do yarn install --production=false

[jhnferraris]

Experienced the same when we pulled a devDependency from a remote repository.

We had to manually execute yarn add <package name> --dev to download it. Luckily, yarn.lock did not change at all after execution.

[vamshi9666]

i tried everything in here , but still facing this error . i mentioned script to start server and

`npm start

[email protected] start /home/vamshi/myApps/node/first nodemon index.js

module.js:540 throw err; ^

Error: Cannot find module './node.js' at Function.Module._resolveFilename (module.js:538:15) at Function.Module._load (module.js:468:25) at Module.require (module.js:587:17) at require (internal/module.js:11:18) at Object. (/home/vamshi/myApps/node/first/node_modules/nodemon/node_modules/debug/src/index.js:9:20) at Module._compile (module.js:643:30) at Object.Module._extensions..js (module.js:654:10) at Module.load (module.js:556:32) at tryModuleLoad (module.js:499:12) at Function.Module._load (module.js:491:3) npm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! [email protected] start: nodemon index.js npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the [email protected] start script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in: npm ERR! /home/vamshi/.npm/_logs/2018-02-28T15_45_22_939Z-debug.log and when i tried to install add dependencies . yarn install v1.3.2 [1/4] Resolving packages... success Already up-to-date. Done in 0.86s. `

[squadette]

@vamshi9666, you're using npm, and this issue is about yarn. If you installed packages with yarn I guess you should use yarn run start.

This throws err; thing is a sure sign that you're running two different programs that pick up a different list of packages, so you need to fix that.

[vamshi9666]

even yarn start gives same error in log @squadette

[avalanche1]

I had this issue but turns out it was not wit yarn but with Webstorm. I did yarn install several times and Webstorm kept reporting that packages were not installed. Then I deleted node_modules folder altogether and did yarn install. And Webstorm didn't even show me node_modules in the project tree! But in reality it was present in my project folder. Turns out it's a bug with Webstorm and to remedy it you have to File->Synchronise to make it see updated file structure.

[Macil]

Think of a multi developer environment where one developer installs a module and updates the source control with the updated yarn.lock file and/or offline cache but not the node_modules directory. Another developer updates and then runs yarn install, which should not report back that everything is up to date.

I don't think this scenario is actually broken. The reproduction steps you mentioned don't match this scenario. When you run yarn, it creates a "node_modules/.yarn-integrity" file listing out everything it installed in node_modules if that file isn't already present. If it is present, yarn compares that file (and not the rest of node_modules) against yarn.lock to see what needs to be changed.

So if one developer commits some changes to package.json and yarn.lock (because they run yarn add foo on their machine and committed the changes while node_modules was .gitignore-d.), and then a different developer pulls those changes and runs yarn, their yarn will correctly see that foo is in their yarn.lock but not in node_modules/.yarn-integrity, and it will add foo to node_modules.

[dakotahawkins]

I think this happened to me in a docker container, when package versions have been upgraded. The container runs Alpine linux, which only has yarn 1.7.0 in its stable repository. Installing yarn 1.12.3 from the edge repositories fixed the problem, but here's what happens anyway in case other people are in the same boat:

• package.json and yarn.lock updated on development machine with yarn upgrade
• package.json and yarn.lock are updated in container, node_modules is not
• Container basically does yarn check --integrity && yarn check --verify-tree || yarn install
  • yarn check --integrity fails
  • yarn install appears to work
• Subsequent yarn check --integrity succeeds
• Subsequent yarn check --verify-tree fails!
  • Packages in node_modules are still at their old versions

Here's the relevant output of the commands:

$ yarn check --integrity

success Folder in sync.
Done in 0.23s.

$ yarn check --verify-tree

success Folder in sync.
Done in 6.00s.

$ rsync -av --delete --exclude 'node_modules' /project/src/front-end/ /project/front-end

sending incremental file list
./
package.json
yarn.lock

sent 239,342 bytes  received 76 bytes  478,836.00 bytes/sec
total size is 425,735  speedup is 1.78

$ yarn check --integrity

warning Integrity check: Top level patterns don't match
error Integrity check failed
error Found 1 errors.

$ yarn check --verify-tree

error "@angular/animations" is wrong version: expected "^7.1.1", got "7.1.0"
...
error Found 16 errors.

$ yarn install --production=false

[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
Done in 38.73s.

$ yarn check --integrity

success Folder in sync.
Done in 0.24s.

$ yarn check --verify-tree

error "@angular/animations" is wrong version: expected "^7.1.1", got "7.1.0"
...
error Found 22 errors.

But here's some information from yarn install --production=false --verbose:

verbose 3.462 Selecting "@angular/[email protected]" at level 0 as the peer dependency of "@angular/[email protected]".
...
verbose 4.8 Skipping copying of file "/home/project/.cache/yarn/v1/npm-@angular/animations-7.1.1-8fecbd19417364946a9ea40c8fdf32462110232f/package.json" as the file at "/project/front-end/node_modules/@angular/animations/package.json" is the same size (1295) and mtime (499162500000).

Sure enough, it skipped a ton of stuff:

$ head -n 3 node_modules/@angular/animations/package.json

{
  "name": "@angular/animations",
  "version": "7.1.0",

The only thing that works without upgrading is blowing away the node_modules folder before running install, so I have to choose between that and installing a bunch of stuff from edge repositories :(

[kserjey]

I've run into this issue. I solve this problem with:

rm -rf node_modules yarn.lock
npm install
yarn import
yarn install --check-files

Looks like it worked!

[Sahan-srt]

I had the same issue, 1st => remove existing node modules(Might not mandatory in some cases ), 2nd run 'yarn cache clean && yarn install'

thats it.

[mfolnovic]

Out of all comments, the only thing that helps in my case is: yarn install --production=false.

In all other cases, after deleting node_modules and running yarn install (or other suggestions), I only get .yarn-integrity file in node_modules.

EDIT: also, after each yarn add, I need to run yarn install --production=false as it deletes all devDependencies.

[kg-currenxie]

EDIT: also, after each yarn add, I need to run yarn install --production=false as it deletes all devDependencies.

I also get this. yarn 1.17.3

[LouizFC]

Out of all comments, the only thing that helps in my case is: yarn install --production=false.

In all other cases, after deleting node_modules and running yarn install (or other suggestions), I only get .yarn-integrity file in node_modules.

EDIT: also, after each yarn add, I need to run yarn install --production=false as it deletes all devDependencies.

This saved me, thank you. I am working with mixed project (java, kotlin, javascript and typescript) in a monorepo. I was thinking that the problem was on the frontend-maven-plugin, but after using your workaround it works perfectly.

For people that may stumble upon this issue: I am using workspaces and maven multimodules, maven coordinates build hierarchy, workspaces coordinates yarn dependencies (it looks complicated but it is actually very simple, I am surprised that there are no tutorials of it around the web, maybe it is just to niiche).

There is a section on the plugin called environmentVariables, mine looks like this:

<environmentVariables>
  <NODE_ENV>production</NODE_ENV>
  <BABEL_ENV>build</BABEL_ENV>
  <TS_NODE_PROJECT>\"./tsconfig.eslint.json\"</TS_NODE_PROJECT>
</environmentVariables>

Now, for some god knows why reason, if I remove this variables, the build works, but I need to prefix every script with cross-env NODE_ENV=production BABEL_E.... etc.

With --production=false flag it works perfectly, now I am guessing that the problem lies in the interaction that yarn has with NODE_ENV.

That concludes my random rambling. I hope this turns out to be useful for someone. Thank you.

EDIT: https://classic.yarnpkg.com/en/docs/cli/install/#toc-yarn-install-production-true-false yep. --production=true or NODE_ENV=production makes yarn ignores devDependencies.

[klodoma]

Any progress on this one?

I have the situation that I need to deploy apps on the server and update there the node_modules. If something is being deleted in node_modules folder(by any reasons) then yarn install doesn't fix it. npm install on the other hand does fill in the missing directories.

I've tried several solutions proposed here, but not all are working well.

For me, it seems that yarn install --skip-integrity-check kind of works, but it's fairly slow in contrast to plain npm install

[dsaw]

I am facing the same issue with transitive dependencies. One component is getting installed fine and yarn.lock is updating too, but when checking node_modules some of its indirect dependencies are not present at all. This is what happens when I run build (yarn start)

[scorpion35]

People asking the same question 4 years later, even with solid repro? 🙊 😅

I ran into same issue as well. Any news on this please? Here's my repro (same as mentioned in the OP's repo)

...
"dependencies": {
  "project_folder": "file:submodules/some_submodule",

   ...

1. Add a dependency for a submodule using file:... syntax in package.json
2. Run yarn install. Notice the submodule is properly copied to node_modules/project_folder location
3. Remove the folder - rm -rf node_modules/project_folder
4. Run yarn install again, and notice the project_folder dependency is not restored under node_modules/project_folder location.

Let me know if you need more information please. I am on yarn 1.22.10