[BUG] npm ci still does not work with overrides correctly

[jchiem]

Is there an existing issue for this?

• [X] I have searched the existing issues

This issue is related to: https://github.com/npm/cli/issues/4684, https://github.com/npm/cli/issues/4732, https://github.com/npm/cli/issues/4732

This issue exists in the latest npm version

• [X] I am using the latest npm

Current Behavior

The release of [email protected] has made a fix for the aforementioned issues, but the same error occurs in my example repository on [email protected], and for any version above 8.6.0 for that matter, I tested #4732 with 8.7.0 and the error presented in that specific issue is resolved with any newer version of npm.

The bug is that if npm ci is used with certain dependencies and override combination the following error occurs:

npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.

This happens with any version of npm above 8.5.5, if 8.5.5 is used the error does not occur.

Furthermore, if you follow the instruction provided in the error to run npm install the security vulnerabilities that were originally resolved with the "overrides" are restored and the package-lock is updated. If you run npm ci after that the overrides are still not taken into consideration, so the overrides have been ignored.

I've set up a repository where the behavior is described in more detail which is listed under Steps To Reproduce

Expected Behavior

With npm ci I expect that the latest version of npm is able to use npm ci with the defined "overrides" without errors.

With npm install or npm install --package-lock-only I expect that the "overrides" configured are still respected and the package-lock is not updated and ignoring the overrides.

Steps To Reproduce

To test this do the following:

• Have npm 8.6.0 or greater installed (bug occurs for any version above 8.5.5)

1. Clone the repository git clone [email protected]:jchiem/npm-overrides-bug.git
2. Run npm ci
3. Run npm install or npm install --package-lock-only

For Step 2 you should get an error like:

For Step 3, it results in no errors, but the package-lock is updated and the node modules installed were done without overrides since security vulnerabilities that should have been resolved using the "overrides" in package.json are no longer 0.

If you run npm ci at this point, the installed packages retain the vulnerabilities from Step 3's npm install and the `"overrides" are ignored.

Environment

• npm: version above 8.5.5 (for example, 8.6.0 or 8.10.0)

• Node.js: 16.14.2

• OS Name: mac OS 12.3.1 (Monterey)

• System Model Name: MacBook Pro (16-inch, 2019)

• npm config:

; node bin location = /Users/joakim.chiem/.nvm/versions/node/v16.14.2/bin/node
; node version = v16.14.2
; npm local prefix = /Users/joakim.chiem/git/npm-overrides-bug
; npm version = 8.10.0
; cwd = /Users/joakim.chiem/git/npm-overrides-bug
; HOME = /Users/joakim.chiem
; Run `npm config ls -l` to show all defaults.

[darmentrout]

I just upgraded npm from 8.5.0 to 8.12.1 and I'm getting the same errors. Removing package-lock.json, deleting node_modules, and rerunning npm install do not help.

$ npm ci
npm ERR! code EUSAGE
npm ERR!
npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm ERR!
npm ERR! Invalid: lock file's [email protected] does not satisfy [email protected]
npm ERR! Missing: [email protected] from lock file
npm ERR! Missing: [email protected] from lock file
npm ERR! Missing: [email protected] from lock file
npm ERR! Missing: [email protected] from lock file
npm ERR!
npm ERR! Clean install a project
npm ERR!
npm ERR! Usage:
npm ERR! npm ci
npm ERR!
npm ERR! Options:
npm ERR! [--no-audit] [--foreground-scripts] [--ignore-scripts]
npm ERR! [--script-shell <script-shell>]
npm ERR!
npm ERR! aliases: clean-install, ic, install-clean, isntall-clean
npm ERR!
npm ERR! Run "npm help ci" for more info

[darmentrout]

Running npm install twice makes the npm ci error go away. https://github.com/npm/cli/issues/4859#issuecomment-1120018666

[jchiem]

Oh interesting, perhaps this will be solved with #4859 then? 🤔 for now I've just kept using npm 8.5.5 and the package-lock generated with that version, it works with my specific dependencies at least, I'll try out if npm install twice as mentioned in the other issue works for my project as well.

[jchiem]

@darmentrout

could you elaborate how npm install resolves the issue for you? For me, doing rm -rf package-lock.json rm -rf node_modules npm install npm install

Still results in vulnerabilities being installed with the 2nd npm install command, which to me indicates that the npm overrides still do not work correctly even with that method for me at least, using npm 8.12.1

[MRobertEvers]

npm install does not resolve the issue for us. npm >= 8.6 fails with the same errors.

[krasingneves]

I've experienced this with npm v8.11.0 as well

[tvsbrent]

The issue as I've encountered is when I've tried to pass in a range that I'd like to override. For example:

"overrides": {
  "got@<11.8.5": "11.8.5",
  "got@>=12.0.0 <12.1.0": "12.1.0"
}

The above will fail. In looking at the generated package-lock.json, it looks like the dependency that was pulling in the overridden version was not updated. For example:

"package-json": {
  "version": "6.5.0",
  "resolved": "<snip>",
  "integrity": "<snip>",
  "dev": true,
  "requires": {
    "got": "^9.6.0",
    "registry-auth-token": "^4.0.0",
    "registry-url": "^5.0.0",
    "semver": "^6.2.0"
  }
}

However, if I do this in the package.json, it does work:

"overrides": {
  "[email protected]": {
    "got": "11.8.5"
  }
}

The entry in the package-lock.json is the following in that case:

"package-json": {
  "version": "6.5.0",
  "resolved": "<snip>",
  "integrity": "<snip>",
  "dev": true,
  "requires": {
    "got": "11.8.5",
    "registry-auth-token": "^4.0.0",
    "registry-url": "^5.0.0",
    "semver": "^6.2.0"
  }
},

[krasingneves]

I was using

"overrides": {
  "package-to-override": {
    "got": "11.8.5"
  }
}

So, specifying the package but not the version. So I may give this a try

[Joinatic]

The second npm install seems to ignore the overrides, so no solution for me

[bellis-ai]

Do the overrides actually work?

Doing an npm install on this package.json

{
  "dependencies": {
    "better-docs": "^2.7.2"
  },
  "overrides": {
    "lodash": "^4.17.21"
  }
}

Still ends up with an install of [email protected]

[DzianisMakeichyk]

Double npm i still fix the issue. npm v8.6.0

[jchiem]

Double npm i still fix the issue. npm v8.6.0

Double npm i with 8.6.0 does not resolve the issue, it just introduces a new package-lock in which overrides are no longer being followed, please check the package-lock, please provide example of where the package-lock actually follows the override conditions after doing npm i twice if this is actually working for you.

[jchiem]

The problem persists with npm 8.17.0

With npm 8.5.5:

(1 vulnerability has been introduced since this issue for the repo mentioned in the description of this issue: https://github.com/jchiem/npm-overrides-bug)

[jchiem]

The problem persists in npm version 8.19.2

[jchiem]

I've ran some more tests regarding this and found that it seems to be specifically when < is used for the overrides for the condition.

Writing it without the < seems to allow the overrides to be interpreted correctly even with the latest version of npm, I think if it shouldn't work for overrides declared using < sign for the version, then that should be added to the documentation and this issue can be closed, if it should work however, this bug is still relevant for this specific type of override.

[grigorii-merck]

Facing similar issue, specifying version without < does not seem to make any difference.

[pawan-bobbili-ntnx]

is this issue fixed in any latest npm versions ?

[20BBrown14]

is this issue fixed in any latest npm versions ?

I tested [email protected] and [email protected] today and am still seeing this issue that i'm pretty sure is related to the issues brought up here.

[10hendersonm]

I believe my team is seeing this in [email protected]

[m-nathani]

i had the same problem, was overrding @babel/core from lib, after install my lib to my app, it wasnt overriding the @babel/core,

untill i had to remove node_modules and package-lock.json , and run npm i .

on which when i did npm ls @babel/core it showed the correct version.

[petruki]

This works for me (npm 9.5.1). I was fixing a vulnerability introduced by a downstream dependency in angular/localize and I could utilize npm ci with overrides.

Here is how I did:

1. Remove your overrides from the package.json and run a clean npm install
2. Move your overrides back to the package.json and run npm update
3. If you use shrinkwrap, run this after updating

Do not use "$" dollar sign as a version reference from your dependencies or devDependencies. Currently, npm ci cannot interpret when verifying if package-json and lock files are synced.

[gaving]

Switching from legacy yarn back to npm on a large project and getting burned by this one, not particularly sure how to proceed. npm ci still moans about package-lock.json not being in sync even with the previous post solution.

[victorgois]

I am dealing with the same issue. My application stopped running on docker in the google cloud build. Using [email protected] and [email protected]. However, when I add a npm i before the npm ci in the dockerfile, it builds. I do not use any override config in my package.

[netishix]

Still experiencing this issue with [email protected]. Downgrading to [email protected] fixed the problem for me.

[Sweetog]

Still experiencing this issue with [email protected], [email protected], always resolved by downgrading to [email protected]. PLEASE FIX THIS, the issue is absolutely problematic and should not be happening.

I believe it has to be changes in v8.6.0 release that have caused this regression: https://github.com/npm/cli/releases/tag/v8.6.0

[edwincoronado]

Experiencing this issue too.

[Momotoculteur]

I had this issue with nodejs 20.11.1 and npm 10.2.4. Had to double npm install to work...

[alexgomezlf]

We are experiencing the same issue. Using npm install instead of npm ci worked for us, but it is not ideal and the package-lock.json changes every time we run npm install. Our only other option was to downgrade npm to [email protected]

[Sweetog]

@petruki What is the purpose of your reply? I ask, because I don't want anyone to be confused by your comment and think there is any type of work around for any OP here because there is not. This issue is actually ridiculous, it is a fundamental failure. When a user declares overrides and runs npm ci, any version of npm > 8.5.5 borks out.

[petruki]

@Sweetog , oh, sorry my friend, I didn't mean to expose any hack for this issue. I just wanted to share what worked for me and perhaps help someone who cannot wait for a fix. I removed the reply anyway as you may be right. Thanks for the advice. 😉