Plans for npm 9

[MylesBorins]

Hey all. The npm team is hard at work on getting npm 9 together with the hope that we could have the version of npm included in Node.js 18 prior to LTS. A big part of wanting to do this is to ensure we are in a good place for maintenance of the CLI with 18 LTS being supported until 2025.

We are intending to start cutting release candidates for npm 9 in the next week or two. Is there a certain amount of time prior to the October 25th LTS date that folks would want to see us release a GA of npm 9 to feel comfortable including it in node.js 18?

[mhdawson]

Are there no breaking changes in npm 9 even though its a major? I think by default our policy is no breaking change once we've cut the major.

[MylesBorins]

@mhdawson the breaking changes can be seen in the link I shared above

https://github.com/npm/statusboard/issues/443

There are a number of them, but none that seem to be extremely disruptive to folks for the mainstay use case of "publish" and "install".

We have prior art of updating npm Semver-Majors during a release line. Node.js 16 was updated from npm 7 -> npm 8 while it was current in v16.11.0 in early october of 2021 (a similar timeline to now). It included a breaking change of dropping support to the require('npm') interface, changing support matrix, and updating dependencies to drop support for node 10.

From a stability stand point I think it is the right thing to do for the ecosystem to ensure that the latest Node.js LTS is using a version of npm that will be getting regular bug fixes and features.

[MylesBorins]

as a caveat perhaps we need to have a broader conversation about versioning of npm in node.js, and our release schedules to try and come up with an agreement for future LTS + Majors to avoid needing to revisit this conversation each year (or major) 😓

[mhdawson]

@nodejs/release I think an opinion from the release team would be good and adding @nodejs/tsc in case as well for awareness.

[mcollina]

I find it unfortunate that npm decided to ship a major release the same month Node hit lts.

I think it's time we have this conversation of aligning the releases timelines. I think the best thing is for npm to release a major in March or April, so that there is plenty of time for all bugs to be fixed before then.

My 2 cents: there is a long list of breaking changes that still need to be written on user configs etc likely to cause havoc without some proper testing. I don't want to test in LTS.

Given all the above, I’m -1 in shipping Npm v9 in a Node LTS release.

[mhdawson]

@mcollina I like the suggestion of aligning LTS releases between Node and npm. If there was a major npm release in March or April that would be a great way to get a recent major into the Node.js version that will be the next LTS along with enough backing time to minimize impact/surprises for the ecosystem.

@MylesBorins I know that is probably a separate discussion from the current ask, but do you think that would be possibe in the future?

[MylesBorins]

I'm 100% on board with moving forward with improving coordination for future releases.

For the sake of Node.js 18 and thinking about long term stability and feature compatibility I think it is critical we land npm 9. We are putting together an explanation of the changes and I'd be willing to push back on the npm team to delay some of these changes to a future major release if it would allow us to be aligned on landing npm 9 in node.js 18

[aduh95]

I think ideally npm versions should not be coupled with Node.js at all, that would set npm free from doing releases however they please, and Node.js could release LTS versions without having to worry what npm is planning. At some point, we need to ask ourselves if the benefits of bundling a precise version of npm outweigh the downsides. Also, it gives npm an unfair advantage over other package manager clients, and maintains confusion between the Node.js project and the npm one.

I personally would like to see the TSC decide to wind down the bundling of npm with Node.js releases, as to me it appears as the most reasonable and fairest solution.

[kapouer]

The open source point of view totally seconds that. Having npm bundled in nodejs never made sense (from that point of view).

[MylesBorins]

Summary of npm9 Breaking Changes

Command Removals

bin

This command is not accurate, and having it available encourages users to rely on an incomplete implementation.

set-script

This has been deprecated in favor of the npm pkg set command. birthday

birthday

This is an easter egg and has been flagged by automated malware tooling.

Breaking Changes to Commands

adduser

This command will no longer be an alias for login.

Config Removals

Bare _auth, _authToken, username, password, email

Bare configs like _auth are confusing and dangerous and will be removed. Instead they will be scoped to a specific registry.

sso-type and auth-type (except for legacy and webauthn)

This removes support for deprecated auth types.

loglevel=timing

This loglevel will be removed and instead timing info will be displayed only with --timing only which will decouple logging from generating timing information.

local, global-style, legacy-bundling

These configs will be replaced with a consolidated --strategy config with the above as values. Default Config Changes lockfile-version=3 npm install will create a v3 lockfile by default when creating a new lockfile. These lockfiles are incompatible with npm6.

auth-type=web

Make auth-type=web the default for login and publish for the public registry only.

install-links=true

npm install will install linked dependencies by default

init-private=true

npm init will create a package.json with private: true set by default.

access=public

npm publish will default to access=public for all packages. This is currently the default for non-scoped packages, so this change will now apply to scoped packages as well.

HEAD instead of master for opening links

Commands like npm docs will default to HEAD instead of master as the default ref for opening links.

Other Changes

Error when setting deprecated configs

npm9 will not run npm config set when key is a deprecated config item.

packlist ignore patterns

Two changes here:

Remove shrinkwrap.json as an unignorable file.

npm9 will standardize and document the order of operations of ignoring files from any command that produces a tarball. This is a subtle change but is being treated as breaking change since it will affect the files included in a tarball when multiple ignore types are used. npm/npm-packlist#88 has a full explanation of the specifics of this change.

Folder Permissions

npm will no longer attempt to set folder permissions. This is a complex issue and npm/rfcs#546 has the most up to date information about the specific changes.

Timing Log Files

With timing: true npm will write a process specific timing file similar to how it treats log files instead of appending a newline delimited json file.

engines: ^14.17.0 || ^16.0.0 || >=18.0.0

npm9 will set engines to the above. npm9 will also start to use node@14 specific features and fail when run on npm@12.

stdout and stderr Output

npm9 will standardize output between stdout and stderr. Errors will now be output on stdout when they can possibly be used programmatically by consumers.

Remove Progress Bar

npm9 will remove the progress bar to make it less error prone to receive user input during long running commands. This is not strictly a breaking change since npm does not consider display features to be part of its semver contract, but this change is being made now since it will be a visual change.

Normalize x, x@, & x@*

These specs are all the same but treated and resolved slightly differently. In npm9 they will all be resolved the same. This is a subtle change and is being included in npm9 in order to err on the side of caution. See npm/npm-package-arg#45 for the specific changes in the resolved values.

Remove path support from explain

Only allow for npm explain <package> instead of paths

[MylesBorins]

I think it would make sense to have a separate conversation if we want to talk about unbundling npm from Node.js .

We have to think about the usability concerns of permanently decoupling the two, and I think it would be good to discuss independently

[MylesBorins]

Quick ping on this to @nodejs/releasers and @nodejs/tsc

[BethGriggs]

Would landing npm@9 in Node.js 19, waiting a couple of months for feedback, and then backporting the update to Node.js 18 during LTS be an alternative option?

I personally think having a good amount of baking time for us to asses impact would be preferable to us rushing to land it in advance of the LTS transition the same month. (As we'll be breaking semver rules anyway, maybe we should do it once we've had more feedback from Node.js 19/Current so we can advise of any anticipated impact?)

[MylesBorins]

Waiting to update is definitely a reasonable solution, I'd still like us to review the proposed breaking changes and flag which changes are going to be "deal breakers" in advance.

[ruyadorno]

I'm a +1 from the point of view of a releaser. It's going to make it simpler to manage the v18.x release line moving forward, given that there's always extra work involved when managing multiple release lines of npm.

Personally, what I'm more concerned about is limiting the impact of breaking changes. With that in mind the more we can provide configurations that allows for users to opt-in to backwards compatibility the better. Would the npm team be willing to compromise on some of these intended changes (I believe removing bin and tweaking adduser cmds might be somewhat disruptive) in order to reduce any risk involved?

[jasnell]

I'd certainly prefer to wait on landing it in 18.

[richardlau]

If we have to put npm 9 into Node.js 18 then I would prefer to wait and I would also like that as far as possible breaking changes be kept to a minimum (ideally avoided).

We do need to have the separate discussion about aligning releases as the Node.js LTS transition is supposed to be a production-ready stability marker and is not intended to be a point that breaking changes can drop in before (that point is the initial semver major *.0.0 Node.js release).

[MylesBorins]

Would the npm team be willing to compromise on some of these intended changes (I believe removing bin and tweaking adduser cmds might be somewhat disruptive) in order to reduce any risk involved?

We are 100% willing to compromise on any + all proposed breaking changes. I'll surface those two in particular to the team.

I'm personally less concerned about adduser as the existing behavior is confusing and we've had a warning in place. For some changes we can also consider landing breaking changes in v9.0.0, gathering feedback, and reverting if they prove to be disruptive.

Thoughts?

[MylesBorins]

Would it make sense to coordinate a call between the release team + the npm team to review the breaking changes or do folks feel like we can handle this async? I can share a doc for comments if that would work.

[ruyadorno]

Would landing npm@9 in Node.js 19, waiting a couple of months for feedback, and then backporting the update to Node.js 18 during LTS be an alternative option?

I prefer the original proposition from Myles given that if there's any breaking change (even if they're kept to a minimal as described above) it's safer (and more clearly signalled to users) to land that during the Current release line rather than during LTS.

Would it make sense to coordinate a call between the release team + the npm team to review the breaking changes or do folks feel like we can handle this async? I can share a doc for comments if that would work.

I would be happy to help coordinate a call if there's enough interest and agreement.

[BethGriggs]

I prefer the original proposition from Myles given that if there's any breaking change (even if they're kept to a minimal as described above) it's safer (and more clearly signalled to users) to land that during the Current release line rather than during LTS.

I disagree with the assessment that it's safer to drop a significant semver-major change into the release line within a few weeks of the LTS date. The expectation from semver is that there are no breaking changes during either Current or LTS. So, I think we'd honour our stability guarantees of LTS more by landing npm@9 in Node.js 19 so we can properly assess the end-user impact and know what workarounds, etc. we need to provide as information to end-users. Our group's assessment of the breaking changes will be good and useful, but there will be inevitable edge cases exposed from real-world usage.

I also am concerned about the timelines of trying to rush it into Node.js 18 considering the expected GA date of October 2022 for npm@9. The LTS transition date is set for 2022-10-25 - by policy, that release will only contain the release commit to promote it to LTS. So going by our schedule, the last release for npm@9 to make will be the one scheduled on 2022-10-04. Is the GA of npm@9 going to be before 2022-10-04? We also try to keep the standard baking time between the last 'Current' release and the LTS transition - so there wouldn't be much, if any, time for an interim release to patch or revert any of the breaking changes before LTS.

[MylesBorins]

In general I think that the case I'm trying to make is that upgrading from npm 8 -> npm 9 should not be considered Semver-Major in the eyes on the Node.js project. The changes we want to land should be considered non-breaking for the primary use cases that npm has for Node.js including installing and publishing packages as well as running scripts.

Quite a few of the breaking changes are being done to improve security posture, others are to clean up rough edge cases and broken APIs.

With this in mind, I think that it would be reasonable to land the release in Node.js 19, gather feedback to confirm it is indeed "non-breaking", if there are things that are breaking we can revert or come up with a reasonable solution. Once we reach that point we can backport the release to npm 18 as a Semver-Minor, which based on discussion should be a bit easier to do 😉.

We are cutting the first r.c. of 9 this week (unless computers / automations want to fight us), and we can start integration testing almost immediately.

The most important thing for us right now is to identify any of the "breaking for npm" changes we want to land that folks would feel would be "breaking for node.js" so we can avoid setting ourselves up for failure.

From a stability + maintainability perspective I think it is incredibly important that we get npm 9 into Node.js 18, this is imho the primary goal of LTS.

I also want to be very clear that once we navigate the npm 9 situation the npm team will work closely with the Node.js team to ensure we have a coordinated release process moving forward that can scale to the unique maintenance requirements of both projects. I think it is reasonable for us to entertain any and all solutions and I look forward to the discussion.

[nlf]

i think one significant thing we can do here to help us avoid the concern about breaking things is to define what exactly the node team would like to ensure doesn't break. can we work together to determine a set of commands/options/behaviors that should always work? if so, we can codify them as tests in the npm cli itself and we can be certain we don't break that (hopefully) minimal feature set. changes to those tests would give us a strong signal of when a semver-major of npm happens to also require a semver-major of node as well.

[mcollina]

Here is a non-exclusive list:

1. a package-lock.json file from the previous version of npm is left untouched after npm install
2. a package-lock.json file from the previous version of npm is being kept editable by the previous version of npm shipped with node at x.0.0 mark.
3. the credentials defined within .npmrc keep working across releases for both install and publish
4. Installation instructions in READMEs keep working, e.g. npm i -g something is still there.
5. generically, CI keeps being green...

[GeoffreyBooth]

1. a package-lock.json file from the previous version of npm is left untouched after npm install

Even on the same version of npm, successive runs of npm install often cause a package-lock.json to be modified. (This was my feature request at the collaborator summit, a command that does npm ci‘s “install from lockfile” without deleting node_modules first.)

[MylesBorins]

Even on the same version of npm, successive runs of npm install often cause a package-lock.json to be modified.

@GeoffreyBooth I don't believe that should be the case anymore. If there are no conflicts within your package-lock.json running npm install should respect the package-lock and not modify either the package-lock or the package.json

Here is a non-exclusive list

@mcollina this is a great starting point. We'll review the changes with this in mind. We can also think a bit about how we can turn some of these requirements into tests we can have in the Node.js tree!

[mhdawson]

I like the suggestion from @nlf If we had a matrix of npm commands with a checkbox of breaking/non breaking for node-js that would help to discuss as well as to document for the community what they should expect.

[nlf]

Here is a non-exclusive list:

1. a package-lock.json file from the previous version of npm is left untouched after npm install

this is the case if the package-lock.json is already lockfileVersion: 2, very old package-lock.json files from npm6 are currently upgraded to a v2 the first time they're consumed in npm7 or greater. npm 6 can still read from these lockfiles, but writing to them from npm6 is likely going to cause some churn the next time the same lockfile is acted on by npm>=7

once a lockfile exists as v2, we will not upgrade it to v3 without being explicitly asked. i think this bar meets this criteria.

2. a package-lock.json file from the previous version of npm is being kept editable by the previous version of npm shipped with node at x.0.0 mark.

npm>=7 has full support for lockfileVersion 2, support for npm6 and all versions of node that ship with npm6 has already ended, so that's our minimum bar. i think we're good on this criteria too.

3. the credentials defined within .npmrc keep working across releases for both install and publish

this one we do have a breaking change that will be taking place in npm9. currently we still allow users to define a top level _authToken (or other auth related configs) without scoping them to a specific registry. this is a security risk since those credentials will be passed to whatever default registry is in use. because of this we have deprecated the use of these top level keys in favor of requiring credentials to be scoped to the specific registry they belong to. while breaking due to the security risk involved in not making this change, i think we should consider this to be a good thing to land.

4. Installation instructions in READMEs keep working, e.g. npm i -g something is still there.

i believe the only significant change here will be the separation of npm login and npm adduser, folks who run npm login and do not have an account will no longer have one created for them, instead they'll be informed they need to run npm adduser or visit the npmjs.com website. since we're at the very least informing the user of next steps and account registration is (hopefully) not being run in non-interactive ways i hope we can call this one acceptable.

5. generically, CI keeps being green...

this one we will definitely maintain

[BethGriggs]

support for npm6 and all versions of node that ship with npm6 has already ended,

@nlf, can I just check this statement? Apologies if I am interpreting it out of context, but isn't npm6 still expected to be supported until the EOL of Node.js 14 in April 2023?

(Feel free to hide this tangential comment/query after)

[mcollina]

this is the case if the package-lock.json is already lockfileVersion: 2, very old package-lock.json files from npm6 are currently upgraded to a v2 the first time they're consumed in npm7 or greater. npm 6 can still read from these lockfiles, but writing to them from npm6 is likely going to cause some churn the next time the same lockfile is acted on by npm>=7

once a lockfile exists as v2, we will not upgrade it to v3 without being explicitly asked. i think this bar meets this criteria.

I consider the automatic lockfile migration one of the major mistakes we made in upgrading to npm v6 during a LTS cycle. Avoid forcing the upgrade to the lockfile format v3 is great.

this one we do have a breaking change that will be taking place in npm9. currently we still allow users to define a top level _authToken (or other auth related configs) without scoping them to a specific registry. this is a security risk since those credentials will be passed to whatever default registry is in use. because of this we have deprecated the use of these top level keys in favor of requiring credentials to be scoped to the specific registry they belong to. while breaking due to the security risk involved in not making this change, i think we should consider this to be a good thing to land.

I agree this is a good thing to land, however, I'm concerned with all those folks that rely on this. Have you got statistics on how widespread this usage is?