[RRFC] Custom Dependency Lists

[cawa-93]

Motivation ("The Why")

It is very lacking to separate all dependencies to an unlimited number of lists and, most importantly, to establish dependencies of these lists independently.

This is primarily important for CI / CD. We could shorten work time if you install not all dependencies, but only those necessary for the current task. Now, in order to set dependencies and guarantee the correspondence of their versions and trees use the npm ci command.

Unfortunately, this command downloads absolutely all dependencies. But in many cases some subset needed.

Example

Imagine that we have several workflows:

• Linter.
• Types checking.
• Testing.
• Deploy.

Now, each of these must call npm ci and download a full dependencies tree. But it is obvious that only a subset of this tree is required at each workflow.

How

Current Behaviour

At a time, there is no possibility using npm ci to install a subset of dependencies. For these purposes, you have to use npm install.

Desired Behaviour

It would be possible to allow package.json (or introduce a separate file exclusively for a dependency list) any number of custom lists:

{
  "customDependencyLists": {
    "lint": {
      "eslint": "^0.0.0"
    },
    "typechecking": {
      "typescript": "^0.0.0"
    },
    "deploy": {
      "typescript": "^0.0.0",
      "deploy-tools": "^0.0.0"
    }
  }
}

In the package-lock.json, you can store in each dependence of the lists of lists to which it belongs.

{
"node_modules/typescript": {
      "version": "4.2.3",
      "resolved": "...",
      "integrity": "...",
      "dev": true,
      "bin": {
        "tsc": "bin/tsc",
        "tsserver": "bin/tsserver"
      },
      "engines": {
        "node": ">=4.2.0"
      },
      "lists": ["typechecking", "deploy"]
    }
}

And add a special parameter to specify which lists of dependencies must be installed:

npm ci --lists=lint

npm ci --lists="lint,typechecking"

References

• Related to https://github.com/npm/npm/issues/10395

[ljharb]

(see also: https://github.com/npm/feedback/discussions/295, https://github.com/npm/feedback/discussions/57)

[sandstrom]

See also: https://github.com/npm/rfcs/issues/592

Same overall theme, but different ideas around package.json key naming (I'm biased but I'd say my ideas for json structure may have some advantages).

[philippefutureboy]

Hi there!

I would like to suggest to use a syntax similar to that used by poetry in the python ecosystem:

package.json:

{
  "dependencies": {},
  "dependencies.group.ci": {},
  "dependencies.group.dev": {},
  "dependencies.group.dist": {},
  "dependencies.group.test": {}
}

with the --group flag for npm install:

npm install --group=dev npm install --group=dev,ci npm install --group dev ci

I think it's a better approach and syntax as it:

1. Is really clear, concise, and explicit
2. Does not require custom flags (e.g. npm install --{my-list-name}) like in some of the comments of https://github.com/npm/npm/issues/10395
3. Does not require long flag args, e.g. npm install --list=devDepedencies or some implicit parsing e.g. npm install --list=my-list implied from myListDependencies
4. Is alphabetically sortable in the package.json ({group}Depedencies is not)
5. Has a flat structure in package.json, which improves readability

If you want some more context, how this was implemented for the python ecosystem, I recommend the following reads:

• python-poetry/poetry: Support dependencies groups
• Python Enhancement Proposal (PEP) for Dependency Groups in pyproject.toml

Cheers!

[ljharb]

The only way you could introduce this (for runtime deps, at least) without breaking everyone using "not the latest" installation tool would be if everything still sits in "dependencies", but the groups are instead arrays of dependency names.

[philippefutureboy]

Thanks for your comment! I'm a bit confused as to how this would be a breaking change; Can't this be an optional feature with a deprecation warning if not used?

The dependencies and devDependencies properties could be kept intact; but starting at {some future version} then this could be made mandatory. Additionally, a property could be added to package.json to enforce the optional feature, similarly to type was used in the past to migrate from commonjs to es modules (alternatively a flags section could be added to that effect, cause that's effectively a feature flag)

Does that seem like a reasonable approach?

[ljharb]

@philippefutureboy if a package removes something from dependencies to add it to a group, then any tool that doesn't understand groups (at the moment, every version of every tool) wouldn't install the proper things. Thus, it could never truly become mandatory.

[philippefutureboy]

Huh, so it's some kind of grey transition zone where tools can't rely on the new format because there's not enough adoption? Is the issue primarily because the new feature has an overlap on the property dependencies? If so would using group.dependencies.?{group-name} instead of dependencies.?group.{group-name} fix the issue?

Honestly I'm really not knowledgeable on how the ecosystem works when it comes to rolling out tooling upgrades like these, so I'm not too sure how to modify my proposition in a way that makes sense 😅

[ljharb]

Sorry if i wasn't being clear :-) what i mean is, if ever, 100% of the runtime dependencies aren't present in the literal key dependencies, then silently incorrect behavior is likely for tools that lack support for whatever it's replacement is. Therefore, I think the cost of replacing it is never going to be worth it, and I'd suggest finding an approach that is additive, and thus non-breaking.

This concern does not exist for devDependencies, ofc, because only runtime and peer dependencies are "viral" in this manner (they affect consumers)

[philippefutureboy]

Aaaah OK, so basically this is a variant of the browser version issue, right? A package can never migrate from dependencies to group.dependencies since we can never assume that there will not be an outdated npm version somewhere that will only install dependencies.

So the solution should be dependencies + group.dependencies.{group-name} where each {group-name} is optional/additive to the core runtime dependencies defined in dependencies. Did I get that right?

[ljharb]

Yes, that's the modification I'm suggesting to make it even a viable thing to consider :-)

[philippefutureboy]

That works with me :)

Summary of the proposition:

package.json:

{
  "dependencies": {},
  "group.dependencies.ci": {},
  "group.dependencies.dev": {},
  "dependencies.group.test": {},
  "dependencies.group.{group-name}": {},
}

with the --group flag for npm install:

npm install - for installing dependencies npm install --group=dev - for installing group.dependencies.dev npm install --group=dev,ci - for installing group.dependencies.dev and group.dependencies.ci

where

• dependencies: Contains necessary/runtime public dependencies
• group.dependencies.{group-name}: Contains optional dependencies that are relevant to the package, privately.

👍

[ljharb]

: [], not : {}, otherwise yes.

[sandstrom]

I'm not against this idea, but just to throw alternatives into the mix, with regards to naming it could also be something along these lines:

• dependencies -> production group [existing group]
• devDependencies -> dev group [existing group]
• lintingDependencies -> linting group [example]

The general pattern would be this:

{prefix}Dependencies -> {prefix} group

These groups could then be used with install, e.g. npm install only=linting or npm install only=dev,test.

The vanilla npm install would work exactly like today, and install everything in the dependencies and devDependencies group, so it would remain backwards compatible.

Source: https://github.com/npm/rfcs/issues/592

[ljharb]

The "separate group" pattern works if it's only for dev-time, or for adding extra functionality that's properly tested for at runtime, but if it's going to be for runtime, it still carries the risk that someone will make a package that breaks in unexpected ways.

[philippefutureboy]

@sandstrom That's definitely a valid alternative too! I have a personal preference for the approach used in the python community for the reasons listed in my original post but I understand that the JS ecosystem also has it's own semantics & styles which may be preferred over the python proposed solution 👍