[RRFC] Implement equivalent of Python's PIP's REQUESTED #6463

[paololazzari]

When working with Python, it's possible to determine which package was originally installed by the user by checking for the presence of the REQUESTED file.

For instance:

$ pip install -t package requests
$ ls package/charset_normalizer-3.1.0.dist-info/REQUESTED
ls: package/charset_normalizer-3.1.0.dist-info/REQUESTED: No such file or directory
$ ls package/requests-2.30.0.dist-info/REQUESTED
package/requests-2.30.0.dist-info/REQUESTED

so then one can determine that the requests was installed explicitly, whereas charset_normalizer wasn't.

---

It would be nice if npm did the same thing

[wraithgar]

Isn't this what the package.json file is?

[paololazzari]

For the project I am working on, I don't have a package.json. I only have a node_modules folder

[wraithgar]

npm already stores that information in package.json, package-lock.json and node_modules/.package-lock.json. I don't know if adding it to a fourth location is needed, especially since the files in each node_modules subfolder should only be changed by the modules themselves. What would happen for instance if a project shipped with a REQUESTED file? npm can't add things to package folders, it can only track things like it does now.

[paololazzari]

From node_modules/.package-lock.json, how can I determine which packages were requested?

[paololazzari]

npm can't add things to package folders, it can only track things like it does now.

How about adding a requested property in node_modules/.package-lock.json ?

[ruyadorno]

That info is actually missing from node_modules/.package-lock.json it seems. But even then, having that info already in package.json and package-lock.json sounds like should be sufficient, is there any restriction preventing you from using these files @paololazzari ?

[ljharb]

Put another way, I can understand why you might find yourself with an old project that has node_modules but not package.json - but moving forward (since this new feature would only be available moving forward), when would you ever have a node_modules folder without a package.json somewhere in version control, and likely also a lockfile?

[paololazzari]

Well, in my specific example I am working with AWS lambda functions, which can be deployed just fine with only the node_modules folder. So what I am trying to do for these functions is find out which libraries were requested at install time, to determine what the smallest package.json would be.

To me, this is something that could be beneficial while also being trivial to add (correct me if I'm wrong). Happy to help out

[ljharb]

I'm a bit confused. I thought you were asking to differentiate package.json-installed packages from user-installed packages - there's no way for npm to know which packages, or files, are used at runtime.

Typically what's done for Lambda is npm prune --production prior to shipping everything up to AWS.

[paololazzari]

Here's an example:

$ npm install aws-xray-sdk

$ ls node_modules/
@aws-sdk        aws-xray-sdk          aws-xray-sdk-postgres  pg-int8         postgres-bytea     shimmer      yallist
@types          aws-xray-sdk-core     cls-hooked             pg-protocol     postgres-date      stack-chain
async-hook-jl   aws-xray-sdk-express  emitter-listener       pg-types        postgres-interval  tslib
atomic-batcher  aws-xray-sdk-mysql    lru-cache              postgres-array  semver             xtend

$ cat package.json
{
  "dependencies": {
    "aws-xray-sdk": "^3.5.0"
  }
}

the node_modules folder can be deployed as is (so without package.json).

Now, given node_modules only, how can I find out that it was only aws-xray-sdk that I had installed initially?

[ruyadorno]

I'm also a bit confused. In this example you just described, what difference would make to know that aws-xray-sdk was the package requested by the user?

In other words: what are we accomplishing by making that distinction?

[paololazzari]

what difference would make to know that aws-xray-sdk was the package requested by the user?

I would then know that to recreate the same node_modules directory, I would only need to have aws-xray-sdk in a package.json

[ljharb]

It sounds like you're asking that all of the relevant contents of package.json live in node_modules - but at that point why not just be sure to keep package.json around? That file is required at runtime too, so it's not ever something that's safe to delete.

[paololazzari]

node_modules can be distributable by itself, as shown with the AWS lambda example.

I don't see why node_modules/.package-lock.json couldn't include this information?

[ljharb]

It can, but you wouldn't ever want to - you'd want it to be present in a Lambda also, because it has runtime impact on node.

[paololazzari]

you'd want it to be present in a Lambda also, because it has runtime impact on node.

What do you mean, are you referring to package.json ? As illustrated by my example, package.json isn't needed and node_modules by itself is valid

[ljharb]

Yes, it is needed, because node at runtime uses it. It’s incorrect to ship an app anywhere, including lambda, without package.json.

[ruyadorno]

it may not seem to be needed from a legacy CJS point of view but as @ljharb pointed above, the package.json file holds important info used by the Node.js runtime such as module type and Policies is another one that comes to my mind (there may be more).

[ruyadorno]

I would then know that to recreate the same node_modules directory, I would only need to have aws-xray-sdk in a package.json

it sounds to me you're chasing reproducible installs which is tangential to many other conversations held in this RFC repo today.

It's a VERY elusive topic since there are MANY hidden configs that will affect a npm install in its current architecture, the .npmrc file can hold many config values that will change an install, such as legacy-peer-deps, force, dep types, to mention a few. In the package.json file a npm install will also look into properties such as overrides, workspaces, that can completely change the shape of the node_modules folder produced as a result.

[ljharb]

Even in (not legacy) CJS, node uses it for self-references with the exports and imports fields.