Accept package-lock.json files in v2 format

[ffendt]

npm since v7 uses a new lockfile format for storing dependencies in package-lock.json.

Current behavior when the CLI is used with the new format: No dependencies are found and thus no licenses are checked.

Expected behavior: CLI can also parse the new format and check the licenses.

Main changes in the lockfile format as far as I can overlook it:

1. dependencies has been renamed to packages
2. There is a dependency describing the dependencies of the library itself (dependency with key "")
3. All other dependency names start with prefix "node_modules/"

As an example an old format lockfile:

{
	"name": "bumlux",
	"version": "1.0.0",
	"lockfileVersion": 1,
	"requires": true,
	"dependencies": {
		"@babel/code-frame": {
			"version": "7.5.5",
			"resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.5.5.tgz",
			"integrity": "sha512-27d4lZoomVyo51VegxI20xZPuSHusqbQag/ztrBC7wegWoQ1nLREPVSKSW8byhTlzTKyNE4ifaTA6lCp7JjpFw==",
			"dev": true,
			"requires": {
				"@babel/highlight": "^7.0.0"
			}
		}
}

and the same one in v2:

{
  "name": "bumlux",
  "version": "1.0.0",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "name": "bumlux",
      "version": "2.0.0",
      "license": "SEE LICENSE IN LICENSE",
      "dependencies": {},
      "devDependencies": {
        "@babel/code-frame": "7.5.5"
      }
    },
    "node_modules/@babel/code-frame": {
      "version": "7.12.13",
      "resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.12.13.tgz",
      "integrity": "sha512-HV1Cm0Q3ZrpCR93tkWOYiuYIgLxZXZFVG2VgK+MBWjUqZTundupbfx2aXarXuw5Ko5aMcjtJgbSs4vUGBS5v6g==",
      "dev": true,
      "dependencies": {
        "@babel/highlight": "^7.12.13"
      }
    }
}

[waynebeaton]

The current implementation is something of a quick-and-dirty just-what-we-need convenience hack. Extending it to support multiple versions shouldn't be too hard (so we should do it), but I'm concerned that there may be some subtlety in parsing the file that we're missing.

It would be better to have npm do the parsing for us (even is only as a backup for cases where the hypothetical subtlety that I'm concerned about bites us).

AFAICT, the npm-ls command is the magic that gives us the dependency list, but I don't see a configuration that generates a simple flat file. The parseable option, even when combined with the long option doesn't seem to give us something that is any more workable than just the straight hierarchy list.

AFAICT, the simplest get-npm-to-do-it solution is this:

npm ls --all | grep -Poh "[\w\-\/]+@\d+(?:\.\d+){2}" | sort | uniq

and then pipe the result into the tool.

Do you have any further insight @ffendt ?

Can you provide me a link to the package-lock.json file for your project so that I can test with it?

[waynebeaton]

I've been looking a bit harder at the file format.

There's a "dev" option; when this is set to true, the library is required only at development time (i.e., it is specifically not required at runtime). I'm thinking that we can (at least optionally) skip these, or try to otherwise use the feature to identify "works with" dependencies.

[waynebeaton]

Rudimentary support for v2 has been added with 448b95bd9a025ef5dea069b4cc2bdaf4ed0483f1

[ffendt]

Thanks a lot for the work you've already put into this. Sadly I don't have further insights as I just stumbled upon this. Afaik, the dev option also was there for files in v1 format.

In our CI we're splitting the existing package-lock.json files at the dev option and run the dash-licenses tool once for the lockfile with only prod dependencies and once for the lockfile with only dev dependencies. You can find one of the bigger lockfiles we're running on in ditto-clients/javascript/lib/node/package-lock.json. (Side note: Sadly ClearlyDefined also times out sometimes for this amount of packages).

I'd really like to test your enhancements, but I'm not sure when I'll get to this as the next two weeks are already quite packed.