syft icon indicating copy to clipboard operation
syft copied to clipboard

feat: Add support for npm lockfile version 3

Open robcresswell opened this issue 3 years ago • 1 comments

This PR adds support for npm lockfile version 3, which drops the "dependencies" key and uses "packages" instead. I've refactored the lockfile parser to make the distinction between the versions explicit rather than the implicit behaviour before. It might be worth splitting into separate files at some point, but the logic is so minimal that I haven't done it.

Some open questions;

  • Does the code look vaguely correct? I don't know Go well at all
  • I can't find good documentation around the presence of the "license" key under the "packages" entries. It seems to be present in the v2 fixture, but I couldn't recreate that locally.
  • Are there other places that I need to add / update tests?

Fixes #1203

robcresswell avatar Sep 15 '22 13:09 robcresswell

Benchmark Test Results

Benchmark results from the latest changes vs base branch
name                                                       old time/op    new time/op    delta
ImagePackageCatalogers/alpmdb-cataloger-2                    11.5ms ± 1%    11.4ms ± 9%    ~     (p=0.190 n=4+5)
ImagePackageCatalogers/ruby-gemspec-cataloger-2              1.32ms ± 1%    1.33ms ± 7%    ~     (p=0.690 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2            3.37ms ± 1%    3.29ms ± 4%    ~     (p=0.151 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2    1.09ms ± 1%    1.07ms ± 1%  -2.01%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         753µs ± 0%     739µs ± 2%  -1.82%  (p=0.008 n=5+5)
ImagePackageCatalogers/node-binary-cataloger-2               6.78µs ± 1%    6.77µs ± 1%    ~     (p=1.000 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                     867µs ± 1%     832µs ± 3%  -4.00%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpm-db-cataloger-2                    1.28ms ± 1%    1.24ms ± 1%  -2.65%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                      14.4ms ± 1%    14.1ms ± 2%  -1.81%  (p=0.016 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                     1.30ms ± 1%    1.23ms ± 0%  -5.17%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2          6.79µs ± 1%    6.90µs ± 3%    ~     (p=0.222 n=5+5)
ImagePackageCatalogers/dotnet-deps-cataloger-2               1.37ms ± 1%    1.41ms ± 3%  +2.79%  (p=0.032 n=5+5)
ImagePackageCatalogers/portage-cataloger-2                    719µs ± 0%     743µs ± 1%  +3.30%  (p=0.008 n=5+5)
ImagePackageCatalogers/sbom-cataloger-2                      4.44ms ± 0%    4.61ms ± 2%  +3.79%  (p=0.008 n=5+5)

name                                                       old alloc/op   new alloc/op   delta
ImagePackageCatalogers/alpmdb-cataloger-2                    5.26MB ± 0%    5.26MB ± 0%    ~     (p=0.841 n=5+5)
ImagePackageCatalogers/ruby-gemspec-cataloger-2               205kB ± 0%     205kB ± 0%    ~     (p=0.095 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2             961kB ± 0%     961kB ± 0%    ~     (p=0.730 n=4+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2     217kB ± 0%     217kB ± 0%    ~     (p=1.000 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         159kB ± 0%     159kB ± 0%    ~     (p=0.310 n=5+5)
ImagePackageCatalogers/node-binary-cataloger-2               1.12kB ± 0%    1.12kB ± 0%    ~     (all equal)
ImagePackageCatalogers/dpkgdb-cataloger-2                     199kB ± 0%     199kB ± 0%    ~     (p=0.548 n=5+5)
ImagePackageCatalogers/rpm-db-cataloger-2                     303kB ± 0%     303kB ± 0%    ~     (p=0.690 n=5+5)
ImagePackageCatalogers/java-cataloger-2                      3.49MB ± 0%    3.49MB ± 0%    ~     (p=0.421 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                     1.26MB ± 0%    1.26MB ± 0%    ~     (p=0.151 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2          1.12kB ± 0%    1.12kB ± 0%    ~     (all equal)
ImagePackageCatalogers/dotnet-deps-cataloger-2                374kB ± 0%     375kB ± 0%  +0.12%  (p=0.008 n=5+5)
ImagePackageCatalogers/portage-cataloger-2                    139kB ± 0%     138kB ± 0%    ~     (p=0.056 n=5+5)
ImagePackageCatalogers/sbom-cataloger-2                       722kB ± 0%     722kB ± 0%  +0.02%  (p=0.008 n=5+5)

name                                                       old allocs/op  new allocs/op  delta
ImagePackageCatalogers/alpmdb-cataloger-2                     85.7k ± 0%     85.7k ± 0%    ~     (p=0.159 n=5+5)
ImagePackageCatalogers/ruby-gemspec-cataloger-2               4.24k ± 0%     4.24k ± 0%    ~     (all equal)
ImagePackageCatalogers/python-package-cataloger-2             16.5k ± 0%     16.5k ± 0%    ~     (p=0.444 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2     5.50k ± 0%     5.50k ± 0%    ~     (p=1.000 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         3.33k ± 0%     3.33k ± 0%    ~     (all equal)
ImagePackageCatalogers/node-binary-cataloger-2                 38.0 ± 0%      38.0 ± 0%    ~     (all equal)
ImagePackageCatalogers/dpkgdb-cataloger-2                     4.46k ± 0%     4.46k ± 0%    ~     (all equal)
ImagePackageCatalogers/rpm-db-cataloger-2                     8.11k ± 0%     8.11k ± 0%    ~     (all equal)
ImagePackageCatalogers/java-cataloger-2                       57.5k ± 0%     57.5k ± 0%    ~     (p=1.000 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                      5.45k ± 0%     5.45k ± 0%    ~     (p=1.000 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2            38.0 ± 0%      38.0 ± 0%    ~     (all equal)
ImagePackageCatalogers/dotnet-deps-cataloger-2                7.12k ± 0%     7.12k ± 0%    ~     (all equal)
ImagePackageCatalogers/portage-cataloger-2                    3.58k ± 0%     3.58k ± 0%    ~     (all equal)
ImagePackageCatalogers/sbom-cataloger-2                       24.4k ± 0%     24.4k ± 0%    ~     (all equal)

github-actions[bot] avatar Sep 15 '22 14:09 github-actions[bot]

Just to get this on the main thread this PR is BLOCKED: https://github.com/npm/cli/issues/5532

spiffcs avatar Oct 19 '22 16:10 spiffcs

made some follow up comments https://github.com/anchore/syft/commit/9d8244bae6aab260d5ddb504bc1de278d0f4bf10

addressing in PR: https://github.com/anchore/syft/pull/1349

Mikcl avatar Nov 18 '22 22:11 Mikcl