Exponential size increase in NPM rules

[AlvaroAMCode]

Hi,

For weeks now, I have been seeing the size of NPM rules increase dramatically. Week after week, I download the ZIP file all.zip of NPM rules from https://osv-vulnerabilities.storage.googleapis.com/NPM/all.zip and see that the size increases significantly from one week to the next.

Example:

• 4 November 2025: The ZIP file size is 53,505,950 12 Nov 10:30 all.zip (53,505,950 bytes (≈ 53.5 MB))

• 12 November 2025: The size is 142,294,351 bytes (≈ 142.3 MB)

The difference is 142,294,351 − 53,505,950 = 88,788,401 bytes (88,788,401 bytes ≈ 84.68 MB) The new ZIP file is approximately 84.7 MB larger than last week's (1.65 times larger)

I see this week after week. Is it normal for there to be such an increase in NPM rules alone? Could there be duplicates? This situation is disrupting our scanning behaviour, causing errors and extremely high latencies.

Thank you.

[G-Rath]

fwiw @another-rex I was going to mention this at our next catchup - I'd not yet dug into it, but I felt like the NPM database was taking longer to load for osv-detector and it looks like we've started getting OOMs in our daily auditor which I suspect is a result of this.

I've transferred this to osv.dev as that's where the databases come from - @AlvaroAMCode if you wouldn't mind posting that 4th of Nov db somewhere, I can dig into it (I've not got a copy of the older smaller db to compare to 😓)

[G-Rath]

ok so I found an old version of the database locally, so comparing that:

date	files	size
Thu, 18 Sep 2025 20:42:34 GMT	68683	69680 KB
Wed, 12 Nov 2025 18:19:41 GMT	175458	205619 KB

Sadly it looks like this is just a result of new advisories, specifically a massive flood of MAL(ware) advisories:

[another-rex]

Yeah, seems like there was a flood of malicious packages with the goal of generating new malicious packages 🙃

https://www.sonatype.com/blog/devs-flood-npm-with-10000-packages-to-reward-themselves-with-tea-tokens

NPM is amazing.

[michaelkedar]

I can run gcloud storage ls -l --readable-sizes gs://osv-vulnerabilities/npm/all.zip --soft-deleted to see info on old exports, and see the sizes trend upwards starting 10th November:

  55.84MiB  2025-11-10T06:09:42Z  gs://osv-vulnerabilities/npm/all.zip#1762754982634404  metageneration=1
  55.84MiB  2025-11-10T08:40:05Z  gs://osv-vulnerabilities/npm/all.zip#1762764005319960  metageneration=1
  55.84MiB  2025-11-10T11:09:46Z  gs://osv-vulnerabilities/npm/all.zip#1762772986148668  metageneration=1
  55.84MiB  2025-11-10T13:40:16Z  gs://osv-vulnerabilities/npm/all.zip#1762782016634141  metageneration=1
  55.84MiB  2025-11-10T16:27:47Z  gs://osv-vulnerabilities/npm/all.zip#1762792067321676  metageneration=1
  62.06MiB  2025-11-10T19:01:11Z  gs://osv-vulnerabilities/npm/all.zip#1762801271582703  metageneration=1
  62.06MiB  2025-11-10T21:41:42Z  gs://osv-vulnerabilities/npm/all.zip#1762810902697759  metageneration=1
  62.06MiB  2025-11-11T00:25:08Z  gs://osv-vulnerabilities/npm/all.zip#1762820708456091  metageneration=1
  76.58MiB  2025-11-11T03:25:21Z  gs://osv-vulnerabilities/npm/all.zip#1762831521235136  metageneration=1
  96.35MiB  2025-11-11T06:25:31Z  gs://osv-vulnerabilities/npm/all.zip#1762842331397754  metageneration=1
  97.73MiB  2025-11-11T09:10:34Z  gs://osv-vulnerabilities/npm/all.zip#1762852234383580  metageneration=1
  97.73MiB  2025-11-11T11:40:23Z  gs://osv-vulnerabilities/npm/all.zip#1762861223245751  metageneration=1
  97.73MiB  2025-11-11T15:11:07Z  gs://osv-vulnerabilities/npm/all.zip#1762873867708312  metageneration=1
 116.71MiB  2025-11-11T17:42:39Z  gs://osv-vulnerabilities/npm/all.zip#1762882959124097  metageneration=1
 116.71MiB  2025-11-11T20:27:03Z  gs://osv-vulnerabilities/npm/all.zip#1762892823216768  metageneration=1
 129.33MiB  2025-11-11T22:58:33Z  gs://osv-vulnerabilities/npm/all.zip#1762901913678270  metageneration=1
 135.70MiB  2025-11-12T01:41:33Z  gs://osv-vulnerabilities/npm/all.zip#1762911693264512  metageneration=1
 135.73MiB  2025-11-12T04:11:55Z  gs://osv-vulnerabilities/npm/all.zip#1762920715505505  metageneration=1
 146.51MiB  2025-11-12T07:11:46Z  gs://osv-vulnerabilities/npm/all.zip#1762931505947967  metageneration=1
 146.51MiB  2025-11-12T10:26:40Z  gs://osv-vulnerabilities/npm/all.zip#1762943200240634  metageneration=1
 146.51MiB  2025-11-12T13:15:40Z  gs://osv-vulnerabilities/npm/all.zip#1762953340016302  metageneration=1
 146.51MiB  2025-11-12T15:56:49Z  gs://osv-vulnerabilities/npm/all.zip#1762963009373791  metageneration=1
 167.45MiB  2025-11-12T18:43:53Z  gs://osv-vulnerabilities/npm/all.zip#1762973032949208  metageneration=1
 176.13MiB  2025-11-12T21:57:22Z  gs://osv-vulnerabilities/npm/all.zip#1762984642004623  metageneration=1
 180.09MiB  2025-11-13T00:27:39Z  gs://osv-vulnerabilities/npm/all.zip#1762993659823569  metageneration=1

[AlvaroAMCode]

Hi, It is obvious that the natural progression of vulnerability repositories is always to grow, but at this rate, the only way to address the problem is to increase resources to prevent a denial of service.

Do you know of any solutions that I may be overlooking?

[AlvaroAMCode]

The size is now 19,634,685,200 bytes (196.35 MB). I tried removing all the DEV dependencies in package-lock, which reduced the scan time, but it was not enough. I needed to use double the size of memory and CPU to run the scan compared to before.

Is there a solution, a flag or something else that can fix this?

Maybe I'm missing something. Thank you!