Trim down npm tarball size

[vladimyr]

Trim down tarball size by leveraging the files field. :tada:

Before

npm notice === Tarball Details === 
npm notice name:          diff-so-fancy                           
npm notice version:       1.3.0                                   
npm notice filename:      diff-so-fancy-1.3.0.tgz                 
npm notice package size:  62.8 kB                                 
npm notice unpacked size: 250.7 kB                                
npm notice shasum:        1559a03fc66cc79c5c8a53cdb2837b732e679d22
npm notice integrity:     sha512-IrMM3X1Avqwa1[...]uV3BuXab2AWbA==
npm notice total files:   82                                      
npm notice 
diff-so-fancy-1.3.0.tgz

---

After

npm notice === Tarball Details === 
npm notice name:          diff-so-fancy                           
npm notice version:       1.3.0                                   
npm notice filename:      diff-so-fancy-1.3.0.tgz                 
npm notice package size:  27.0 kB                                 
npm notice unpacked size: 85.1 kB                                 
npm notice shasum:        db2c37ae6759dec883a7fc1bf6e1a869e29ba0f4
npm notice integrity:     sha512-w2IZsbrGpe/6a[...]cz760H6fJk56Q==
npm notice total files:   10                                      
npm notice 
diff-so-fancy-1.3.0.tgz

[scottchiefbaker]

@stevemao / @paulirish thoughts on this? NPM isn't my area of expertise.

[paulirish]

@vladimyr i'd rather just use an .npmignore file with the sole line of:

test

... trying that out locally it brings the numbers down to roughly the same size.

npm notice package size:  29.8 kB
npm notice unpacked size: 91.7 kB

and in my experience it's easier to maintain a blocklist of extraneous files than keep up to date the list of critical files.

[vladimyr]

Let's say I took the .npmignore approach, here's is what you got:

diff-so-fancy on  package.json [!?] is 📦 v1.3.0 via ⬢ v10.17.0
❯ cat .npmignore && npm pack --dry-run
───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: .npmignore
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ test
───────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────
npm notice
npm notice 📦  [email protected]
npm notice === Tarball Contents ===
npm notice 249B   .gitmodules
npm notice 25.8kB diff-so-fancy
npm notice 34.4kB third_party/build_fatpack/diff-so-fancy
npm notice 1.1kB  LICENSE
npm notice 657B   package.json
npm notice 1.1kB  hacking-and-testing.md
npm notice 610B   history.md
npm notice 1.5kB  pro-tips.md
npm notice 4.2kB  README.md
npm notice 485B   reporting-bugs.md
npm notice 3.2kB  third_party/ansi-reveal/ansi-reveal.pl
npm notice 3.9kB  third_party/build_fatpack/build.pl
npm notice 4.2kB  third_party/term-colors/term-colors.pl
npm notice 7.0kB  lib/DiffHighlight.pm
npm notice 1.3kB  report-bug.sh
npm notice 303B   update-deps.sh
npm notice 1.0kB  .travis.yml
npm notice 233B   appveyor.yml
npm notice 493B   .circleci/config.yml
npm notice === Tarball Details ===
npm notice name:          diff-so-fancy
npm notice version:       1.3.0
npm notice filename:      diff-so-fancy-1.3.0.tgz
npm notice package size:  29.8 kB
npm notice unpacked size: 91.7 kB
npm notice shasum:        860c10eed206e0691c0f485ea7dcb70ccc4a9e6c
npm notice integrity:     sha512-sp6VTUXXA+Hn0[...]K77pxYOQnCETg==
npm notice total files:   19
npm notice
diff-so-fancy-1.3.0.tgz

Now let's compare that with the files allowlist approach:

diff-so-fancy on  package.json is 📦 v1.3.0 via ⬢ v10.17.0
❯ cat package.json | tail -10 && npm pack --dry-run
  },
  "files": [
    "diff-so-fancy",
    "lib",
    "third_party"
  ],
  "scripts": {
    "test": "bats test"
  }
}
npm notice
npm notice 📦  [email protected]
npm notice === Tarball Contents ===
npm notice 25.8kB diff-so-fancy
npm notice 34.4kB third_party/build_fatpack/diff-so-fancy
npm notice 1.1kB  LICENSE
npm notice 725B   package.json
npm notice 610B   history.md
npm notice 4.2kB  README.md
npm notice 3.2kB  third_party/ansi-reveal/ansi-reveal.pl
npm notice 3.9kB  third_party/build_fatpack/build.pl
npm notice 4.2kB  third_party/term-colors/term-colors.pl
npm notice 7.0kB  lib/DiffHighlight.pm
npm notice === Tarball Details ===
npm notice name:          diff-so-fancy
npm notice version:       1.3.0
npm notice filename:      diff-so-fancy-1.3.0.tgz
npm notice package size:  27.0 kB
npm notice unpacked size: 85.1 kB
npm notice shasum:        db2c37ae6759dec883a7fc1bf6e1a869e29ba0f4
npm notice integrity:     sha512-w2IZsbrGpe/6a[...]cz760H6fJk56Q==
npm notice total files:   10
npm notice
diff-so-fancy-1.3.0.tgz

Numbers are roughly the same but file count differs significantly. If you want to achieve the same effect your blocklist would need to be much more exhaustive:

diff-so-fancy on  package.json is 📦 v1.3.0 via ⬢ v10.17.0
❯ cat .npmignore && npm pack --dry-run
───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: .npmignore
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ .git*
   2   │ .circleci/
   3   │ .travis.yml
   4   │ appveyor.yml
   5   │ test
   6   │ *.sh
   7   │ *.md
───────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────
npm notice
npm notice 📦  [email protected]
npm notice === Tarball Contents ===
npm notice 25.8kB diff-so-fancy
npm notice 34.4kB third_party/build_fatpack/diff-so-fancy
npm notice 1.1kB  LICENSE
npm notice 725B   package.json
npm notice 610B   history.md
npm notice 4.2kB  README.md
npm notice 3.2kB  third_party/ansi-reveal/ansi-reveal.pl
npm notice 3.9kB  third_party/build_fatpack/build.pl
npm notice 4.2kB  third_party/term-colors/term-colors.pl
npm notice 7.0kB  lib/DiffHighlight.pm
npm notice === Tarball Details ===
npm notice name:          diff-so-fancy
npm notice version:       1.3.0
npm notice filename:      diff-so-fancy-1.3.0.tgz
npm notice package size:  27.0 kB
npm notice unpacked size: 85.1 kB
npm notice shasum:        db2c37ae6759dec883a7fc1bf6e1a869e29ba0f4
npm notice integrity:     sha512-w2IZsbrGpe/6a[...]cz760H6fJk56Q==
npm notice total files:   10
npm notice
diff-so-fancy-1.3.0.tgz

Which brings me to:

and in my experience it's easier to maintain a blocklist of extraneous files than keep up to date the list of critical files.

With all due respect, I would argue that maintaining an allowlist of 3 entries:

diff-so-fancy
lib
third_party

is way easier than dealing with/reasoning about the following blocklist:

.git*
.circleci/
.travis.yml 
appveyor.yml
test
*.sh
*.md

especially because .npmignore makes you think that readme and changelog will get omitted while in fact, they are present inside dist tarball no matter what you put inside your blocklist.

In general, using .npmignore is always a bad idea because it assumes a bunch of hidden defaults plus even npm docs is suggesting to consider the files field instead.

Also, there is borderline relevant but nevertheless excellent writeup about npm CLI and .gitignore/.npmignore interplay: https://medium.com/@jdxcode/for-the-love-of-god-dont-use-npmignore-f93c08909d8d

And sole existence of this shinny thing → https://github.com/inikulin/dmn proves my point that it is extremely hard to get .npmignore right.

OFC you are more than welcome to dismiss my bikeshedding about a few files/KBs that don't do any harm anyway but my point here isn't to revert the climate change with those disk savings I'm just trying to promote the right ways of doing npm publishing. And the reason I'm thinking it is important is that my gut tells me I'm not the only one out there who owes a significant part of his personal success to watching and replicating @paulirish's way of doing things (kudos goes to the rest of the crew too!) 😉

[paulirish]

Numbers are roughly the same but file count differs significantly.

True.... but is there a problem? If you're trying to reduce the number of files in your node_modules folders, I can't imagine diff-so-fancy is a top culprit...

With all due respect, I would argue that maintaining an allowlist of 3 entries: is way easier than dealing with/reasoning about the following blocklist:

Yes... your blocklist ain't great. Which is why I'm pitching a very simple, hard-to-fuck-up, one-line blocklist :) We exclude tests, which obviously is never gonna change. From a maintenance risk POV, this is absolutely more safe than your 3-line allowlist. This project's history did change both the structure of third_party and of lib. But test has always been in the same place.

And sole existence of this shinny thing → inikulin/dmn proves my point that it is extremely hard to get .npmignore right.

TBH, what I take away from dmn is that npmignore is the easier mechanism to remove package bloat. 😁

plus even npm docs is suggesting to consider the files field instead.

i'm really glad they do! files is definitely useful and it'd be weird if the two didn't crosslink eachother's docs.

and here i am "considering"! and i do not "find .npmignore to be a maintenance headache". 👺 not on this project.
i'm sure there are other of my projects where i'd agree the DX of files wins.

oh, and i appreciate you taking the time to pull in all those resources as links. that's always a good thing!

three more thoughts..

1. i'm so curious why you chose to sort the package.json keys? Is sort-package-json pretty canonical these days? (first i've heard of it).. I appreciate you separating it into its own commit though. 💅
2. I'm not using this as an argument, but.. at least we can have comments in an npmignore file. i'm super impatient for package.json moving to json5.
3. completely unrelated... but .. since we're talking... when it comes to your developer tools.. would you prefer they represent network file sizes as 1024 bytes == 1KiB or as 1000 bytes == 1KB? 🤓

[OJFord]

network file sizes as 1024 bytes == 1KiB or as 1000 bytes == 1KB?

As long as they get b/ib/B/iB right for whatever they chose, they come out ahead in my experience!

[scottchiefbaker]

With the release of v1.4.1 I'm strongly encouraging packages to use the fatpacked version of d-s-f. It's included in the binary section of the release, and/or it's easily built using the third_party/build_fatpack/ script.

Really you only need the diff-so-fancy script itself, and whatever documentation you want to include. Should be pretty small.

[scottchiefbaker]

Can this either be updated or closed. Packagers should use the fatpacked version, which is currently 39k.

[scottchiefbaker]

I'm marking this closed as packagers should be using the fatpacked version, which is a single file and much smaller.