bundling - is it a goal?

[refack]

Continued from https://github.com/nodejs/node/issues/22722#issuecomment-419286805:

IMHO bundle-ability is a goal for general tooling enablement (we can argue on priority later). I believe that single file cohesive packing has many benefits for tools. But at the moment CJS based tools are hard to bundle, and node doesn't provide enough hook point to achieve this.

Perceived benefits:

1. Opacity to IDEs - they should not try to parse/analyse/fix
2. Compact size
3. 0 install - single file, just copy&run
4. Cohesive - can't break it by misplacing parts
5. Maybe faster?
6. Easy to SCM - single file to track, light on FS and on git.

[ljharb]

Bundling is primarily for browsers, and secondarily for published packages to reduce their reachable entry points.

I don’t see the benefit here. Install is a good thing, as is tracking changes per file. Node core has a unique and rare scenario where it commits third party code directly to its repo, which differs from the best practice in the wider ecosystem - it’s conceivable that there could be some benefit to bundling for node, i suppose, but not for many other repos.

[silverwind]

You know, we could also just require users to npm install and drop them from the repo, but I think that may upset a few people used to clone and run. Also, there may be an issue with people forgetting to update their local installations.

[silverwind]

Bundling is primarily for browsers, and secondarily for published packages to reduce their reachable entry points.

I have to raise you yarn which bundles their CLI into a single file, resulting in nearly instantaneous installation and probably at least some gains in runtime performance. I think it's a good fit for heavy CLIs.

[ljharb]

https://github.com/nodejs/tooling/issues/3#issuecomment-419471430 is exactly what I’d expect; why would someone be capable of git clone but not npm install for the initial download?

I don’t think offline development is a particularly compelling use case, but if that’s desired, then I’d suggest making a separate repo that can act as an offline registry for all versions of deps that node has ever relied on - it’d require a bit more setup for the edge case of offline dev, but it’d let the common case mirror the rest of the ecosystem.

[mcollina]

Regardless of node core internals, I think support bundling of CLI tools is a interesting use case. It guarantees a shorter startup for them, as the load time of Node.js grows with the numbers of js files loaded.

Note that the install time is not a good argument, as bundledDependencies can achieve the exact same thing.

[ljharb]

Bundling of CLI tools works fine when there’s static requires; requires to node_modules, core modules, and dynamic requires would just be left untouched. The challenge is programmatically knowing which files you can delete, after.

[benjamingr]

I'd like to see actual benchmarks for tools (like yarn's?) before we make any performance claims about the merits or lack thereof of bundling.

In my experience trying this with bluebird the impact is really marginal.

I can see the value of doing this for simplicity (single file) regardless.

[refack]

I'd like to see actual benchmarks for tools (like yarn's?) before we make any performance claims about the merits or lack thereof of bundling.

I wrote this point (5) with a question mark. I would assume performance will vary on a case by case basis. But there is a placebo effect. Psychologically if I see a tool that is a single file, it makes me feel like it is faster...

[boneskull]

I've abused Rollup to bundle Node.js packages for deployment to cloud function providers... it's a good alternative to uploading a zip file, that's for sure.

Assuming there are clear benefits, and deferring any question of priority, we need to answer this question:

What can Node.js core do--if anything--to make it easier to bundle a Node.js program?

[MylesBorins]

ESM and how we implement loaders will significantly affect how bundlers can work with Node (hopefully in a very positive way). We also have an opportunity with new technologies to re examine this relationship... Whether it be a binary ast, a V8 memory snapshot, or a combination of lots of modules and the PRPL pattern... Lots of exciting things node can do to improve cold boot time.

On Mon, Sep 10, 2018, 1:31 PM Christopher Hiller [email protected] wrote:

I've abused Rollup to bundle Node.js packages for deployment to cloud function providers... it's a good alternative to uploading a zip file, that's for sure.

Assuming there are clear benefits, and deferring any question of priority, we need to answer this question:

What can Node.js core do--if anything--to make it easier to bundle a Node.js program?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/nodejs/tooling/issues/3#issuecomment-419995262, or mute the thread https://github.com/notifications/unsubscribe-auth/AAecVzw6hjWN1874ep-UDhqasFQglYYbks5uZqHbgaJpZM4WeAlt .

[bnb]

@boneskull Functions as a Service are probably an excellent use case for this exact kind of thing.

Also I'm guessing that there could be some interest from individuals who are shipping Docker images, though that can be done relatively easily already with Dockerfiles. Would be curious to see what – if any – improvements we could make to that flow?

There are also tools like pkg that do an interesting job and provide an atypical deployment experience for Node.js applications.

[styfle]

There are quite a few tools like pkg that solve this problem. Check out this list: https://github.com/styfle/awesome-desktop-js#packaging

[boneskull]

@MylesBorins

Whether it be a binary ast, a V8 memory snapshot, or a combination of lots of modules and the PRPL pattern... Lots of exciting things node can do to improve cold boot time.

Speeding up Node.js' boot time seems like its own goal; userland bundlers are just one use case that'd see a major impact.

Re-packaging the runtime (pkg, nexe, etc.) is also its own set of concerns; depending on the strategy, "bundling" JS might not matter.

@bnb I don't understand how people use Node w/ Docker in the real world; maybe you could expand on what you mean?

Maybe it's best at this point to actually ask people working on bundlers what they think. :smile:

[boneskull]

I also wanted to mention "native source map support"

[boneskull]

@lukastaegert (Rollup) and @sokra / @thelarkinn (Webpack)

As a bundling tool author, what features are missing from Node.js that'd make improve your tool, reduce friction for your users, or improve your workflow? Any other notable bugs or pain points?

I don't know of any other tools to bundle Node.js apps. @devongovett does Parcel do this?

[refack]

What can Node.js core do--if anything--to make it easier to bundle a Node.js program?

One approach is hooking the FS into an archive a la electron and asar. Or in more general terms, adding a loader hook to CJS.

[akrawchyk]

does Yarn Plug'n'Play https://github.com/yarnpkg/rfcs/pull/101 change the approach to bundling at all?

[arcanis]

I don't think Plug'n'Play affects that - the static map we generate doesn't have to map to actual file on the filesystem, it also can reference virtual ids.

I've actually experimented with loading files directly through zip archives, and it already works quite well (we initially went with the asar format that @refack mentioned, but the lack of support in third-party tools was annoying; zip had the characteristics we needed and is an ubiquitous file format with production-ready libraries like libzip). Some notes:

• It would be good to separate fs from the module loading (less likely for a faulty plugin to break fs)
• The extension/index.js resolution currently requires to readdir/stat, which is annoying with virtual ids
• The way require.resolve is used (readFile(require.resolve())) might be a problem in this regard

I don’t see the benefit here. Install is a good thing, as is tracking changes per file. Node core has a unique and rare scenario where it commits third party code directly to its repo, which differs from the best practice in the wider ecosystem - it’s conceivable that there could be some benefit to bundling for node, i suppose, but not for many other repos.

Checking-in the dependencies in the repository is the direction we're taking with Yarn. The long-term vision is for Yarn to become a developer tool only; you would run it when you want to add/remove dependencies, or manage your workspaces, etc, but never on production (at least by default; we'll never remove yarn install, but it'll become more of a debug tool than part of the workflow).

This would finally give perfect guarantees that What You See In Dev Is What You Get In Prod, make branch switching much faster, safer, and smarter (since you wouldn't have to run yarn install), and would prevent installs from failing when the npm registry goes down.

So even if top-level bundling isn't something needed strictly speaking, I think some kind of bundling support (possibly an interface without implementation that third-party plugins could then implement) will help a lot to make the ecosystem more mature.