bundling

[kapouer]

trafficstars

Hi, long time no talk !

Here are some notes about bundling

Command: npm2deb bundle ...

where either has no version, in which case the latest satisfying package.json is used (this is the preferred form), or is written as mymodule@myversion.

Q what is bundling A we want to be able to get the upstream tarball of a node module, the upstream tarballs of a specific list of sub-modules, extract each one of those sub-modules in node_modules/ and repack a new upstream tarball

Q is there any difficulty of doing so ? A if all modules have a "repository" field in their package.json, and if that repository has tagged releases, it's easy.

Q what happens when a sub-module has a more recent version available ? A bundling is a modification of the upstream release, and as such, should be written somehow in the upstream version number, typically with a version bundle number, like 1.2.3~bundle1. Should the tarball exclude some files, it could be written as 1.2.3~dfsg1+bundle1. So when a sub-module is upgraded, the suffix after bundle must be incremented.

Q then bundling must keep track of which versions are in the bundle - how to do it ? A mainly by comparing what is going to be in the new bundle with what is currently in the bundle ! All node_modules//package.json define the current version of each module ! If the new bundle has any change, increment its number by one.

Q how to watch sub-modules new releases ? A you don't really care - it's upstream's job to do that

Q really, i fear that many modules won't ever update their bundled sub-modules if no automatic tracking of updates is available A well, if we really want that feature, then an upstream tracker service must do that job. The list of submodules should be written in debian/watch.

Q how to install the bundled modules ? A just debian/install node_modules usr/lib/nodejs/

Q in which cases bundling is justified ? A this is not made to avoid packaging work ! bundling is useful when four of five of those conditions are met:

• the submodule is ridiculously small
• the submodule is abandonned upstream (can be checked on github usually)
• the submodule has no actual use elsewhere (can be checked on npmjs.org stats) despite it being possibly useful
• the submodule is too specific to be used elsewhere, so there is little chance it will ever be.
• there is a better alternative in the debian archive AND patching current module to use the alternative is too complicated, too much work, and upstream won't accept any patch regarding a change to using that other better module.

Mind that

• bundling is not something that simplifies copyright maintenance, because the upstream tarball of each submodule (and not just a part of it) is included, so it asks for more copyright review work.
• bundling is better than including source in debian/ dir (as patch or whatelse), because it's easier and cleaner to maintain and update.
• bundling allows running two modules A and B with the same submodule Ca and Cb with different incompatible versions. Usually it's fixable but sometimes bundling could save a maintainer life.
• bundling a module already in the debian archive is a MUST NOT (except for the specific case above).

[RossGammon]

On 09/16/2015 10:20 PM, Jérémy Lal wrote:

An excellent summary - thanks Jeremy!

bundling a module already in the debian archive is a MUST NOT (except for the specific case above).

So the resulting install of the Debian package (except for the location) will look almost exactly like the user did "npm install" as upstream intended. Except where we exclude a node module from the "repack" because it is already packaged in Debian. Should we record this diff somewhere?

Cheers,

Ross

[kapouer]

Well, almost exactly yes - with the little variation that the modules should be "deduped" if we want a proper installation, compatible with the possibility that one of the sub-sub-modules are actually available as debian packages. It's tempting to record this diff somewhere, but at any time we can deduce it from what's it's node_modules (in the bundled tarball) and what's in debian/control, so i'm not so sure it's a good idea to have another place holding the same information. Could be useful to debian/watch, though.

[kapouer]

Just came across this interesting use case: https://stream-utils.github.io/

in this use case, "stream-utils" is not an upstream bundle with a bundle version, so it's up to debian to use an artificial incremented version number whenever one of the bundled module need to be updated.

[kapouer]

More information about how debian/watch is used: https://lists.debian.org/debian-devel/2015/10/msg00013.html

[simevo]

can this be manually achieved by performing this in the packaging repo master branch ?

npm install
nom dedupe # https://docs.npmjs.com/cli/dedupe
# manually remove from node_modules all modules that we already have in debian
git add node_modules
# debian/install: node_modules usr/lib/nodejs/

given the very low frequency that modules are bundled nowadays and that we 're actively trying to unbundle (as in node-asap and node-ms), does it make sense to have a dedicated bundle command in npm2deb ?

[kapouer]

Update: using pkg-components >= 0.10 i've setup an example with node-tar 4.4.1+ds-2, and came up with some kind of concept of "solitary module". See https://salsa.debian.org/js-team/node-tar/blob/master/debian/README.source

[simevo]

now that the accepted procedure for bundling seems to be https://wiki.debian.org/Javascript/GroupSourcesTutorial, npm2deb could support it

the tool could automatically detect which modules are not yet in Debian, and:

• generate debian/watch and debian/gbp.conf
• add Provides in debian/control
• debian/rules:
  • for build deps, add the required ln -sf ... statements in override_dh_auto_build
  • add rm -rf node_modules in override_dh_clean
• install stuff in debian/install (better install only required files such as index.js and package.json for each embedded module)

user should then need to:

1. check if they really are embeddable (i.e. tiny, no build process, not used elsewhere)
2. decide whether to track embedded module version in package version (i.e. group vs ignore opt)
3. complete group sources tutorial procedure

step 1 could even be made semi-automatic:

• tiny = LOC count < THRESHOLD (i.e. 200)
• no build process = absence of build key in package.json
• sum of weekly downloads of all dependents listed on npm registry < THRESHOLD (i.e. 100000)