Build still railroads embeded stuff, cannot use system packages

[alerque]

Thanks for the help cleaning up the build process so this is easier to package natively for Linux distros. We're not quite there yet.

Now that I got node-sword-interface packaged okay I'm trying to update Ezra to use it — and with it, the system sword installation.

Unfortunately the way package.json is setup it is railroading me into doing things the old way even when I'm trying to strip it. The very first thing my packaging is doing after downloading the source (or in the current case I'm working on an -git package that builds from HEAD, so this is a clean git clone) is trying to remove things from the package.json file that are going to be provided by the system:

npm uninstall --no-audit --save-dev electron node-gyp
npm uninstall --no-audit node-addon-api node-sword-interface

Even as the first of those two commands runs, it somehow triggers a whole chain of things that does a "node-gyp rebuild" and starts doing a git clone/build of SWORD! That's exactly what I'm trying to stop from happening.

I'm guessing the offender in this case is the postinstall script, but I'm not sure.

P.S. The install-node-prune script is a dirty hack... downloading and installing binaries on a side channel during a build process is something that would get be banned as a packager on Arch Linux! I can work around it by not using the install script and hence not triggering it, but wouldn't just having that as a devDependency be better?

[alerque]

It wasn't the postinstall, I got the same behavior after patching that out on a clean clone. It seems like npm is doing an install all on it's own before running the uninstall, hence triggering exactly the thing I'm trying to remove. I'm not sure there is a way to do this other than patching the package.json to manually remove those deps before starting anything.

If you have a better idea I'm all ears.

[alerque]

I think I finally have a build that uses (and works) with my system SWORD. In the end I used the nuke-and-pave method completely deleting all the npm script definitions in package.json and hand writing the correct variations in the package definition. Here are the relevant bits (with confusing Arch specific bits redacted).

First, some key meta data that will be provided by the package, not the app's package.json:

depends=('electron'
         'icu'
         'node'
         'node-addon-api'
         'node-sword-interface'
         'sword')

makedepends=('node-gyp'
             'node-prune'
             'node-pug-cli'
             'npm')

With that in place, we can strip all the system-provided things from the package file (and also nuke all the script definitions so the don't accidentally trip us up):

prepare() {
    jq 'del(.scripts[], .dependencies["node-addon-api", "node-sword-interface"], .devDependencies["electron", "node-gyp", "pug-cli"])' package.json |
        sponge package.json
}

Now and only now it's safe to run npm inside the project source:

# Note $_electron has the system's electron version
build() {
    npm install
    pug --client --no-debug --pretty -n verseListTemplate templates/verse_list.pug
    pug --client --no-debug --pretty -n tagListTemplate templates/tag_list.pug
    "$(npm bin)"/electron-rebuild --version="$_electron"
    node-prune node_modules
    "$(npm bin)"/electron-packager ./ ezra-project --electron-version="$_electron"
    ./build_scripts/purge_build_artifacts.sh
    "$(npm bin)"/electron-packager ./ ezra-project --electron-version="$_electron" --overwrite --asar --platform=linux --arch=x64 --prune=true
}

The only thing left its building that it should probably use from the system is sqlite. Once that is added as a system dependency I think the two stage eletron-packager runs with the purge in between can go away, but I can take care of that separately.

Now that we have everything built, we can put the only thing we need from that build (the asar file) where it will go on the system:

package() {
    install -Dm644 -t "/usr/lib/ezra-project/resources/" ezra-project-linux-x64/resources/app.asar
}

Separately, this wrapper gets installed in /usr/bin/ezra-project:

#!/usr/bin/env sh
exec electron /usr/lib/ezra-project/resources/app.asar "$@"

The result is a system package that is 16.43 MiB over the wire and 65.50 MiB installed. Compare to 70+ MiB rpm/deb packages and 300+ MiB installed size.

Does this look sane to you? It's a lot simpler than trying to make use of the currently defined build scripts, but it seems like there should be a way for this build sequence to be built into the project so it doesn't have to be re-invented by other packagers. For example I know Gentoo, Alpine, NixOS, and others will have to do some or all of the same steps.

Thoughts?

[tobias-klein]

Wow, you invested a lot of time into this! Thank you so much!

The end result (package size 16.43 MiB) is certainly impressive and I can see how this can only be achieved by depending on system-installed packages. This example shows the power of Linux packaging concepts!

One consequence based on this approach is that you cannot link to specific versions of Electron any longer, because the version may not be available as a distribution package yet? In such cases the resulting software may be broken if it depends on new features in the latest Electron versions. What are your thoughts on that?

I think we can certainly consider using this approach for other Linux targets besides Arch Linux. It does require the availability of an electron package for the respective distribution, though. I didn't even find an electron package for Debian or Ubuntu. I think that's generally a pre-condition.

In any case the existing approach is probably still necessary at least for Windows and macOS.

The install-node-prune script is a dirty hack... downloading and installing binaries on a side channel during a build process is something that would get be banned as a packager on Arch Linux! I can work around it by not using the install script and hence not triggering it, but wouldn't just having that as a devDependency be better?

Yeah, you're right. I think I can turn this into a dev dependency. I will have a look.

[alerque]

The end result (package size 16.43 MiB) is certainly impressive and I can see how this can only be achieved by depending on system-installed packages.

Down to 11 MiB with system supplied node-sqlite3 ;-)

One consequence based on this approach is that you cannot link to specific versions of Electron any longer, because the version may not be available as a distribution package yet? In such cases the resulting software may be broken if it depends on new features in the latest Electron versions. What are your thoughts on that?

This is the distro packager's problem. Arch Linux is a rolling release so there are typically newer packages available than upstreams tend to use. For example as of yesterday the electron package was 8.0.3 whereas you are using 8.0.0 upstream. As of today 8.1.0 is available and I'll be bumping shortly if it works. My package actually has newer not older electron, so you depending on new features is not an issue. Far more common is the opposite problem: upstream projects frequently don't work on newer dependencies. In the case of Electron Arch actually has electron 2, 3, 4, 5, 6, and 7 packages available besides the default 8 package that can be used for projects that don't support newer generations of electron yet. Similar is true of other programs.

One example will be if Sword comes out with a new release that Ezra doesn't support yet. Updating Ezra will require either not upgrading SWORD or packaging an additional old version of it for compatibility or waiting to update Ezra until it is compatible with the latest dependencies.

In any case the existing approach is probably still necessary at least for Windows and macOS.

Yes, there isn't as much you can do on those platforms. Arch and some other distros are built for this, but you as an upstream aren't is a position to make it work for platforms like Windows. For Arch it is a requirement that the system be kept as lean as possible by not bundling dependencies (like electron). The best you can do is make it easier for people like me trying to make that happen by not making so many assumptions about how the build will happen.

[tobias-klein]

Thanks for the explanations! That helps.

Regarding this issue - what do we do with it now?

The install-node-prune script is a dirty hack... downloading and installing binaries on a side channel during a build process is something that would get be banned as a packager on Arch Linux! I can work around it by not using the install script and hence not triggering it, but wouldn't just having that as a devDependency be better?

Yeah, you're right. I think I can turn this into a dev dependency. I will have a look.

This turns out to be difficult based on the fact that I'm not using a regular npm package, but my own fork which requires the execution of a script for installation. I had forked it, because it was deleting more than it should and I patched it ... still ugly, I totally admit that! I could raise a request on the actual node-prune project to make the list of deleted folders configurable.

[alerque]

Regarding this issue - what do we do with it now?

I would like to find a way to build without monkey patching the package.json file first, but I don't actually see a way to do that right now, and that's largely npm's fault.

I'll look into perhaps breaking the script aliases & script files down in such a way that it would be easier to pick and choose what to run without re-inventing the whole process. Maybe assign this issue to me for that. I probably won't get to it soon, I'm fast approaching another time drain (expecting a baby any day now).

I could raise a request on the actual node-prune project to make the list of deleted folders configurable.

Yes, lets start with that at least!

[alerque]

I could raise a request on the actual node-prune project to make the list of deleted folders configurable.

Yes, lets start with that at least!

What images exactly are being deleted that shouldn't be? Perhaps the problem is they are in the wrong place to start with. There is something to be said for not fighting tooling expectations if there is a reason to generally remove the images directory.

[tobias-klein]

I could raise a request on the actual node-prune project to make the list of deleted folders configurable.

Yes, lets start with that at least!

What images exactly are being deleted that shouldn't be? Perhaps the problem is they are in the wrong place to start with. There is something to be said for not fighting tooling expectations if there is a reason to generally remove the images directory.

That could be true. I will check whether the patch for node-prune could be avoided by doing some refactoring on my end.

[alerque]

Given the case of:

1. node-addon-api, node-sword-interface, and node-sqlite all having been built for the current system and installed globally (.i.e all the things that create system arch specific artifacts using node-gyp),
2. having those same said modules linked into the projects node_modules after everything else is installed,

Is there any need then to run npx electron-rebuild? Since there are no system/architecture specific things going on in any of the other node dependencies it seems to me like I can drop that step too right?

[tobias-klein]

This is step is only needed when you're still handling native modules as part of the build process. But since you're depending on system packages in your case the step should not be needed anymore!

On 3/7/20 2:12 PM, Caleb Maclennan wrote:

Given the case of:

1. |node-addon-api|, |node-sword-interface|, and |node-sqlite| all having been built /for the current system/ and installed globally (.i.e all the things that create system arch specific artifacts using |node-gyp|),
2. having those same said modules linked into the projects node_modules after everything else is installed,

Is there any need then to run |npx electron-rebuild|? Since there are no system/architecture specific things going on in any of the other node dependencies it seems to me like I can drop that step too right?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tobias-klein/ezra-project/issues/23?email_source=notifications&email_token=AEHLA577IR7FCHO6ZV7PP23RGJB4PA5CNFSM4LC4NWD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEODZCFQ#issuecomment-596087062, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEHLA5YR5UJADMX46MF43BLRGJB4PANCNFSM4LC4NWDQ.

[tobias-klein]

Note that meanwhile I have commented on an existing issue of node-prune to see if they can provide configurability, so that we don't need to use a fork any longer.