Publish compiled binaries with new releases

[philknows]

In order to help reduce or mitigate the risks which come from building Lodestar locally (example: NPM), this proposal is to add binary releases to our release process. This will simplify and reduce the risks of running Lodestar bare metal on Linux.

All the other 4 mainnet clients do this currently and we should add this as part of moving Lodestar to a production-ready client.

TO-DO:

• Research bundling nodeJS apps
• Start publishing new releases with a .tar.gz binary for AMD64 infrastructure.
• Publish binaries with checksum for verification of bundle.

[philknows]

Will any of these suffice @dapplion ? I think it's important to figure this out to help reduce the risks of NPM or building from source.

https://github.com/nexe/nexe https://github.com/leafac/caxa https://nectar.js.org/

[wemeetagain]

I appreciate the simplicity and approach of caxa.

In the mean time, our docker image gives users something similar (guards against installation-time npm risks). While this is in progress, we can update our docs to recommend docker, or call out this benefit or the problem/risk explicitly.

[philknows]

@wemeetagain Would it be difficult for us to implement this as part of the releases that we do so we push binaries alongside the source code .zip/.tar.gz? I'm thinking something similar to what Prysm does for their releases.

I agree, documentation update to recommend docker would be a key change to make.

[philknows]

Error from Windows on CI build with caxa:

C:\Users\Phil-ChainSafe\Downloads>lodestar-windows-next.exe C:\Users\PHIL-C~1\AppData\Local\Temp\caxa\applications\lodestar-windows-next\eirrbjsq6f\1\node_modules.bin\lodestar:2 basedir=$(dirname "$(echo "$0" | sed -e 's,\,/,g')") ^^^^^^^

SyntaxError: missing ) after argument list ←[90m at Object.compileFunction (node:vm:352:18)←[39m ←[90m at wrapSafe (node:internal/modules/cjs/loader:1031:15)←[39m ←[90m at Module._compile (node:internal/modules/cjs/loader:1065:27)←[39m ←[90m at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)←[39m ←[90m at Module.load (node:internal/modules/cjs/loader:981:32)←[39m ←[90m at Function.Module._load (node:internal/modules/cjs/loader:822:12)←[39m ←[90m at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)←[39m ←[90m at node:internal/main/run_main_module:17:47←[39m

[dapplion]

basedir=$(dirname "$(echo "$0" | sed -e 's,,/,g')")

Where is this coming from? I don't see this in the codebase

[wemeetagain]

I think this is coming from caxa

[philknows]

@wemeetagain , I spoke to @mpetrunic today and he has some experience using https://github.com/vercel/pkg for packaging up node applications if you wanted to give this one a try instead.

[wemeetagain]

• prerequisite for #4177

[GregTheGreek]

has anyone been working on this?

[wemeetagain]

we have a branch that builds binaries for linux and macos, had issues with windows, and things stalled there. https://github.com/ChainSafe/lodestar/compare/unstable...cayman/release-binary

[philknows]

Adding some discussion points from Devconnect here after speaking to @nflaig and conversations with other node operations teams. It seems like this would be a preferred approach to helping adoption and enabling node operators to work with Lodestar in their setups if this issue is completed.

• Ethereum on ARM would be able to benefit from this and potentially look at running Lodestar on smaller, lower powered devices.
• Other large node operators have expressed this to be an option for running Lodestar
• We also benefit by having users of Lodestar running executables rather than building or using Docker to prevent user issues relating to package incompatibilities and it should always force users to use the most up-to-date supported dependencies such as node v20.
• Should benefit UX for most node operators

We previously tried Caxa, but ran into issues with Windows binaries. This might not be a huge issue anymore if Windows users are able to benefit from WSL.

[nflaig]

I have been in contact with Ethereum on ARM and they have already integrated Lodestar (fpm-package-builder/l1-clients/consensus-layer/lodestar) . Caxa does not seem to support cross-compiles but we might be able to just use a arm64 runner to built the binary for arm.

I quickly looked at two other libraries similar to caxa

• https://github.com/nexe/nexe
• https://github.com/vercel/pkg

Further testing is required, need to do some research on different approaches used to bundle the binary.

Ideally we want to publish binaries on github with every release, support linux (amd64/arm64), macos, windows, and also publish a verifiable signature for each binary.

[nflaig]

Research

I have researched and tested a few different tools but have not found an ideal solution yet

pkg

• does not support node 20 https://github.com/vercel/pkg/discussions/1972 yet, project has maintenance issues
• does not support ES modules https://github.com/vercel/pkg/discussions/1388
• does not support worker threads https://github.com/vercel/pkg/discussions/2023

nexe

• native module support is limited (native-modules), nexe-natives offers potential workaround
• prebuilt node binaries are not available for node 20 and must be compiled from source

boxednode

• does not support multiple js files, requires to bundle Lodestar in single js file first
• does not support ES modules https://github.com/mongodb-js/boxednode/issues/26

sea

• does not support ES modules
• still experimental (stability 1)

caxa

• has been archived / deprecated few days ago (Nov 21, 2023)
• requires to arm64 runners to build arm64 binaries https://github.com/actions/runner-images/issues/5631

package

• successor of caxa, see difference between caxa and package
• does not produce single binary (might not be an issue)
• requires to arm64 runners to build arm64 binaries

Another summary comparing different tools can also be found here.

Conclusion

Based on the research I think the only long-term viable solution that does not break between node versions is caxa (or package) as those just provide as thin wrapper around the source code and dependencies (great video on how it works). All the other tools try to create "real" binaries but this causes too many issues especially due to the use of native modules and worker threads (and other low level APIs) in Lodestar. We also already know that caxa works on arm64 if build on an arm machine (generatebinary.sh).

Issues

• caxa does not support cross-compiles which means we have to use a arm64 runner to build the binary but github does not yet natively support it
• attempt to built arm64 binary (https://github.com/ChainSafe/lodestar/compare/unstable...nflaig/arm64-binary) does not fully work due to native modules as those are built for x86-64 and I haven't found a way to build aarch64 binaries

Possible solutions

• self-hosted arm64 runner as suggested in comment
• wait for native arm64 runners on github (Q1 2024?)

[dapplion]

Caxa looks good, thank for researching these options! Why not support amd archs for now, and arm latter?

[nflaig]

Why not support amd archs for now, and arm latter?

We can definitely do that but the only known consumer of binaries right now is Ethereum on ARM and I am not sure if it makes sense to invest more time into this now (to get it production ready) if we can't even support them yet. I currently also have some doubts using caxa as the author just deprecated it (Nov 21, 2023) and package (the successor library) still has some issues and does not produce a single binary file which I don't like that much. I wanna make sure before we commit to using caxa that we could potentially maintain it ourselves, it does not look like there are many updates (if any) required to keep the library compatible with new node versions.

[jeluard]

I experimented with node single executable applications (see a guide). Some things are not supported (e.g. this top-level-await) and ultimately it fails to properly start, but it looks promising.

Some more details

Final binary size is 83MB. Error when starting:

(node:28835) ExperimentalWarning: Single executable application is an experimental feature and might change at any time
(Use `server --trace-warnings ...` to show where the warning was created)
node:internal/url:775
    this.#updateContext(bindingUrl.parse(input, base));
                                   ^

TypeError: Invalid URL
    at new URL (node:internal/url:775:36)
    at normalizeReferrerURL (node:internal/modules/helpers:298:10)
    at importModuleDynamically (node:internal/modules/cjs/loader:1261:45)
    at importModuleDynamicallyWrapper (node:internal/vm/module:431:21)
    at importModuleDynamicallyCallback (node:internal/modules/esm/utils:176:14)
    at ./packages/cli/bin/lodestar.js:3:1
    at embedderRunCjs (node:internal/util/embedding:37:10)
    at node:internal/main/embedding:18:8 {
  code: 'ERR_INVALID_URL',
  input: './packages/cli/bin/lodestar.js'
}

[nflaig]

Some things are not supported (e.g. this top-level-await) and ultimately it fails to properly start

It's listed in https://github.com/ChainSafe/lodestar/issues/3633#issuecomment-1826397373 (sea), last time I checked it didn't even support ES modules, and even if they fix this there will likely be issues with native modules, worker threads, and other low level APIs. Although stability got moved from experimental to active development, definitely something to follow and keep up to date.

[jeluard]

Gave another look at this after discussing with @nflaig. Sounds like the more reasonable approach to address this considering our constraints is to use caxa. Unfortunately caxa has just been deprecated, so it makes sense for us to fork it.

• [ ] fork caxa into chainsafe organization
• [ ] update release workflow to build/sign/publish those artifacts (see previous attempt or what ethereumonarm is doing)
• [ ] use new GitHub runners to produce arm64 builds (or workaround if it's not doable)

Later, we can update our doc website to have a comprehensible list of releases, similar to what geth is doing.