stacks-core icon indicating copy to clipboard operation
stacks-core copied to clipboard

[CI] Improve release build process

Open wileyj opened this issue 3 months ago • 4 comments

Current process relies on docker and cross-compilation (including cross-compiling for different os'es). Under the best circumstances, it takes roughly 30+ mins to complete all binary builds for all platforms. under less than ideal circumstances, a single failure of any of the binary builds requires a re-run, which delays a release until all binary builds have completed.

The first improvement i think that could be made here is to remove docker from the process - we can build the binaries on a runner (and in the case of macos, use a macos runner for that task). the second option (and this is relevant for retries) is that i think we should also be using workflow artifacts vs the dockerfile downloding the archive from teh github release.

if we keep using the archive from the github release - what can happen is a release looks like it has been published with the binaries, but there may be a problem with the release Dockerfile so the images are not built and the entire workflow needs to be re-run.

propose:

  1. matrix to run the binary builds on the runner natively
  2. save the binaries as a workflow artifact
  3. update release dockerfile to use the workflow artifact
  4. publish a github release using the workflow artifact in draft mode.

currently, the binary builds all happen in a composite. part of this work may involve moving that workflow back into the stacks-core repo (at a minimum, the matrix needs to be triggered from stacks-core, so it makes sense to move all jobs unless used elsewhere).

side note: this process update may also benefit #6556 , since the same process could be used to create an ad-hoc mutli-platform image for testing. building the multi-platform image in docker on a default runner takes ~2 hours, whereas building teh binaries takes ~15 minutes or so.

wileyj avatar Oct 14 '25 18:10 wileyj

small snag in what i was planning, needto find an alternative: https://github.com/orgs/community/discussions/27906

basically, the aarch64 musl build needs a specific linker - in this case https://musl.cc/aarch64-linux-musl-cross.tgz. however, musl dot cc is blocking azure ip ranges which the runners are using. 🤔

wileyj avatar Oct 14 '25 23:10 wileyj

small snag in what i was planning, needto find an alternative: https://github.com/orgs/community/discussions/27906

basically, the aarch64 musl build needs a specific linker - in this case https://musl.cc/aarch64-linux-musl-cross.tgz. however, musl dot cc is blocking azure ip ranges which the runners are using. 🤔

There is a mirror in github, so it's a workaround but probably a good idea to keep a static copy somewhere safe. https://github.com/musl-cc/musl.cc/releases/tag/v0.0.1

wileyj avatar Oct 14 '25 23:10 wileyj

~13 mins for the release builds (minus macos aarch64): https://github.com/wileyj/stacks-core/actions/runs/18697263795/job/53317819898

will need to polish up the process a bit and add in the next steps:

  • create/save binary archives as workflow artifacts
  • create docker image from the same artifacts (need a better way to put specific binaries on the image based on the tag - i.e. singer image doesn't need stacks-node and vice-versa)
  • attest archives and images
  • create github release

wileyj avatar Oct 21 '25 21:10 wileyj

update:

  • binaries are being built natively on vm's (no longer using docker) and stored as a build artifact.
  • docker components are in progress, but the expected data looks corect and i expect to run a full test today or tomorrow
  • signer builds (i.e. a signer only release) has not been tested yet.
  • identified some other small qol changes that i'll be adding to the eventual PR

wileyj avatar Oct 29 '25 15:10 wileyj