feat(dist): add cargo-dist

[ashleygwilliams]

addresses #552 - marked as draft as i made some calls about what targets and installers you'd like to support. happy to talk through those and update.

you can run cargo dist plan locally to see what this workflow will generate. i've also put the output here:

announcing v0.4.0
  cargo-cyclonedx 0.4.0
    cargo-cyclonedx-installer.sh
    cargo-cyclonedx-installer.ps1
    cargo-cyclonedx-aarch64-apple-darwin.tar.xz
      [bin] cargo-cyclonedx
      [misc] CHANGELOG.md, LICENSE, README.md
      [checksum] cargo-cyclonedx-aarch64-apple-darwin.tar.xz.sha256
    cargo-cyclonedx-x86_64-apple-darwin.tar.xz
      [bin] cargo-cyclonedx
      [misc] CHANGELOG.md, LICENSE, README.md
      [checksum] cargo-cyclonedx-x86_64-apple-darwin.tar.xz.sha256
    cargo-cyclonedx-x86_64-pc-windows-msvc.zip
      [bin] cargo-cyclonedx.exe
      [misc] CHANGELOG.md, LICENSE, README.md
      [checksum] cargo-cyclonedx-x86_64-pc-windows-msvc.zip.sha256
    cargo-cyclonedx-x86_64-unknown-linux-gnu.tar.xz
      [bin] cargo-cyclonedx
      [misc] CHANGELOG.md, LICENSE, README.md
      [checksum] cargo-cyclonedx-x86_64-unknown-linux-gnu.tar.xz.sha256
    cargo-cyclonedx-x86_64-unknown-linux-musl.tar.xz
      [bin] cargo-cyclonedx
      [misc] CHANGELOG.md, LICENSE, README.md
      [checksum] cargo-cyclonedx-x86_64-unknown-linux-musl.tar.xz.sha256

[ashleygwilliams]

to test this out we could push a pre-release tag and it would generate a github release. let me know if that sounds like a good plan and if you'd like to add or remove any targets or installers i picked.

[Shnatsel]

Thanks for the PR! I'll take a closer look tomorrow.

[Shnatsel]

Thanks! Two things of note:

1. We would need to also remove the build steps added in #488
2. Is there an easy way to add aarch64 Linux as well? I'm pretty sure cross can handle that.

[ashleygwilliams]

i can write up an additional workflow that could do aarch64 linux- cargo-dist doesn't currently support it (the workflow committed in this PR is fully generated, i can also hand edit the release workflow, but it's generally better to add an additional workflow that does the additional target). and happy to remove the other workflow.

[Shnatsel]

In that case I'd prefer to stick to the fully generated workflow in the interest of reducing the maintenance burden and making upgrading to newer cargo dist easier. I don't want aarch64 linux prebuilt binaries badly enough to hand-roll them.

The previous release workflow would need to be removed, but other than that this looks good to go.

[ashleygwilliams]

ok cool! also, now that we have a request for aarch64 linux we can look into getting it into cargo-dist :) i'll get the changes up momentarily for this PR

[Shnatsel]

Thanks!

My thinking is that Raspberry Pis need prebuilt binaries really badly, and ARM servers are increasingly popular as well because of the way cloud providers price them.

But it's not really needed for cargo cyclonedx. You would usually run cargo cyclonedx either on the developer machine or on CI, but RPis make terrible developer machines, and there is still no CI provider with easily available (let alone free) ARM servers.

[lfrancke]

That said... We just build an ARM GitHub runner and would love this :) but yeah... Not too important

[ashleygwilliams]

hey! realized i fell off on this and would love to pick it back up. will update to the latest cargo-dist now, and let me know what else i need to add to get this over the finish line! sorry about the delay

[ashleygwilliams]

re linux arm- if there's acgustom runner we can point to i can add this as a target. at axo we use buildjet for this- costs us an incredibly small amount of money, would recommend (but also understand not wanting to).

[Shnatsel]

The PR is still marked as draft. I understand it is ready for review now?

[lfrancke]

@Shnatsel this is ready for review now. Would you be willing to take a look?

[Shnatsel]

Yes! Thank you. I will take a look in the next few days.

[stevespringett]

Any chance of getting a review of this PR?

[Shnatsel]

Sorry for dragging my feet on the review.

Unfortunately this runs into some thorny issues with the way the crates are currently published. None of the active maintainers have access to the publishing token, and the original maintainers are incommunicado. This leaves us with publishing releases to crates.io using a token stored in a github action secret.

We recently had to add another crate to the project, because procedural macros require a separate crate for technical reasons. This complicates the release process even further. I want to sort out the release process with the extra added crate first, which is required for the crucial CycloneDX v1.5 support in the library, and only then deal with further less critical changes including this PR.

[stevespringett]

@Shnatsel ping me on Slack and we'll sort it out.

[Shnatsel]

There already is a workflow we inherited from the previous maintainers that compiles and publishing binaries to Github releases. Having two of them at the same time is obviously not a good idea. Compiling binaries should be disentangled from the publishing workflow before I can land this PR.

I am being extra cautious with changes to the publishing workflow because without access to the publishing token we wouldn't be able to yank published crates, which makes correcting any mistakes in the release process very difficult.

[ashleygwilliams]

a point that may be valuable to note: this current workflow does not include publishing to crates.io. i agree that extra consideration should be made there because it is a one way door. the pr included is for building binaries, and then also building a curl | sh installer, and a powershell installer. these installers are preferable on CI platforms because they do not require compilation (which a cargo install does). we can also add a homebrew and npm installer if that's desirable with a bit of config, but that is not part of this PR.

everything in this PR generates something that is a two way door: all bins, installers, and github releases are mutable (deletable/fixable).

[ashleygwilliams]

it is possible that a sync conversation about this may help any confusion. i am happy to make myself available on slack and/or a zoom call

[Shnatsel]

I understand that this PR does not publish to crates.io. We have a legacy workflow which does, and which also compiled binaries. The binary publishing part will have to be removed from the legacy workflow before this one can be merged. This should (hopefully) be as easy as reverting #488. Actually doing that is a prerequisite for landing this PR.

We already have a Github release with published binaries for the current version of cargo cyclonedx, so this PR will only take effect once we cut a new release. So here is my proposal for an actionable plan to get this merged:

1. Complete CycloneDX v1.5 support in cyclonedx-bom crate, tracked as #646 (@justahero)
2. Create a publishing workflow for cyclonedx-bom-macros crate and publish an initial release (@pvdrz, @Shnatsel)
3. Publish a new release of cyclonedx-bom crate (@Shnatsel)
4. Update cargo cyclonedx with spec version 1.5 support (@Shnatsel)
5. Revert #488 and double-check for any other edits required to the legacy publishing/tagging workflow (@Shnatsel)
6. Double-check the tag names in this PR and make sure everything aligns with the legacy publishing/tagging workflow, then merge this PR (@Shnatsel)
7. Publish a new version of cargo cyclonedx, finally making it accessible to cargo binstall et al.

My guesstimate for the ETA on all of this is 2 weeks.

I hope this sounds good?

[Shnatsel]

We have sorted out the publishing permissions (thanks, Steve!) so the release process should be a lot less risky now.

I was about to ask how to go about upgrading to a newer version, but the cargo-dist book already covers it. Nice!

[ashleygwilliams]

@Shnatsel awesome news. let me know how i can help. additionally- if it comes up that there's targets or package manager systems this project needs that cargo-dist does not yet support, we are extremely open and excited about feature requests, so don't hesitate to let us know.

once this is merged, my goal is to integrate cyclonedx BOM generation as a configurable feature in cargo-dist. i've filed an issue to track work on it here: https://github.com/axodotdev/cargo-dist/issues/1016

[Shnatsel]

Oh, SBOMs in cargo-dist sounds great! If you run into any issues with the tool, or have any questions, please let me know. I am happy to help!

[ashleygwilliams]

also FYI- cargo-dist supports arm linux, the main thing youll need is a custom github runner (since we don't do it as a cross... yet). we use buildjet for cargo-dist and it costs us pennies (you can see our config here: https://github.com/axodotdev/cargo-dist/blob/main/Cargo.toml#L39). i dont know if owasp gives ya'll a budget, but i can set up the arm linux target if you'd like so long as you provide me a name for a custom runner!

[ashleygwilliams]

anything more i can do to help on this? (hate to be a bother, but really hoping to use this for a demo)

[Shnatsel]

I suppose I could trigger this workflow to build executables for an existing release without blocking on shipping CycloneDX 1.5.

The workflow is currently only triggered by pushing a version tag. I understand we would need to manually modify it to allow triggering it manually for a specific tag?

If that is correct, I would be OK with merging a workflow with those modifications and triggering a binary release manually, but I'd like to revert to a more easily maintainable state without manual modifications to auto-generated files afterwards.

[Shnatsel]

Alternatively I should be able to get the new release out of the door in a day or two.

[Shnatsel]

I've opened a PR for shipping cyclonedx-bom v0.6.0: #710

Once that's done, I can finally revert #488, merge this PR, and have the tag automatically trigger a build with the cargo dist workflow.

[ashleygwilliams]

hey @Shnatsel there's a few options for changing how stuff is triggered without having to manually edit the generated workflow. if i'm understanding your concern correctly i think one of the following would work well, with the first being my preference:

• workflow dispatch (docs): this creates a tag at the end, and is only manually triggered. people who REALLY want to treat tags as immutable like this strategy.
• tag namespace (docs): if you want to temporarily require that the tags it triggers on have an arbitrary string as a namespace you could use this

[ashleygwilliams]

also- would you like me to update the version of cargo-dist this uses? (we are also about to release 0.15.0 that has some cool features like more hash options)