sarif-rs icon indicating copy to clipboard operation
sarif-rs copied to clipboard

Release additional static binaries to use with cargo binstall

Open echoix opened this issue 2 years ago • 2 comments

I'd like to request adding more binary releases to the tools published here, especially static binaries (ie: with x86_64-unknown-linux-musl and aarch64-unknown-linux-musl), in order to be able to quickly use the tools without having to compile from source if the released binaries aren't compatible with the target.

More context:

The tools shellcheck-sarif and sarif-fmt are used in MegaLinter to support outputing the results of the linter shellcheck and to convert sarif output into the console. Currently, our docker images build times are long in CI, and these two small tools make up most of the 5-6 minutes for the rust tools installation (all but the first 19 seconds), which represents 20-25% of the 20-30 min build times for the full image, which includes hundreds of tools. And it needs to run for all PRs, and most flavours, and some individual linters need it, plus other CI tasks daily. So it is worth it. In my issue here (I'm part of MegaLinter's team) https://github.com/oxsecurity/megalinter/issues/2382, we are planning on switching to cargo binstall, that this repo is configured to use, but since the docker images are based on alpine, the target triplet is x86_64-unknown-linux-musl, so everything must be recompiled.

I could try and help on working with the CI to accomplish that in the next weekends, but I'm not experienced yet with rust building, so I might have some trouble.

Let me know if you're open to that.

echoix avatar Feb 22 '23 23:02 echoix

@echoix hi, definitely open to this.

unfortunately the way the CI is written right now is that the target binary is same as the host it's running on (ie. no cross compiling), and it just runs on two separate hosts right now which correspond to each target it releases to (https://github.com/psastras/sarif-rs/blob/main/.github/workflows/main.yml#L138).

in principle rust supports cross compilation with built in flags, so it should be easy (even if you're less familiar with rust) -- i'd take a look here: https://rust-lang.github.io/rustup/cross-compilation.html

so most of the work IMO is just in the CI pipeline.

if you have time and want to dig into this great!

otherwise if / when i find the time i can take a look.

psastras avatar Feb 26 '23 23:02 psastras

From what I learned, if there isn't any OpenSSL needed in one of the dependencies, the solution might be as simple as expanding the CI config for the matrix to include the added build cases, add the apt package musl-tools (and maybe pkg-config if it's not enough), and add the target with rustup target add x86_64-unknown-linux-musl and rustup target add aarch64-unknown-linux-musl, then build normally.

Of course that means to adapt the workflow in order to have the file names for the outputs to work.

If it doesn't work like this (that was the case that would keep most of your CI config the same), then there are other completely different options that exists, either with some GitHub actions in the marketplace to help out these tasks, or the cargo cross project, that will build the files in a container from an image maintained by that project (before it was from the rust tools team).

The cargo binstall uses a very different and unique building pattern that was changed recently, so it's not really an example.

echoix avatar Feb 27 '23 00:02 echoix