Catch-22 for CI Builds

[aj-foster]

Hello there 👋🏼 Thanks for all of your work on this project.

I've run into a confusing situation when publishing a package from CI. The overall intended flow is this:

• Job 1: Precompile Rust assets
  • Largely taken from the example GitHub Action
• Job 2: Publish package to Hex
  • Checkout
  • Setup Elixir/Erlang/Rust
  • Generate the checksum file **
  • mix hex.build
  • mix hex.publish

** I'm unsure how to accomplish this with the current setup.

If I configure CI to force a build of the Rust package, then the metadata file is not created and mix rustler_precompiled.download fails with an error metadata about current target for the module ... is not available. On the other hand, if I don't force a build, I'm unable to run mix rustler_precompiled.download because the prerequisite mix compile step fails with an error the precompiled NIF file does not exist in the checksum file.

Do you have any advice? Can we optionally skip the checksum check or create a nice way to generate the checksum without first compiling the package?

It is, of course, possible to download all of the artifacts and generate the checksum file manually, but this seems brittle.

Thanks for your help!

[philss]

Hi @aj-foster :wave:

Sorry for the delay.

I'm not sure if I got it. But would you mind to try a compilation of the project (mix compile) while forcing the compilation of your crate, and then after that running the mix rustler_precompiled.download YourModule --all without forcing the create?

Something like:

$ FORCE=true mix compile
$ mix rustler_precompiled.download YourModule --all

[aj-foster]

Hello!

I gave this a try (at least, I think I followed the instructions correctly):

# ...
RUSTLER_FORCE_BUILD=true mix compile
mix rustler_precompiled.download [Module] --all --print

The first step succeeded, however the second step failed with the following error:

** (RuntimeError) metadata about current target for the module [Module] is not available. Please compile the project again with: `mix compile --force`

[philss]

I'm sorry, but I don't know exactly what is going on :/ If you can share the source code, so I can play with, would be great!

[aj-foster]

Certainly, you can find the latest run of the action here and the active workflow file is here. Thanks for your support on this. There's probably something simple I'm missing from the setup.

[philss]

@aj-foster sorry for the delay. One thing I'm doing is always force compilation of the NIF in all the steps. Maybe setting the env var in the entire job can do the trick. Have you tried this?

[aj-foster]

I was able to get around this problem with the following method: 🎉

1. mix compile --force || true — deliberately attempt to compile the project, without forcing a build, without a checksum file. This will fail with error ** (RuntimeError) Error while downloading precompiled NIF: the precompiled NIF file does not exist in the checksum file. However, this will create the metadata file.
2. RUSTLER_FORCE_BUILD=true mix compile --force — run a successful compilation with a forced build. This is a necessary prerequisite for running the manual download.
3. mix rustler_precompiled.download Rust.OT --all --print — successfully download the built artifacts.

In short, the issue is that a metadata file is not generated when a build is forced. If it were, the additional compilation step would not be necessary.

[philss]

@aj-foster thanks for the details! :purple_heart: I think it's fixed in https://github.com/philss/rustler_precompiled/pull/37 I'm going to test a little bit more and release a new version soon if everything is good.

[philss]

@aj-foster there is a new version, 0.5.3, that removes the need for the first step. Thank you for investigating this!

[mlwilkerson]

@philss thank you!

I've been experiencing the same problem as @aj-foster. I was just coming here to report my issue when I noticed you posted #37 ! I'm eager to give this a try and see if it resolves my problem too.

[mlwilkerson]

Yes! #37 works great for me. Awesome.