libblst does not get statically linked when sccache is present

[charles-cooper]

Bug Description

per title, a project which depends on the blst package ends up with GLOBAL (as reported by readelf) references to blst_p2_to_affine instead of LOCAL symbols when compiling with sccache enabled in a github action (using maturin-action). the resulting wheel fails at runtime with ImportError... undefined symbol: blst_p2_to_affine

Your maturin version (maturin --version)

1.5.1

Your Python version (python -V)

n/a

Your pip version (pip -V)

n/a

What bindings you're using

pyo3

Does cargo build work?

• [ ] Yes, it works

If on windows, have you checked that you aren't accidentally using unix path (those with the forward slash /)?

• [ ] Yes

Steps to Reproduce

https://github.com/paradigmxyz/pyrevm/actions/runs/8680598504

i've archived the log for linux-x86_64 at https://gist.github.com/charles-cooper/2fd01dc60fabea7815e8b8cc62fc8652. the same problem exists for both cffi and pyo3 (i did not check other bindings).

[lmmx]

Since this has gone uncommented on for a year, I supplied the details to ChatGPT o1 for any ideas or approaches, included below the fold

Click to show

This kind of error—where symbols turn up missing or become globally visible in unexpected ways when using sccache—often comes down to how the library is built and how sccache handles intermediate artifacts. While it’s hard to pinpoint without seeing the exact build logs and the library’s build scripts, here are some common causes and potential workarounds:

---

1) sccache Not Reproducing the Exact Compile Environment

sccache captures compiler invocations and replays them to produce the same object files. If blst (or a crate that wraps it) has a build script using environment-dependent flags (e.g., custom CFLAGS, link scripts, or conditional defines), it might be that:

• When sccache replays the compile, it’s missing certain environment variables or flags.
• As a result, it produces object files with incorrect symbol visibility or references.

Possible fix:

• In your build script (or via environment variables), ensure you pass consistent flags even under sccache.
• Sometimes setting RUSTC_WRAPPER=sccache is enough, but the library might also need CC="sccache cc" or similar.
• Make sure the environment variables that blst’s build script depends on are either whitelisted by sccache or embedded in the preprocessor/command line invocation.

---

2) Different Link-Time Optimization or Visibility Flags

Some libraries rely on LTO or -fvisibility=hidden / -fPIC / -fPIE for generating local vs. global symbols. If sccache is used, it might inadvertently skip or alter the link step. Or if the library is built in multiple steps (C/assembly + final link), sccache might not see the final link step properly.

Possible fix:

• Try disabling LTO for the blst crate or the entire build. For example, in Cargo.toml:

  [profile.release]
  lto = false
  

• Ensure any visibility=hidden or -fPIC flags are applied consistently in all compilation units.

---

3) Mix of Static and Dynamic Linking

If blst is expected to be statically linked but ends up partially dynamically linked (or vice versa), you can get undefined symbol errors at runtime. sccache might be reusing an object file from a prior build that was built with different link flags.

Possible fix:

• Double-check if the library is really compiled as a static library (.a) or a dynamic library (.so), and that you’re consistently linking it.
• If the crate’s build.rs has logic like “if on Linux do this, if on Mac do that” and you’re switching between runs, sccache might be reusing the wrong object.

---

4) A Bug in the Library’s build.rs or in cc-rs

If the blst crate is using cc crate or a custom build.rs, it might produce assembly or object files with local/global symbol scoping that’s not consistent. Possibly, the presence of sccache triggers a code path.

Possible fix:

• Check if the build.rs has a path like if env::var("CC_WRAPPER").is_ok() { ... } or uses env::var("CC").
• Look for #[cfg(feature = "some_sccache_related_config")] or anything that toggles compiler flags based on environment.

---

5) Workaround: Disable sccache for blst

Because this is a known bug that apparently has not been resolved in maturin-action, sometimes the easiest short-term solution is to disable sccache for that particular library (or for the entire build) if you absolutely need blst. This might not be ideal, but it prevents these symbol issues.

You can try:

• Per-crate override: There’s no built-in Cargo feature to selectively disable RUSTC_WRAPPER for just one crate, but you can sometimes do hacky approaches like building blst separately or changing its build script to skip using sccache.
• Disable globally: In your GitHub Actions file, set sccache: false for Maturin if you only see this bug for that one project.

---

6) Manually Forcing Local Symbols

If the problem is that the symbols are unexpectedly global, you might force them local using something like -fvisibility=hidden or appropriate assembly directives if blst is shipping .s or .asm code. That’s more advanced, though, and typically requires patching the blst build scripts.

---

Summary

• Root Cause: Something about using sccache is causing a mismatch in how blst gets compiled or linked, resulting in missing or incorrectly scoped symbols at runtime.
• Potential Solutions:
  1. Ensure consistent environment variables and flags (especially for local vs. global symbol definitions).
  2. Check LTO/visibility flags and disable them if necessary.
  3. Confirm static vs. dynamic linking.
  4. As a last resort, disable sccache for the library or the entire build if that’s the only surefire workaround.

If you need to keep sccache on, you may need to open an issue/PR in blst’s repo or maturin-action to see if they can handle this scenario more gracefully.