The no-op always-success RUSTC_WRAPPER breaks build scripts

[RalfJung]

It is not an uncommon pattern for build scripts to invoke rustc to detect whether some features are available. Popular crates like anyhow and eyre do that, and the autocfg crate (14 million "recent" downloads, >100 million all-time downloads) even provides a convenient interface to write such build scripts.

RA entirely breaks that by having a RUSTC_WRAPPER that always returns success. I think that is a bug in RA.

[lnicola]

This was never reported before. Skimming autocfg, it doesn't seem to use RUSTC_WRAPPER, which might explain the difference. You can use disable rust-analyzer.cargo.buildScripts.useRustcWrapper as a workaround.

Specifically about anyhow, I don't think it should be testing for a feature that no longer exist, but we don't control that.

It would be nice for cargo to have a "run the build scripts, but don't check the code" mode.

[RalfJung]

This wrapper might also be responsible for https://github.com/rust-lang/rust/issues/99538; at least I have not seen that problem since I disabled proc macros and build scripts.

Skimming autocfg, it doesn't seem to use RUSTC_WRAPPER

Oh, good point.

anyhow was specifically asked to use RUSTC_WRAPPER in https://github.com/dtolnay/anyhow/pull/157. And I tend to agree with the author that using RUSTC_WRAPPER is more correct. But neither eyre nor autocfg are using RUSTC_WRAPPER so they happen to not hit this problem. I think that is an accident though, and arguments could be made that they should use RUSTC_WRAPPER.

Specifically about anyhow, I don't think it should be testing for a feature that no longer exist, but we don't control that.

Well, the probe needs updating to latest nightly, yeah. And soon the feature will be stable and this will only be relevant for old compilers. But that doesn't really help with the general problem.

[lnicola]

One solution might be to detect nested invocations of our wrapper: if we're not the outer one, we could pass through the call to rustc (or even to a RA_ORIGINAL_RUSTC_WRAPPER that we've saved).

[bjorn3]

Build scripts don't cause nested invocations of the wrapper.

[lnicola]

Build scripts don't cause nested invocations of the wrapper.

Doesn't the anyhow build script do just that?

[RalfJung]

Yes, it does.

[bjorn3]

Anyhow's build script invokes the wrapper, but the build script doesn't run underneath the wrapper, but directly under cargo. As such the wrapper invocation is not nested within the wrapper.

[lnicola]

Am I missing something here?

1. RA sets RUSTC_WRAPPER=rust-analyzer
2. RA runs cargo check
3. cargo check compiles the anyhow build script
4. cargo check runs the anyhow build script, presumably without clearing RUSTC_WRAPPER
5. the build script checks for RUSTC_WRAPPER and uses it to compile some code

[bjorn3]

That is correct. But how do we detect that anyhow's build script invoked the wrapper rather than cargo itself? Both use pretty much the same env vars and because at step 4 the wrapper isn't used to run the build script, we can't set an env var either to signal that a build script ran rustc with the wrapper as opposed to cargo.

[lnicola]

Can't we set RA_RUSTC_WRAPPPER=$RUSTC_WRAPPER in step 1, then we check for it (in step 5) and run that instead?

[bjorn3]

How do you make sure that it doesn't run for rustc invoked by cargo then?

[bjorn3]

I think we can check env vars like NUM_JOBS or PROFILE which only get set for build scripts.

[RalfJung]

It would be nice for cargo to have a "run the build scripts, but don't check the code" mode.

Yeah, fundamentally this is a hack to work-around not being able to tell cargo what you really want to do. Is that being tracked on the cargo side somewhere, are the cargo devs aware of this?

[lnicola]

How do you make sure that it doesn't run for rustc invoked by cargo then?

Hmm...

Is that being tracked on the cargo side somewhere, are the cargo devs aware of this?

I suppose not, but a request for cargo build --dependencies-only (which isn't that different) has been open since 2016 without any end in sight.

[lnicola]

We could use a file to pass information from one invocation to another, but I don't think we can figure out when all the build scripts finished compiling (multiple crates in the workspace might have one).

Adding @vlad20012, since IntelliJ Rust might be interested in this too.

[bjorn3]

In https://github.com/rust-lang/rust-analyzer/issues/12973#issuecomment-1208162732 I suggested checking NUM_JOBS or PROFILE. Those are only set for build scripts.

[vlad20012]

Hi! Previously we tried to add --only-build-scripts-and-proc-macros option to Cargo https://github.com/rust-lang/cargo/pull/9071. Maybe now someone could push it forward?

This issue is not the only problem with RUSTC_WRAPPER approach. A more common problem that I run into regularly - cargo stops the build immediately if one rustc instance or buildscript instance fails, even if it would be possible to compile other crates that does not depends on the error crate. Consider a project that has a buildscript in its workspace that compiles some C dependency using GCC, and also uses a bunch of proc macros. When a user downloads this project and opens it in an IDE, the IDE invokes cargo check in order to compile buildscripts and proc macros. But if a user misses GCC (or some required external dependency), the buildscript fails. In this case cargo stops immediately and will likely not compile proc macros that does not depends on that builscript and otherwise could be compiled. Ideally, cargo should continue the build of independent crates even if an error occurs. I don't see how this could be achieved with RUSTC_WRAPPER

[bjorn3]

Ideally, cargo should continue the build of independent crates even if an error occurs.

There is the unstable --keep-going flag for that.

[vlad20012]

There is the unstable --keep-going flag for that

Wow, this seems like exactly what we need!

[matklad]

🤔 I think —-keep-going (which we maybe should rename to be consistent with analogous flag of libtest) changes the calculus here significantly. It should allow us to remove the RUSTC_WRAPPER hack, as the main problem why we need that is not performance, but reliability.

[Veykril]

A quick test seems to not really change the times for me when running with and without the wrapper (though that might be because cargo check gets skipped due to caching here), I'd be in favor of just removing the wrapper, as it also makes cargo be confused about diagnostics (in rare cases it seems to succeed cargo checks on crates that have actual errors in them, see https://github.com/rust-lang/rust-analyzer/pull/12808)

[lnicola]

I think it's going to matter a lot for people who disable rust-analyzer.checkOnSave.enable. But we can probably remove or disable it until we get some proper support in Cargo.

[Veykril]

Opened a zulip topic in the t-cargo stream to see if we can work on this from the cargo side of things as our own options here are rather limited and to me it sounds like it would make more sense to just push https://github.com/rust-lang/cargo/issues/7178 forward https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/cargo.20build.20--compile-time-deps.20cargo.237178/near/293136753

[dtolnay]

https://github.com/rust-lang/rust-analyzer/issues/12973#issuecomment-1208162732 sounds the most promising to me. Here are Cargo's build script environment variables: https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-sets-for-build-scripts. The criteria for selecting one for this purpose are:

1. It has to be set by Cargo during build script execution
2. It must always be set during build script execution, e.g. unlike CARGO_CFG_UNIX which is set on unix only
3. It must not be set during compilation of crates or the build scripts
4. There should be no reason for someone to set it globally for a whole build, so in particular it should not be one of the environment variables read by Cargo
5. There should be no reason for it to appear coincidentally in someone's environment, like HOST

These are the ones that I see fit the bill:

• CARGO_CFG_TARGET_ARCH
• CARGO_CFG_TARGET_ENDIAN
• CARGO_CFG_TARGET_FAMILY
• CARGO_CFG_TARGET_OS
• CARGO_CFG_TARGET_POINTER_WIDTH
• CARGO_CFG_TARGET_VENDOR
• NUM_JOBS
• OPT_LEVEL
• PROFILE

So the plan would be for rustc_wrapper.rs to do its early return thing only if std::env::var_os("CARGO_CFG_TARGET_ARCH").is_none().

[Veykril]

Picking one of the CARGO_CFG_TARGET_* ones seems most reasonable, I could see someone setting one of the later 3 on their system for something unrelated given their short names (though rather unlikely still of course).

[Veykril]

PR https://github.com/rust-lang/rust-analyzer/pull/13010 is up, though I can't test whether it works as I am actually not running into the issue with anyhow for some reason.

[sehz]

disabling the RUSTC_WRAPPER flag fixed the crash issue

[dtolnay]

This has been fixed by https://github.com/rust-lang/rust-analyzer/pull/13010.