Stabilize target-applies-to-host feature.

[jameshilliard]

Details: #9453, #9322

[rust-highfive]

r? @alexcrichton

(rust-highfive has picked a reviewer for you, use r? to override)

[alexcrichton]

#9322 is quite sprawling and I may not have also made this clear enough, but I personally do not think that this is a candidate for stabilization until the issue fixed in https://github.com/rust-lang/cargo/pull/9634 is resolved. Otherwise this flag I don't believe works with cargo build since the same configuration will be used for all units.

[jameshilliard]

Otherwise this flag I don't believe works with cargo build since the same configuration will be used for all units.

I think it works well enough for cross builds since the host build always ends up being used with a non-triple target directory and the target with the triple target directory, they shouldn't really conflict there at least.

[jameshilliard]

FYI I've integrated the target-applies-to-host fix into the buildroot ripgrep package build using the recently released stable 1.54.0 toolchain to fix the build script failure and it seems to work fine so far, this should increase the testing coverage for this feature a good bit. I don't think #9634 is required for properly supporting this flag, however #9634 may be needed for some complex multi-target scenarios I think(buildroot does not use cargo's multi-target build capability so I don't expect we will hit issues there).

[alexcrichton]

I, uh, stand by my previous comment. I don't dispute that the feature works for --target-based builds nor that it's useful for those builds. I want this fixed for cargo build before we stabilize it.

[jameshilliard]

I don't dispute that the feature works for --target-based builds nor that it's useful for those builds. I want this fixed for cargo build before we stabilize it.

What issue are we hitting there? Shared dependencies conflicting?

[bors]

:umbrella: The latest upstream changes (presumably #9613) made this pull request unmergeable. Please resolve the merge conflicts.

[nitsky]

Hi, thank you to everyone here for working on this feature. What is the planned behavior for cargo build without specifying a --target? At the moment I observe that with target-applies-to-host = false, cargo build of a bin ignores both the host and target configuration. In my particular usage, the ideal behavior would be to use the host configuration not just for the build scripts and tests but for the bin as well, because I consider cargo build to be "targeting the host". Is this what is planned? Thank you!

[jameshilliard]

What is the planned behavior for cargo build without specifying a --target?

I think it eventually was supposed to be to have the target config only to apply to target artifact builds, currently they apply to host and target builds unless target-applies-to-host = false is set.

It seems progress has stalled a bit here however since it seems nobody doing review really has all that good of an understanding of this area of the codebase.

At the moment I observe that with target-applies-to-host = false, cargo build of a bin ignores both the host and target configuration.

Are you setting both -Zhost-config and -Ztarget-applies-to-host as well? The bin should always use the target configuration either way, the host config is only ever used for host targets like build-scripts and requires the appropriate nightly flags to enable.

In my particular usage, the ideal behavior would be to use the host configuration not just for the build scripts and tests but for the bin as well, because I consider cargo build to be "targeting the host". Is this what is planned?

No, I don't think that makes much sense, the target bin should always IMO pick up its build configuration from the [target] config, even if the target is the same as the host. The purpose of the host configuration is to allow one to configure settings for the build tools(like build-script-build) for the host, it's not intended to be used to configure anything for target artifact builds(whether they will run on the build host or a different system).

[alex-berger]

Any progress on this? We rely on this feature for cross compiling and I do not feel comfortable with having to set RUSTC_BOOTSTRAP=1 to use this feature from stable Rust. That said, this feature works quite well for us so far.

[jameshilliard]

I do not feel comfortable with having to set RUSTC_BOOTSTRAP=1 to use this feature from stable Rust.

IMO setting __CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS is probably safer, since it's only picked up/used in essentially one place(which is sufficient to enable this one nightly only feature we need) while RUSTC_BOOTSTRAP=1 is getting used in other places including in rustc.

This seems to have fixed things for us: https://github.com/buildroot/buildroot/blob/ac573c55aac3cbb4257f5388c91321c81095c654/package/pkg-cargo.mk#L26-L50

Be aware that hacks to enable nightly features like this are likely to break, we can manage this easily since we pin our rust toolchain versions(we do this with our non-rust toolchains as well to various degrees) and have auto-builders that would likely catch any related regressions quickly, we just really don't want to be pinning rust/cargo toolchains to nightly toolchains since we really only want to enable stable features(this one is just not really optional in practice as even basic x86_64 cross compilation breaks without it).

[saurik]

FWIW, I also ran out of workarounds a month ago after I wanted to upgrade something (either Rust/Cargo or one of the couple Rust libraries I still haven't ported away from... I don't remember the situation, but it involved yet another place where flags were being incorrectly applied to the wrong part of the build process) and have been able to switch to this feature and it is working great (and I will note that even using __CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS this is way more stable and less likely to break than all of my prior hacks that were required to make Rust even semi-usable for serious compilation; I also, though, have somewhat controlled toolchain versions). https://github.com/OrchidTechnologies/orchid/commit/3ced4033c514dd1a98a70eab500c0f2a09ed2221

[bors]

:umbrella: The latest upstream changes (presumably #10395) made this pull request unmergeable. Please resolve the merge conflicts.

[jameshilliard]

rebased

[alexcrichton]

Thanks for the PR, but I'm going to be stepping down from the Cargo team so I'm going to un-assign myself from this. The Cargo team will help review this when they get a chance.

[bors]

:umbrella: The latest upstream changes (presumably #10868) made this pull request unmergeable. Please resolve the merge conflicts.

[jameshilliard]

rebased

[jameshilliard]

@joshtriplett Does this look good to merge?

It would be nice if cross compilation build systems could drop hacks like __CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS="nightly".

Having unstable env variables as a hard requirement for a functional cross compilation environment isn't exactly ideal.

[bors]

:umbrella: The latest upstream changes (presumably #13409) made this pull request unmergeable. Please resolve the merge conflicts.

[sdir]

@jameshilliard conflicts.

The config: &Config change gctx: &GlobalContext from #13759 in function get_target_applies_to_host

[gmorenz]

I don't dispute that the feature works for --target-based builds nor that it's useful for those builds. I want this fixed for cargo build before we stabilize it.

What issue are we hitting there? Shared dependencies conflicting?

I am building a static binary (not interpreted by ld-linux.so, the kind that ldd prints not a dynamic executable for). As a result I need -C relocation-model=static. Proc macros cannot be built with -C relocation-model=static, because they are dylibs. cargo build --target x86_64-unknown-linux-gnu works to build my project because it builds the executable with the relocation-model passed by rustflags, and the proc macros with the default rust flags. This flag, as designed, should do the same (if I understand the proposal correctly). This flag, as implemented, does not, because it builds the binary with the host-flags so I am once again in the situation where the proc-macros and the final binary cannot have different relocation-models.

I believe that if this is stabilized before fixing this issue, it would be backwards incompatible to change. Moreover it would move the situation from we have two ways things are built (cargo build --target <host> and cargo build) to we have three ways (those two, and a new third way where host flags are applied to everything), instead of simplifying things by making this flag be a way to reduce the two ways of building things into one way.

[gmorenz]

@rustbot label: +Z-target-applies-to-host

[gmorenz]

There's some important discussion about eventual stabilization in this review comment, and the discussion on the main thread of that PR after the review. To summarize:

Should target-applies-to-host = true do what current cargo does, or what the name naively implies it does, because those turn out to be different things. If it does what the name naively implies it does, do we add a third option (e.g. = "mixed") that does what cargo currently does?

Because of complexity Alex Crichton (not tagging as they have chosen to step down from this) and @jonhoo appear to have been leaning towards "stabilize directly to target-applies-to-host = false as a breaking change, potentially not even exposing an option to opt in to other behavior". Gating stabilization on fixing the bug that "that cargo build is "all host" rather than the expected "some host some target" builds", and host-config. I'm unclear if the further discussion that they suggested was necessary ever actually occurred.

[saurik]

@gmorenz It isn't clear to me, in the three-state idea, why anyone would ever use =true: new code should use =false, and some older scripts might temporarily need =mixed, but no one should ever be using =true? I can see renaming the flag, but it seems kind of silly and will just cause even more churn, given that I would expect that no one should ever opt into that behavior: at least with this name, the few people who might care are likely to find these issues and the now years of discussion on various forums with people talking about this flag as fixing the current broken behavior and explaining why you might want to turn this legacy behavior back on.

I would then document the flag, explaining merely that =true opts into quickly/legacy behavior. I frankly would not try to document that behavior, as it was, is, and will continue to be nonsensical and the goal of keeping it in the codebase only makes sense for people whose code already happened to work before, and those people do not need documentation of the behavior that is already confusing: they just need to be told to stop relying on it. I would also state that the flag might go away in the future, and so no one should start writing new code that uses =true or ever consider turning it on unless they are doing so to temporarily work around an issue.

At that point, the name seems almost irrelevant. As far as I can tell, all of this discussion about a tri-state flag happened due to a misunderstanding of the intended future of the flag, not due to any real interest in actually having =true be a supported configuration. As such, it seems silly to implement the tri-state feature merely to make the name make more sense, but it also seems silly to lose sleep over the name given that the flag is only to be used in a way where the name can be confusing in exceptional scenarios by people who already rely on whatever that behavior might be, and the flag is coming into existence with the goal of then rapidly phasing out.

Remember: in its current state, cargo does not work! The most common way for serious integrators to build software is using host==target cross-compiling, and so an increasingly-large number of projects are turning to the crazy workaround of using the DO_NOT_USE_THIS nightly override env and are already pushing this flag into production as without it cargo simply doesn't work. It would behoove everyone to push this flag--whatever the default--sooner rather than later, as it isn't clear to me that the people whose code right now could break even if the default were changed should matter more than the people who are using this flag to fix the bug.

[gmorenz]

@saurik I'm sympathetic to that argument with regards to =true, the docs definitely need updating with regards to it though, and the alternative of "stabilize =false behavior without a flag" seems like something whatever cargo team member that eventually picks this up should see and consider.

As currently implemented it seems to me that cargo is just as broken with =false as without it. Either way it's building the artifacts differently depending on whether you're cross compiling or not, it's just the difference between "host artifacts get extra rust-flags" and "target artifacts are missing rust-flags". I'm looking at the diff you linked above for instance, and while I haven't tried to build your codebase it's my understanding that line 135(incoming)/137(outgoing) is only applied when you are cross compiling (passing --target x) despite having turned on target-applies-to-host = false.

I've started poking at the cargo code in an attempt to fix the flag, but no promises.

Edit: Apparently rust-lang/rust wanted target-applies-to-host=true (not mixed) at one point: https://github.com/rust-lang/rust/issues/94556

[gmorenz]

@saurik - Correcting my previous comment, since that build script is always passing a --target flag, my understanding of the current behavior is that target-applies-to-host=false should be doing nothing (both as documented and implemented), so I probably misunderstand something.

What is the difference in behavior that you see by enabling the flag in that context?

Edit: It's that linker arguments isn't used to build host artifacts