rust-lang
extend the "if-unchanged" logic for compiler builds
Implements the first item from this tracking issue.
In short, we want to make "if-unchanged" logic to check for changes outside of certain allowed directories, and this PR implements that. There are exceptions for some paths (such as "src") but the overall goal of this PR is to rely on allowed paths rather than denied paths (which was the case with previous approach).
See https://github.com/rust-lang/rust/issues/131658 for more context.
r? @Mark-Simulacrum
rustbot has assigned @Mark-Simulacrum. They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.
Use r? to explicitly pick a reviewer
This PR modifies src/bootstrap/src/core/config.
If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.
Would it make sense to compute the set of files that may not be changed for download-rustc to work based on the .d files in build/host/stage0-rustc? This is what cargo uses for determining if a crate should be recompiled. It doesn't list the Cargo.toml files though. For the aggregated .d file generated by cargo in build/host/stage0-rustc/<host>/release/rustc-main.d I think it would make sense to add Cargo.toml there too. These aggregated .d files are meant for integration with external build systems AFAIK, which need to track Cargo.toml changes too.
RUSTC_IF_UNCHANGED_DENIED_SUBPATHS is removed to rely on allowed paths only. Also, "src/bootstrap" is not included in the allowed paths on purpose since it can affect compiler builds.
@rustbot ready
Would it make sense to compute the set of files that may not be changed for download-rustc to work based on the .d files in build/host/stage0-rustc?
We are comparing against the nightly compiler, "stage0-rustc" wouldn't help.
:umbrella: The latest upstream changes (presumably #131934) made this pull request unmergeable. Please resolve the merge conflicts.
RUSTC_IF_UNCHANGED_DENIED_SUBPATHSis removed to rely on allowed paths only. Also, "src/bootstrap" is not included in the allowed paths on purpose since it can affect compiler builds.@rustbot ready
Nevermind, allowed src/bootstrap again because modifications on bootstrap could break CI rustc usage and we wouldn't catch that if we used the in-tree compiler. The CI failure would only show up the next day.
:umbrella: The latest upstream changes (presumably #132070) made this pull request unmergeable. Please resolve the merge conflicts.
:umbrella: The latest upstream changes (presumably #132215) made this pull request unmergeable. Please resolve the merge conflicts.
:umbrella: The latest upstream changes (presumably #132371) made this pull request unmergeable. Please resolve the merge conflicts.
By any chance, could you prioritize the review for this PR a bit ? @Mark-Simulacrum
:umbrella: The latest upstream changes (presumably #132815) made this pull request unmergeable. Please resolve the merge conflicts.
:umbrella: The latest upstream changes (presumably #132846) made this pull request unmergeable. Please resolve the merge conflicts.
Applied nits and reverted first commit from https://github.com/rust-lang/rust/pull/132852.
cc @RalfJung
FWIW, I ran my git analysis scripts using the current allowlist in this PR. For merge commits from the last 365 days, only around 3.8% would be cacheable. Before the latest change made a few minutes ago, it was 12.4%.
Here is the script (cannot vouch for its correctness): https://gist.github.com/Kobzol/ba9f56e01dd2d21446eba933c55297f6
The run-make-support library was changed
cc @jieyouxu
Some changes occurred in src/tools/compiletest
cc @jieyouxu
Some changes occurred in src/tools/opt-dist
cc @kobzol
FWIW, I ran my git analysis scripts using the current allowlist in this PR. For merge commits from the last 365 days, only around 3.8% would be cacheable. Before the latest change made a few minutes ago, it was 12.4%.
I guess src/tools is playing serious role on that.
What about PR CIs btw? I think CI rustc brings some serious advantage on that.
The job mingw-check-tidy failed! Check out the build log: (web) (plain)
Click to see the possible cause of the failure (guessed by this bot)
COPY host-x86_64/mingw-check/validate-toolstate.sh /scripts/
COPY host-x86_64/mingw-check/validate-error-codes.sh /scripts/
# NOTE: intentionally uses python2 for x.py so we can test it still works.
# validate-toolstate only runs in our CI, so it's ok for it to only support python3.
ENV SCRIPT TIDY_PRINT_DIFF=1 python2.7 ../x.py test \
--stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
# pip-compile --allow-unsafe --generate-hashes reuse-requirements.in
---
#13 3.009 Building wheels for collected packages: reuse
#13 3.010 Building wheel for reuse (pyproject.toml): started
#13 3.261 Building wheel for reuse (pyproject.toml): finished with status 'done'
#13 3.263 Created wheel for reuse: filename=reuse-4.0.3-cp310-cp310-manylinux_2_35_x86_64.whl size=132720 sha256=026f3bb0f1aa8090b861fd0a0939cb1a782396d84c8aab7875096557d637a0f6
#13 3.263 Stored in directory: /tmp/pip-ephem-wheel-cache-fhh61fya/wheels/3d/8d/0a/e0fc6aba4494b28a967ab5eaf951c121d9c677958714e34532
#13 3.266 Installing collected packages: boolean-py, binaryornot, tomlkit, reuse, python-debian, markupsafe, license-expression, jinja2, chardet, attrs
#13 3.670 Successfully installed attrs-23.2.0 binaryornot-0.4.4 boolean-py-4.0 chardet-5.2.0 jinja2-3.1.4 license-expression-30.3.0 markupsafe-2.1.5 python-debian-0.1.49 reuse-4.0.3 tomlkit-0.13.0
#13 3.670 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#13 4.213 Collecting virtualenv
#13 4.213 Collecting virtualenv
#13 4.277 Downloading virtualenv-20.27.1-py3-none-any.whl (3.1 MB)
#13 4.407 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.1/3.1 MB 24.4 MB/s eta 0:00:00
#13 4.468 Collecting filelock<4,>=3.12.2
#13 4.476 Downloading filelock-3.16.1-py3-none-any.whl (16 kB)
#13 4.499 Collecting distlib<1,>=0.3.7
#13 4.507 Downloading distlib-0.3.9-py2.py3-none-any.whl (468 kB)
#13 4.522 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 469.0/469.0 KB 38.8 MB/s eta 0:00:00
#13 4.560 Collecting platformdirs<5,>=3.9.1
#13 4.569 Downloading platformdirs-4.3.6-py3-none-any.whl (18 kB)
#13 4.652 Installing collected packages: distlib, platformdirs, filelock, virtualenv
#13 4.832 Successfully installed distlib-0.3.9 filelock-3.16.1 platformdirs-4.3.6 virtualenv-20.27.1
#13 DONE 4.9s
#14 [7/8] COPY host-x86_64/mingw-check/validate-toolstate.sh /scripts/
#14 DONE 0.0s
The job mingw-check failed! Check out the build log: (web) (plain)
Click to see the possible cause of the failure (guessed by this bot)
#16 2.904 Building wheels for collected packages: reuse
#16 2.905 Building wheel for reuse (pyproject.toml): started
#16 3.148 Building wheel for reuse (pyproject.toml): finished with status 'done'
#16 3.149 Created wheel for reuse: filename=reuse-4.0.3-cp310-cp310-manylinux_2_35_x86_64.whl size=132720 sha256=026f3bb0f1aa8090b861fd0a0939cb1a782396d84c8aab7875096557d637a0f6
#16 3.149 Stored in directory: /tmp/pip-ephem-wheel-cache-5gl20856/wheels/3d/8d/0a/e0fc6aba4494b28a967ab5eaf951c121d9c677958714e34532
#16 3.152 Installing collected packages: boolean-py, binaryornot, tomlkit, reuse, python-debian, markupsafe, license-expression, jinja2, chardet, attrs
#16 3.554 Successfully installed attrs-23.2.0 binaryornot-0.4.4 boolean-py-4.0 chardet-5.2.0 jinja2-3.1.4 license-expression-30.3.0 markupsafe-2.1.5 python-debian-0.1.49 reuse-4.0.3 tomlkit-0.13.0
#16 3.554 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#16 DONE 3.7s
---
Compiling build_helper v0.1.0 (/checkout/src/build_helper)
error: couldn't read `/checkout/src/build_helper/src/../../../stage0`: No such file or directory (os error 2)
##[error] --> /checkout/src/build_helper/src/stage0_parser.rs:28:26
|
28 | let stage0_content = include_str!("../../../stage0");
|
= note: this error originates in the macro `include_str` (in Nightly builds, run with -Z macro-backtrace for more info)
help: there is a file with the same name in a different directory
|
|
28 | let stage0_content = include_str!("../../stage0");
error: could not compile `build_helper` (lib) due to 1 previous error
failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
Build completed unsuccessfully in 0:00:23
Build completed unsuccessfully in 0:00:23
make: *** [Makefile:98: prepare] Error 1
Command failed. Attempt 2/5:
##[group]Building bootstrap
Compiling build_helper v0.1.0 (/checkout/src/build_helper)
error: couldn't read `/checkout/src/build_helper/src/../../../stage0`: No such file or directory (os error 2)
##[error] --> /checkout/src/build_helper/src/stage0_parser.rs:28:26
|
28 | let stage0_content = include_str!("../../../stage0");
|
= note: this error originates in the macro `include_str` (in Nightly builds, run with -Z macro-backtrace for more info)
help: there is a file with the same name in a different directory
|
|
28 | let stage0_content = include_str!("../../stage0");
error: could not compile `build_helper` (lib) due to 1 previous error
failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
Build completed unsuccessfully in 0:00:00
Build completed unsuccessfully in 0:00:00
make: *** [Makefile:98: prepare] Error 1
Command failed. Attempt 3/5:
##[group]Building bootstrap
Compiling build_helper v0.1.0 (/checkout/src/build_helper)
error: couldn't read `/checkout/src/build_helper/src/../../../stage0`: No such file or directory (os error 2)
##[error] --> /checkout/src/build_helper/src/stage0_parser.rs:28:26
|
28 | let stage0_content = include_str!("../../../stage0");
|
= note: this error originates in the macro `include_str` (in Nightly builds, run with -Z macro-backtrace for more info)
help: there is a file with the same name in a different directory
|
|
28 | let stage0_content = include_str!("../../stage0");
error: could not compile `build_helper` (lib) due to 1 previous error
failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
Build completed unsuccessfully in 0:00:00
Build completed unsuccessfully in 0:00:00
make: *** [Makefile:98: prepare] Error 1
Command failed. Attempt 4/5:
##[group]Building bootstrap
Compiling build_helper v0.1.0 (/checkout/src/build_helper)
error: couldn't read `/checkout/src/build_helper/src/../../../stage0`: No such file or directory (os error 2)
##[error] --> /checkout/src/build_helper/src/stage0_parser.rs:28:26
|
28 | let stage0_content = include_str!("../../../stage0");
|
= note: this error originates in the macro `include_str` (in Nightly builds, run with -Z macro-backtrace for more info)
help: there is a file with the same name in a different directory
|
|
28 | let stage0_content = include_str!("../../stage0");
error: could not compile `build_helper` (lib) due to 1 previous error
failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
Build completed unsuccessfully in 0:00:00
Build completed unsuccessfully in 0:00:00
make: *** [Makefile:98: prepare] Error 1
Command failed. Attempt 5/5:
##[group]Building bootstrap
Compiling build_helper v0.1.0 (/checkout/src/build_helper)
error: couldn't read `/checkout/src/build_helper/src/../../../stage0`: No such file or directory (os error 2)
##[error] --> /checkout/src/build_helper/src/stage0_parser.rs:28:26
|
28 | let stage0_content = include_str!("../../../stage0");
|
= note: this error originates in the macro `include_str` (in Nightly builds, run with -Z macro-backtrace for more info)
help: there is a file with the same name in a different directory
|
|
28 | let stage0_content = include_str!("../../stage0");
error: could not compile `build_helper` (lib) due to 1 previous error
failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
Build completed unsuccessfully in 0:00:00
The job mingw-check-tidy failed! Check out the build log: (web) (plain)
Click to see the possible cause of the failure (guessed by this bot)
COPY host-x86_64/mingw-check/validate-toolstate.sh /scripts/
COPY host-x86_64/mingw-check/validate-error-codes.sh /scripts/
# NOTE: intentionally uses python2 for x.py so we can test it still works.
# validate-toolstate only runs in our CI, so it's ok for it to only support python3.
ENV SCRIPT TIDY_PRINT_DIFF=1 python2.7 ../x.py test \
--stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
# pip-compile --allow-unsafe --generate-hashes reuse-requirements.in
---
#13 2.984 Building wheels for collected packages: reuse
#13 2.985 Building wheel for reuse (pyproject.toml): started
#13 3.244 Building wheel for reuse (pyproject.toml): finished with status 'done'
#13 3.246 Created wheel for reuse: filename=reuse-4.0.3-cp310-cp310-manylinux_2_35_x86_64.whl size=132720 sha256=026f3bb0f1aa8090b861fd0a0939cb1a782396d84c8aab7875096557d637a0f6
#13 3.246 Stored in directory: /tmp/pip-ephem-wheel-cache-tdsbon6m/wheels/3d/8d/0a/e0fc6aba4494b28a967ab5eaf951c121d9c677958714e34532
#13 3.248 Installing collected packages: boolean-py, binaryornot, tomlkit, reuse, python-debian, markupsafe, license-expression, jinja2, chardet, attrs
#13 3.659 Successfully installed attrs-23.2.0 binaryornot-0.4.4 boolean-py-4.0 chardet-5.2.0 jinja2-3.1.4 license-expression-30.3.0 markupsafe-2.1.5 python-debian-0.1.49 reuse-4.0.3 tomlkit-0.13.0
#13 3.660 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#13 4.222 Collecting virtualenv
#13 4.222 Collecting virtualenv
#13 4.271 Downloading virtualenv-20.27.1-py3-none-any.whl (3.1 MB)
#13 4.333 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.1/3.1 MB 52.1 MB/s eta 0:00:00
#13 4.372 Collecting distlib<1,>=0.3.7
#13 4.376 Downloading distlib-0.3.9-py2.py3-none-any.whl (468 kB)
#13 4.384 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 469.0/469.0 KB 76.0 MB/s eta 0:00:00
#13 4.420 Collecting filelock<4,>=3.12.2
#13 4.424 Downloading filelock-3.16.1-py3-none-any.whl (16 kB)
#13 4.457 Collecting platformdirs<5,>=3.9.1
#13 4.460 Downloading platformdirs-4.3.6-py3-none-any.whl (18 kB)
#13 4.543 Installing collected packages: distlib, platformdirs, filelock, virtualenv
#13 4.725 Successfully installed distlib-0.3.9 filelock-3.16.1 platformdirs-4.3.6 virtualenv-20.27.1
#13 DONE 4.8s
#14 [7/8] COPY host-x86_64/mingw-check/validate-toolstate.sh /scripts/
#14 DONE 0.0s
---
DirectMap4k: 335808 kB
DirectMap2M: 7004160 kB
DirectMap1G: 11534336 kB
##[endgroup]
Executing TIDY_PRINT_DIFF=1 python2.7 ../x.py test --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
+ TIDY_PRINT_DIFF=1 python2.7 ../x.py test --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
Finished `dev` profile [unoptimized] target(s) in 0.04s
##[endgroup]
downloading https://static.rust-lang.org/dist/2024-10-16/rustfmt-nightly-x86_64-unknown-linux-gnu.tar.xz
extracting /checkout/obj/build/cache/2024-10-16/rustfmt-nightly-x86_64-unknown-linux-gnu.tar.xz to /checkout/obj/build/x86_64-unknown-linux-gnu/rustfmt
---
Compiling tidy v0.1.0 (/checkout/src/tools/tidy)
Finished `release` profile [optimized] target(s) in 31.97s
##[endgroup]
fmt check
Diff in /checkout/src/bootstrap/src/core/config/config.rs:39:
/// is added here, it will cause bootstrap to skip necessary rebuilds, which may lead to risky results.
/// For example, "src/bootstrap" should never be included in this list as it plays a crucial role in the
/// final output/compiler, which can be significantly affected by changes made to the bootstrap sources.
-pub(crate) const RUSTC_IF_UNCHANGED_ALLOWED_PATHS: &[&str] = &[
- ":!src/tools",
- ":!tests",
- ":!triagebot.toml",
-];
+pub(crate) const RUSTC_IF_UNCHANGED_ALLOWED_PATHS: &[&str] =
+ &[":!src/tools", ":!tests", ":!triagebot.toml"];
macro_rules! check_ci_llvm {
($name:expr) => {
($name:expr) => {
fmt error: Running `"/checkout/obj/build/x86_64-unknown-linux-gnu/rustfmt/bin/rustfmt" "--config-path" "/checkout" "--edition" "2021" "--unstable-features" "--skip-children" "--check" "/checkout/tests/mir-opt/dead-store-elimination/cycle.rs" "/checkout/tests/mir-opt/dead-store-elimination/place_mention.rs" "/checkout/tests/mir-opt/dead-store-elimination/provenance_soundness.rs" "/checkout/tests/mir-opt/issue_91633.rs" "/checkout/tests/mir-opt/issue_72181_1.rs" "/checkout/tests/mir-opt/derefer_inline_test.rs" "/checkout/tests/mir-opt/simplify_if.rs" "/checkout/tests/mir-opt/lower_slice_len.rs" "/checkout/tests/mir-opt/derefer_complex_case.rs" "/checkout/tests/mir-opt/issue_41888.rs" "/checkout/src/bootstrap/src/core/build_steps/test.rs" "/checkout/src/bootstrap/src/core/build_steps/run.rs" "/checkout/src/bootstrap/src/core/build_steps/compile.rs" "/checkout/src/bootstrap/src/core/build_steps/vendor.rs" "/checkout/src/bootstrap/src/core/build_steps/setup.rs" "/checkout/src/bootstrap/src/core/build_steps/toolstate.rs" "/checkout/src/bootstrap/src/core/build_steps/llvm.rs" "/checkout/src/bootstrap/src/core/build_steps/clippy.rs" "/checkout/src/bootstrap/src/core/build_steps/perf.rs" "/checkout/src/bootstrap/src/core/build_steps/install.rs" "/checkout/src/bootstrap/src/core/build_steps/synthetic_targets.rs" "/checkout/src/bootstrap/src/core/build_steps/suggest.rs" "/checkout/src/bootstrap/src/core/build_steps/tool.rs" "/checkout/src/bootstrap/src/core/build_steps/dist.rs" "/checkout/src/bootstrap/src/core/build_steps/clean.rs" "/checkout/src/bootstrap/src/core/build_steps/doc.rs" "/checkout/src/bootstrap/src/core/build_steps/check.rs" "/checkout/src/bootstrap/src/core/build_steps/gcc.rs" "/checkout/src/bootstrap/src/core/build_steps/setup/tests.rs" "/checkout/src/bootstrap/src/core/build_steps/format.rs" "/checkout/src/bootstrap/src/core/build_steps/mod.rs" "/checkout/src/bootstrap/src/core/config/config.rs" "/checkout/src/bootstrap/src/core/config/flags.rs" "/checkout/src/bootstrap/src/core/config/tests.rs" "/checkout/src/bootstrap/src/core/config/mod.rs" "/checkout/src/bootstrap/src/core/sanity.rs" "/checkout/src/bootstrap/src/core/download.rs" "/checkout/src/bootstrap/src/core/builder/cargo.rs" "/checkout/src/bootstrap/src/core/builder/tests.rs" "/checkout/src/bootstrap/src/core/builder/mod.rs" "/checkout/src/bootstrap/src/core/metadata.rs" "/checkout/src/bootstrap/src/core/mod.rs" "/checkout/src/bootstrap/src/utils/cache.rs" "/checkout/src/bootstrap/src/utils/metrics.rs" "/checkout/src/bootstrap/src/utils/shared_helpers/tests.rs" "/checkout/src/bootstrap/src/utils/cc_detect.rs" "/checkout/src/bootstrap/src/utils/tarball.rs" "/checkout/src/bootstrap/src/utils/helpers/tests.rs" "/checkout/src/bootstrap/src/utils/change_tracker/tests.rs" "/checkout/src/bootstrap/src/utils/render_tests.rs" "/checkout/src/bootstrap/src/utils/shared_helpers.rs" "/checkout/src/bootstrap/src/utils/change_tracker.rs" "/checkout/src/bootstrap/src/utils/helpers.rs" "/checkout/src/bootstrap/src/utils/channel.rs" "/checkout/src/bootstrap/src/utils/job.rs" "/checkout/src/bootstrap/src/utils/exec.rs" "/checkout/src/bootstrap/src/utils/mod.rs" "/checkout/src/bootstrap/src/bin/main.rs" "/checkout/src/bootstrap/src/bin/sccache-plus-cl.rs" "/checkout/src/bootstrap/src/bin/rustc.rs" "/checkout/src/bootstrap/src/bin/rustdoc.rs" "/checkout/src/bootstrap/src/lib.rs" "/checkout/src/rustc-std-workspace/rustc-std-workspace-alloc/src/lib.rs" "/checkout/tests/mir-opt/dead-store-elimination/call_arg_copy.rs"` failed.
If you're running `tidy`, try again with `--bless`. Or, if you just want to format code, run `./x.py fmt` instead.
local time: Mon Nov 11 06:51:26 UTC 2024
network time: Mon, 11 Nov 2024 06:51:26 GMT
##[error]Process completed with exit code 1.
Post job cleanup.
This PR modifies config.example.toml.
If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.
I haven't dug into the logic here, so my concerns were entirely about the list of folders that considered "not affecting the compiler". That one has been resolved.
