rustdoc: Add support for --remap-path-prefix

[edward-shen]

Adds --remap-path-prefix as an unstable option. This is implemented to mimic the behavior of rustc's --remap-path-prefix.

This flag similarly takes in two paths, a prefix to replace and a replacement string.

This is useful for build tools (e.g. Buck) other than cargo that can run doc tests.

cc: @dtolnay

[rustbot]

r? @GuillaumeGomez

(rustbot has picked a reviewer for you, use r? to override)

[rust-log-analyzer]

The job mingw-check failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

configure: rust.debug-assertions := True
configure: rust.overflow-checks := True
configure: llvm.assertions      := True
configure: dist.missing-tools   := True
configure: build.configure-args := ['--enable-sccache', '--disable-manage-submodu ...
configure: writing `config.toml` in current directory
configure: 
configure: run `python /checkout/x.py --help`
Attempting with retry: make prepare
---
  |
9 | use rustc_data_structures::sync::Lrc;
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = note: `-D unused-imports` implied by `-D warnings`

error: unused imports: `FilePathMapping`, `SourceMap`
   |
   |
22 | use rustc_span::source_map::{FilePathMapping, SourceMap};

error: could not compile `rustdoc` due to 2 previous errors
Build completed unsuccessfully in 0:01:42

[rust-log-analyzer]

The job x86_64-gnu-llvm-13 failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

Check compiletest suite=rustdoc-ui mode=ui (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)

running 200 tests
.........................................................i.............................. 88/200
...................................................................FF.......F........... 176/200
Some tests failed in compiletest suite=rustdoc-ui mode=ui host=x86_64-unknown-linux-gnu target=x86_64-unknown-linux-gnu
failures:


---- [ui] tests/rustdoc-ui/remap-path-prefix-failed-doctest-output.rs stdout ----

1 
2 running 1 test
- test remapped_path/remap-path-prefix-failed-doctest-output.rs - SomeStruct (line 10) ... FAILED
- test remapped_path/remap-path-prefix-failed-doctest-output.rs - SomeStruct (line 10) ... FAILED
+ test $DIR/remap-path-prefix-failed-doctest-output.rs - SomeStruct (line 10) ... FAILED
5 failures:
6 


- ---- remapped_path/remap-path-prefix-failed-doctest-output.rs - SomeStruct (line 10) stdout ----
+ ---- $DIR/remap-path-prefix-failed-doctest-output.rs - SomeStruct (line 10) stdout ----
8 Test executable failed (exit status: 101).
10 stderr:


- thread 'main' panicked at 'oh no', remapped_path/remap-path-prefix-failed-doctest-output.rs:3:1
+ thread 'main' panicked at 'oh no', $DIR/remap-path-prefix-failed-doctest-output.rs:3:1
13 
14 

15 
15 
16 failures:
-     remapped_path/remap-path-prefix-failed-doctest-output.rs - SomeStruct (line 10)
+     $DIR/remap-path-prefix-failed-doctest-output.rs - SomeStruct (line 10)
19 test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in $TIME
20 



The actual stdout differed from the expected stdout.
Actual stdout saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/remap-path-prefix-failed-doctest-output/remap-path-prefix-failed-doctest-output.stdout
To update references, rerun the tests and pass the `--bless` flag
To only update this specific test, also pass `--test-args remap-path-prefix-failed-doctest-output.rs`

error: 1 errors occurred comparing output.
status: exit status: 101
command: RUST_BACKTRACE="0" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustdoc" "/checkout/tests/rustdoc-ui/remap-path-prefix-failed-doctest-output.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "--json" "future-incompat" "-Ccodegen-units=1" "-Zui-testing" "-Zsimulate-remapped-rust-src-base=/rustc/FAKE_PREFIX" "-Ztranslate-remapped-path-to-local-path=no" "-Zdeduplicate-diagnostics=no" "-Cstrip=debuginfo" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/remap-path-prefix-failed-doctest-output" "-Cdebuginfo=0" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/remap-path-prefix-failed-doctest-output/auxiliary" "--test" "-Z" "unstable-options" "--remap-path-prefix=tests/rustdoc-ui=remapped_path" "--test-args" "--test-threads=1"
running 1 test
test /checkout/tests/rustdoc-ui/remap-path-prefix-failed-doctest-output.rs - SomeStruct (line 10) ... FAILED

failures:
failures:

---- /checkout/tests/rustdoc-ui/remap-path-prefix-failed-doctest-output.rs - SomeStruct (line 10) stdout ----
Test executable failed (exit status: 101).

stderr:
thread 'main' panicked at 'oh no', /checkout/tests/rustdoc-ui/remap-path-prefix-failed-doctest-output.rs:3:1



failures:
---
diff of stdout:

1 
2 running 1 test
- test remapped_path/remap-path-prefix-invalid-doctest.rs - SomeStruct (line 10) ... FAILED
+ test $DIR/remap-path-prefix-invalid-doctest.rs - SomeStruct (line 10) ... FAILED
5 failures:
6 


- ---- remapped_path/remap-path-prefix-invalid-doctest.rs - SomeStruct (line 10) stdout ----
+ ---- $DIR/remap-path-prefix-invalid-doctest.rs - SomeStruct (line 10) stdout ----
8 error: expected one of `!`, `.`, `::`, `;`, `?`, `{`, `}`, or an operator, found `is`
-   --> remapped_path/remap-path-prefix-invalid-doctest.rs:11:6
10    |
11 LL | this is not real code
12    |      ^^ expected one of 8 possible tokens

---
To only update this specific test, also pass `--test-args remap-path-prefix-invalid-doctest.rs`

error: 1 errors occurred comparing output.
status: exit status: 101
command: RUST_BACKTRACE="0" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustdoc" "/checkout/tests/rustdoc-ui/remap-path-prefix-invalid-doctest.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "--json" "future-incompat" "-Ccodegen-units=1" "-Zui-testing" "-Zsimulate-remapped-rust-src-base=/rustc/FAKE_PREFIX" "-Ztranslate-remapped-path-to-local-path=no" "-Zdeduplicate-diagnostics=no" "-Cstrip=debuginfo" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/remap-path-prefix-invalid-doctest" "-Cdebuginfo=0" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/remap-path-prefix-invalid-doctest/auxiliary" "--test" "-Z" "unstable-options" "--remap-path-prefix=tests/rustdoc-ui=remapped_path" "--test-args" "--test-threads=1"
running 1 test
test /checkout/tests/rustdoc-ui/remap-path-prefix-invalid-doctest.rs - SomeStruct (line 10) ... FAILED

failures:
failures:

---- /checkout/tests/rustdoc-ui/remap-path-prefix-invalid-doctest.rs - SomeStruct (line 10) stdout ----
error: expected one of `!`, `.`, `::`, `;`, `?`, `{`, `}`, or an operator, found `is`
   |
LL | this is not real code
   |      ^^ expected one of 8 possible tokens

---
diff of stdout:

1 
2 running 1 test
- test remapped_path/remap-path-prefix-passed-doctest-output.rs - SomeStruct (line 11) ... ok
+ test $DIR/remap-path-prefix-passed-doctest-output.rs - SomeStruct (line 11) ... ok
5 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in $TIME
6 



The actual stdout differed from the expected stdout.
Actual stdout saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/remap-path-prefix-passed-doctest-output/remap-path-prefix-passed-doctest-output.stdout
To update references, rerun the tests and pass the `--bless` flag
To only update this specific test, also pass `--test-args remap-path-prefix-passed-doctest-output.rs`

error: 1 errors occurred comparing output.
status: exit status: 0
command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustdoc" "/checkout/tests/rustdoc-ui/remap-path-prefix-passed-doctest-output.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "--json" "future-incompat" "-Ccodegen-units=1" "-Zui-testing" "-Zsimulate-remapped-rust-src-base=/rustc/FAKE_PREFIX" "-Ztranslate-remapped-path-to-local-path=no" "-Zdeduplicate-diagnostics=no" "-Cstrip=debuginfo" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/remap-path-prefix-passed-doctest-output" "-Cdebuginfo=0" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/remap-path-prefix-passed-doctest-output/auxiliary" "--test" "-Z" "unstable-options" "--remap-path-prefix=tests/rustdoc-ui=remapped_path" "--test-args" "--test-threads=1"
running 1 test
test /checkout/tests/rustdoc-ui/remap-path-prefix-passed-doctest-output.rs - SomeStruct (line 11) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.22s

[edward-shen]

@rustbot ready

[GuillaumeGomez]

The code looks good to me, but it comes a bit out of the blue so needs the rustdoc team to agree on it (very likely through FCP).

But first, what do you think @rust-lang/rustdoc ?

[edward-shen]

but it comes a bit out of the blue so needs the rustdoc team to agree on it

My bad! I didn't think this would be a substantial change since rustc already had this flag. For future reference, should I have dropped in the zulip before the implementation?

[GuillaumeGomez]

At least opening an issue to explain your needs so it makes it easier for everyone to reach a solution. If this PR ends up being rejected, you'd have worked for nothing. :-/

[edward-shen]

Changes:

• rebased
• Realized I made a function pub for a previous implementation attempt. It no longer needed to be pub so I removed it
• Fixed copy-paste error in ui test.

[edward-shen]

At least opening an issue to explain your needs so it makes it easier for everyone to reach a solution. If this PR ends up being rejected, you'd have worked for nothing. :-/

Got it--I'll be sure to do so in the future. Sorry again, and thank you for the gentle explanation!

[dtolnay]

@GuillaumeGomez As far as I can tell from the reference to Buck (which I am familiar with) the point of --remap-path-prefix for rustdoc is to get diagnostics inside of doctests to refer to the source directory tree of the build, not the build system's directory tree. For example without --remap-path-prefix:

---- buck-out/v2/gen/fbcode/882bd00a4ffb1e1d/tracing/stats/__tracing_stats__/__srcs/src/lib.rs - StatsLayer (line 70) stdout ----
error[E0433]: failed to resolve: use of undeclared type `StatsLayer`
 --> buck-out/v2/gen/fbcode/882bd00a4ffb1e1d/tracing/stats/__tracing_stats__/__srcs/src/lib.rs:71:13
  |
3 | let stats = StatsLayer::new("myservice", 8, false)
  |             ^^^^^^^^^^ not found in this scope
  |
help: consider importing this struct
  |
2 | use tracing_stats::StatsLayer;
  |

The build system would want to use --remap-path-prefix buck-out/v2/gen/fbcode/882bd00a4ffb1e1d/tracing/stats/__tracing_stats__/__srcs/=tracing/stats/ to get:

---- tracing/stats/src/lib.rs - StatsLayer (line 70) stdout ----
error[E0433]: failed to resolve: use of undeclared type `StatsLayer`
 --> tracing/stats/src/lib.rs:71:13
  |
3 | ...

It's a common technique for mature build systems to separate the files that get fed to build steps apart from the files that get touched by the programmers. There are many reasons for this; not all of them apply to every build system that uses this technique.

• Determinism, no race conditions: if a user, or IDE, is mutating the source files during the time that a build is already running, the build system can still be 100% sure exactly which iteration of the input files the output of the build was performed against, and the exact delta to the subsequent build. It's not subject to a race condition between when the compiler ends up reading a file vs when the editor writes the file, which would be outside the build system's ability to inspect (leaving aside some kind of kernel module or sophisticated filesystem or syscall interceptor).

• Hermeticity: the build system can sandbox build steps to keep them from reaching into arbitrary filesystem state that is not tracked by the build system's understanding of the precise inputs of every step.

• Integration of code generators: for source files which are generated by code generators during the course of the build (such as protoc or thriftc or bindgen), the build system can map them to easily understood siblings of the crate root, for example using Buck's mapped_srcs and genrule. This is a lot nicer than Cargo's include!(concat!(env!("OUT_DIR" …)) thing.

• Remote execution: build steps in a large build would typically not run on the same machine as the source files are being written on. In general the filesystem structure on the remote build environment are not identical to the filesystem paths on the developer's machine, but the build system is responsible for mapping between them.

• Distributed caching: sharing build step output artifacts (including diagnostics/warnings, not just rlibs/rmeta/binaries) across different devs, or with a cache which is kept perpetually warm by CI. Again different machines could have different path structure (even different OS, as long as at least 1 is cross-compiling).

Edward or I would be happy to elaborate on any of the above if needed.

[bors]

:umbrella: The latest upstream changes (presumably #108096) made this pull request unmergeable. Please resolve the merge conflicts.

[KittyBorgX]

@rustbot author @edward-shen Hey! It looks like there are some merge conflicts in your PR. Can you sort them out? Once you're done type in @rustbot review here so that the label can be changed to S-waiting-on-review and your PR should be good for a review!

[edward-shen]

Hey @KittyBorgX, thanks for the ping. Work has been busy, so the conflicts message slipped through the cracks.

Rebased and fixed merge conflicts.

@rustbot review

[GuillaumeGomez]

Looks good to me, thanks! Let's start an FCP then.

@rfcbot fcp merge

[rfcbot]

Team member @GuillaumeGomez has proposed to merge this. The next step is review by the rest of the tagged team members:

• [x] @GuillaumeGomez
• [x] @Manishearth
• [x] @Nemo157
• [x] @aDotInTheVoid
• [ ] @camelid
• [ ] @jsha
• [x] @notriddle

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[Nemo157]

What are the "minor adjustments"? Will they in some way affect applying the RFC 3127 changes to rustdoc too?

[notriddle]

@Nemo157

The main place that source paths get embedded is the /src tree (https://github.com/rust-lang/rust/issues/98220 https://github.com/rust-lang/rust/issues/18370).

[edward-shen]

What are the "minor adjustments"? Will they in some way affect applying the RFC 3127 changes to rustdoc too?

I think the minor adjustments phrase is a misnomer. A previous implementation of this required different args to pass to rustdoc to match what rustc would do, but that was changed so that no longer applies.

This change can be accurately described as "rustdoc now passes in --remap-path-prefix args to rustc_session, and now prefers remapped paths if they exist over preferring local paths".

This shouldn't affect RFC 3127. If the implementation for that RFC is done in rust_session, we should be able to pick up the changes for free. My understanding is that we just need to expose similar options for rustdoc and thread those args to the implementation as well.

[bors]

:umbrella: The latest upstream changes (presumably #114905) made this pull request unmergeable. Please resolve the merge conflicts.

[rust-log-analyzer]

The job mingw-check failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

GITHUB_ACTION=__run_7
GITHUB_ACTIONS=true
GITHUB_ACTION_REF=
GITHUB_ACTION_REPOSITORY=
GITHUB_ACTOR=edward-shen
GITHUB_API_URL=https://api.github.com
GITHUB_BASE_REF=master
GITHUB_ENV=/home/runner/work/_temp/_runner_file_commands/set_env_5246b8f5-ce46-4c62-ab23-56b2c17aca23
GITHUB_EVENT_NAME=pull_request
GITHUB_EVENT_NAME=pull_request
GITHUB_EVENT_PATH=/home/runner/work/_temp/_github_workflow/event.json
GITHUB_GRAPHQL_URL=https://api.github.com/graphql
GITHUB_HEAD_REF=edward-shen/rustdoc-remap-path-prefix
GITHUB_JOB=pr
GITHUB_PATH=/home/runner/work/_temp/_runner_file_commands/add_path_5246b8f5-ce46-4c62-ab23-56b2c17aca23
GITHUB_REF=refs/pull/107099/merge
GITHUB_REF_NAME=107099/merge
GITHUB_REF_PROTECTED=false
---
GITHUB_SERVER_URL=https://github.com
GITHUB_SHA=7d726e190b98acf35281d5dd5d01fd96f308d4bc
GITHUB_STATE=/home/runner/work/_temp/_runner_file_commands/save_state_5246b8f5-ce46-4c62-ab23-56b2c17aca23
GITHUB_STEP_SUMMARY=/home/runner/work/_temp/_runner_file_commands/step_summary_5246b8f5-ce46-4c62-ab23-56b2c17aca23
GITHUB_TRIGGERING_ACTOR=edward-shen
GITHUB_WORKFLOW_REF=rust-lang/rust/.github/workflows/ci.yml@refs/pull/107099/merge
GITHUB_WORKFLOW_SHA=7d726e190b98acf35281d5dd5d01fd96f308d4bc
GITHUB_WORKSPACE=/home/runner/work/rust/rust
GOROOT_1_18_X64=/opt/hostedtoolcache/go/1.18.10/x64
---
   Compiling basic-toml v0.1.2
   Compiling askama_derive v0.12.1
    Checking askama v0.12.0
    Checking rustdoc v0.0.0 (/checkout/src/librustdoc)
error[E0425]: cannot find function `early_error` in crate `rustc_session`
    |
349 |                 rustc_session::early_error(error_format, err);
    |                                ^^^^^^^^^^^ not found in `rustc_session`


error[E0308]: `match` arms have incompatible types
   --> src/librustdoc/config.rs:349:17
    |
346 |           let remap_path_prefix = match parse_remap_path_prefix(&matches) {
    |  _________________________________-
347 | |             Ok(prefix_mappings) => prefix_mappings,
    | |                                    --------------- this is found to be of type `Vec<(PathBuf, PathBuf)>`
348 | |             Err(err) => {
349 | |                 rustc_session::early_error(error_format, err);
    | |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `Vec<(PathBuf, PathBuf)>`, found `()`
351 | |         };
    | |_________- `match` arms have incompatible types
    |
    |
    = note: expected struct `Vec<(PathBuf, PathBuf)>`

Some errors have detailed explanations: E0308, E0425.
For more information about an error, try `rustc --explain E0308`.
error: could not compile `rustdoc` (lib) due to 2 previous errors

[bors]

:umbrella: The latest upstream changes (presumably #117875) made this pull request unmergeable. Please resolve the merge conflicts.

[Dylan-DPC]

@edward-shen if you can resolve these conflicts we can push this forward for a review. Thanks

[edward-shen]

@Dylan-DPC, thanks for taking another look at this. I've rebased my changes and ran ./x t, so hopefully things should still be correct.

[edward-shen]

@rustbot review

[bors]

:umbrella: The latest upstream changes (presumably #122555) made this pull request unmergeable. Please resolve the merge conflicts.

[GuillaumeGomez]

I added it in the next rustdoc meetup agenda to be discussed.

[GuillaumeGomez]

There is no blocker on our side. Just need members of the team to review this and to approve if they agree.

So here we go. cc @rust-lang/rustdoc. Please take a look at this comment.

[rfcbot]

:bell: This is now entering its final comment period, as per the review above. :bell:

[rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

This will be merged soon.

[GuillaumeGomez]

Thanks @edward-shen for your work and your patience and thanks everyone for your work here!

@bors r+ rollup