Rewrite compiletest out of tree

[oli-obk]

Proposal

Rewrite compiletest from scratch as a stable crate on crates.io. Miri is already using the MVP rewrite called ui_test (it currently lives in the miri repo, but will soon move out via git subtree). I got preliminary approval to get clippy to use it (probably via crates.io), and I am optimistic that it will get merged if I open a PR.

There is a lot more work to be done to get it usable inside rustc (currently only ui tests are supported), but I am planning on doing this in a modular way so that rustc can have the parts that no one else wants live entirely in-tree without any syncs with another repo.

As an MVP I would like to replace the ui test suite logic with the new library (once feature parity has been reached). This will likely happen by changing the in-tree compiletest and the out of tree ui_test until they are in sync behaviour wise and then swapping them out in a PR that has no functional changes.

Advantages

• works on stable, no dependency on libtest
• already implemented https://github.com/rust-lang/compiler-team/issues/512
• reliable support for arbitrary dependencies via a Cargo.toml shared by all tests
• uses an error handling framework for handling any failures related to test parsing and execution making debugging easier

Disadvantages

• a lot of work
• harder to do arbitrary changes (though mitigated with git subtree, still need to sync)

Mentors or Reviewers

any volunteers? 😆

Process

The main points of the Major Change Process are as follows:

• [x] File an issue describing the proposal.
• [ ] A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
  • Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
  • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
• [ ] Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

Comments

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

[rustbot]

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

cc @rust-lang/compiler @rust-lang/compiler-contributors

[pnkfelix]

@rustbot second

[apiraino]

@rustbot label -final-comment-period +major-change-accepted