Add cli test framework

[pnkfelix]

Infrastructure for testing the command line tool.

Note: generates tests into fresh temp dir, which is deleted after testing is done (regardless of success or failure). You can change the which_temp::WhichTempDir type definition to revise this behavior (it implements a trait and there are two implementations; one which generates into a fresh temp dir, and one that generates into /tmp)

This infrastructure includes two tests: tests/cli.rs and tests/ice.rs. Each uses very different strategies for testing cargo-bisect-rustc.

1. tests/cli.rs uses a so-called meta-build strategy: the test inspects the rustc version, then generates build files that will inject (or remove, e.g. when testing --regress=success) #[rustc_error] from the source code based on the rustc version. This way, we get the effect of an error that will come or go based solely on the rustc version, without any dependence on the actual behavior of rustc itself (beyond its version string format remaining parsable).

   • This strategy should remain usable for the foreseeable future, without any need for intervention from cargo-bisect-rustc developers.

2. tests/ice.rs uses a totally different strategy: It embeds an ICE that we know originated at a certain version of the compiler. The ICE is embedded in the file src/ice/included_main.rs. The injection point associated with the ICE is encoded in the constant INJECTION_COMMIT.

   • Over time, since we only keep a certain number of builds associated with PR merge commits available to download, the embedded ICE, the INJECTION_COMMIT definition, and the search bounds defined in INJECTION_LOWER_BOUND and INJECTION_UPPER_BOUND will all have to be updated as soon as the commit for INJECTION_COMMIT is no longer available for download.

   • Thus, this testing strategy requires regular maintenance from the cargo-bisect-rustc developers. (However, it is more flexible than the meta-build strategy, in that you can embed arbitrary failures from the recent past using this approach. The meta-build approach can only embed things that can be expressed via features like #[rustc_error], which cannot currently express ICE's.

[pnkfelix]

on my laptop, it takes about two minutes to run the test suite the first time, with a clean ~/.rustup/ directory.

The test suite uses --preserve, so that future runs will reuse the (per-user shared, for better or for worse) .rustup bisector downloads.

So, on my laptop, subsequent runs take about 30 seconds to run instead of 2 minutes.

[pnkfelix]

(CI test failure. Hmm. I wonder if something is wrong with how I am generated temporary directories?)

[pnkfelix]

Oh; I wonder if the problem is that cargo-bisect-rustc is not on the PATH???

[pnkfelix]

Apparently I might need to use some relatively new functionality...

[chrissimpkins]

I am seeing the following test fail on macOS 10.15.3 as of f239318aa1d5f005d26ba579610dc534bab952b5 (added a few escaped triple ticks so that it renders in a code block):

running 10 tests
test least_satisfying::tests::least_satisfying_5 ... ok
test least_satisfying::tests::least_satisfying_7 ... ok
test least_satisfying::tests::least_satisfying_3 ... ok
test least_satisfying::tests::least_satisfying_4 ... ok
test least_satisfying::tests::least_satisfying_2 ... ok
test least_satisfying::tests::least_satisfying_8 ... ok
test least_satisfying::tests::least_satisfying_1 ... ok
test least_satisfying::tests::least_satisfying_6 ... ok
test test_nightly_finder_iterator ... ok
test least_satisfying::tests::qc_prop ... ok

test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/cli-93cf2b2870ddfd32

running 1 test
test cli_test ... test cli_test has been running for over 60 seconds
test cli_test ... FAILED

failures:

---- cli_test stdout ----
running `cargo-bisect-rustc --preserve --access=github --start 2020-02-06 --end 2020-02-08` in "/tmp/cbr_test_cli_basic"
Command output stdout for cbr_test_cli_basic:
\```
std for x86_64-apple-darwin: 16.44 MB / 16.44 MB [======] 100.00 % 2.06 MB/s 0s
\```
Command output stderr for cbr_test_cli_basic:
\```
installing nightly-2020-02-06
testing...
RESULT: nightly-2020-02-06, ===> No

installing nightly-2020-02-08
testing...
RESULT: nightly-2020-02-08, ===> Yes

installing nightly-2020-02-07
testing...
RESULT: nightly-2020-02-07, ===> No

searched toolchains nightly-2020-02-06 through nightly-2020-02-08
installing nightly-2020-02-08
testing...


********************************************************************************
Regression in nightly-2020-02-08
********************************************************************************

fetching https://static.rust-lang.org/dist/2020-02-07/channel-rust-nightly-git-commit-hash.txt
converted 2020-02-07 to 442ae7f04026c215a03b155eaaf9cde8bb5cf02a
fetching https://static.rust-lang.org/dist/2020-02-08/channel-rust-nightly-git-commit-hash.txt
converted 2020-02-08 to a29424a2265411dda7d7446516ac5fd7499e2b55
looking for regression commit between 2020-02-07 and 2020-02-08
adding local env GITHUB_TOKEN value to headers in github query
adding local env GITHUB_TOKEN value to headers in github query
fetching (via remote github) commits from max(442ae7f04026c215a03b155eaaf9cde8bb5cf02a, 2020-02-05) to a29424a2265411dda7d7446516ac5fd7499e2b55
adding local env GITHUB_TOKEN value to headers in github query
ending github query because we found starting sha: 442ae7f04026c215a03b155eaaf9cde8bb5cf02a
get_commits_between returning commits, len: 5
  commit[0] 2020-02-06UTC: Auto merge of #68893 - Dylan-DPC:rollup-3f2421a, r=Dylan-DPC
  commit[1] 2020-02-07UTC: Auto merge of #68907 - Dylan-DPC:rollup-osm5e8o, r=Dylan-DPC
  commit[2] 2020-02-07UTC: Auto merge of #68499 - ssomers:btree_search_tidying, r=Mark-Simulacrum
  commit[3] 2020-02-07UTC: Auto merge of #68930 - RalfJung:miri, r=RalfJung
  commit[4] 2020-02-07UTC: Auto merge of #68933 - Dylan-DPC:rollup-akz13kj, r=Dylan-DPC
validated commits found, specifying toolchains

installing 442ae7f04026c215a03b155eaaf9cde8bb5cf02a
testing...
RESULT: 442ae7f04026c215a03b155eaaf9cde8bb5cf02a, ===> No

installing a29424a2265411dda7d7446516ac5fd7499e2b55
testing...
RESULT: a29424a2265411dda7d7446516ac5fd7499e2b55, ===> Yes

installing b5e21dbb5cabdaaadc47a4d8e3f59979dcad2871
testing...
RESULT: b5e21dbb5cabdaaadc47a4d8e3f59979dcad2871, ===> Yes

installing f8fd4624474a68bd26694eff3536b9f3a127b2d3
testing...
RESULT: f8fd4624474a68bd26694eff3536b9f3a127b2d3, ===> Yes

searched toolchains 442ae7f04026c215a03b155eaaf9cde8bb5cf02a through a29424a2265411dda7d7446516ac5fd7499e2b55


********************************************************************************
Regression in f8fd4624474a68bd26694eff3536b9f3a127b2d3
********************************************************************************

==================================================================================
= Please file this regression report on the rust-lang/rust GitHub repository     =
=        New issue: https://github.com/rust-lang/rust/issues/new                 =
=     Known issues: https://github.com/rust-lang/rust/issues                     =
= Copy and paste the text below into the issue report thread.  Thanks!           =
==================================================================================

searched nightlies: from nightly-2020-02-06 to nightly-2020-02-08
regressed nightly: nightly-2020-02-08
searched commits: from https://github.com/rust-lang/rust/commit/442ae7f04026c215a03b155eaaf9cde8bb5cf02a to https://github.com/rust-lang/rust/commit/a29424a2265411dda7d7446516ac5fd7499e2b55
regressed commit: https://github.com/rust-lang/rust/commit/f8fd4624474a68bd26694eff3536b9f3a127b2d3

<details>
<summary>bisected with <a href='https://github.com/rust-lang/cargo-bisect-rustc'>cargo-bisect-rustc</a> v0.5.0</summary>


Host triple: x86_64-apple-darwin
Reproduce with:
\```bash
cargo bisect-rustc --access=github --start 2020-02-06 --end 2020-02-08
\```
</details>

thread 'cli_test' panicked at 'assertion failed: stderr.contains(&needle)', tests/cli.rs:89:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    cli_test

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '--test cli'

[chrissimpkins]

I haven't had a chance to fully dig in to your changes but this is fantastic to see btw! :)

[chrissimpkins]

I added cross-platform GH Actions CI workflows in #90

[chrissimpkins]

#90 merged and available for testing with this PR

[pnkfelix]

Sigh: rustfmt does not understand my idiosyncratic module layout...

[pnkfelix]

Sigh: rustfmt does not understand my idiosyncratic module layout...

Seems likely related to this: https://github.com/rust-lang/rustfmt/issues/3794

I will work-around it locally.

[pnkfelix]

Hmm what does this output imply...

2020-05-15T17:42:01.2716820Z fetching https://static.rust-lang.org/dist/2020-02-20/channel-rust-nightly-git-commit-hash.txt
2020-05-15T17:42:01.2717650Z converted 2020-02-20 to 7760cd0fbbbf2c59a625e075a5bdfa88b8e30f8a
2020-05-15T17:42:01.2718440Z fetching https://static.rust-lang.org/dist/2020-02-21/channel-rust-nightly-git-commit-hash.txt
2020-05-15T17:42:01.2719200Z converted 2020-02-21 to 2c462a2f776b899d46743b1b44eda976e846e61d
2020-05-15T17:42:01.2719940Z looking for regression commit between 2020-02-20 and 2020-02-21
2020-05-15T17:42:01.2720150Z ERROR: error decoding response body: missing field `commit` at line 1 column 246
2020-05-15T17:42:01.2720260Z 
2020-05-15T17:42:01.2720430Z ```
2020-05-15T17:42:01.2721160Z thread 'ice_test' panicked at 'assertion failed: stderr.contains(&needle)', tests/ice.rs:100:5
2020-05-15T17:42:01.2721390Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2020-05-15T17:42:01.2721510Z 
2020-05-15T17:42:01.2721600Z 
2020-05-15T17:42:01.2721780Z failures:
2020-05-15T17:42:01.2721950Z     ice_test
2020-05-15T17:42:01.2722040Z 
2020-05-15T17:42:01.2722220Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
2020-05-15T17:42:01.2722340Z 
2020-05-15T17:42:01.2724610Z error: test failed, to rerun pass '--test ice'

[chrissimpkins]

error decoding response body: missing field `commit` at line 1 column 246

Looks reqwest-y :)

It should be possible to reproduce this with cargo test @ https://github.com/rust-lang/cargo-bisect-rustc/pull/88/commits/da1ee9d3888ec20ba23c35507aec47eb7674939d?

[pnkfelix]

@chrissimpkins wrote:

It should be possible to reproduce this with cargo test @ da1ee9d?

I tried on my mac but it does not reproduce there.

I'll be getting a dedicated Linux (well maybe Windows too) desktop system in the near future; I'll give it a shot there and see if it reproduces in that context.