Summarize output result of all test binaries run

[davemilter]

For example if run cargo test for bindgen, you got such output:

running 16 tests
**many lines**
test result: ok. 16 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
running 305 tests
**many lines**
test result: ok. 305 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
running 10 tests
**many lines**
test result: ok. 2 passed; 0 failed; 8 ignored; 0 measured; 0 filtered out

Normally you run it in window that not in fullscreen mode (common scenario that this window belong to editor/ide and you have your code on the same screen at the same time),

so after all you see only last results summary, ordinary doc tests summary result, which are not important at now, it would be nice to have to have option to print summary result at the end, like this:

Summary:
                 passed | failed |  ignored | measured
unit tests        x     |    x    |       x       |      x
func tests       x     |    x    |       x       |      x
doc tests        x     |    x    |       x       |      x

[alexcrichton]

Thanks for the report! This is something we've long wanted to improve but unfortunately haven't had a chance to design/implement yet. Most of this output comes from libtest in upstream rust-lang/rust, and this'd likely fall into the category of "custom test frameworks" as well.

[davemilter]

and this'd likely fall into the category of "custom test frameworks" as well.

And in its turn C-test fall into the category not in cargo?

[stale[bot]]

As there hasn't been any activity here in over 6 months I've marked this as stale and if no further activity happens for 7 days I will close it.

I'm a bot so this may be in error! If this issue should remain open, could someone (the author, a team member, or any interested party) please comment to that effect?

The team would be especially grateful if such a comment included details such as:

• Is this still relevant?
• If so, what is blocking it?
• Is it known what could be done to help move this forward?

Thank you for contributing!

(The cargo team is currently evaluating the use of Stale bot, and using #6035 as the tracking issue to gather feedback.)

If you're reading this comment from the distant future, fear not if this was closed automatically. If you believe it's still an issue please leave a comment and a team member can reopen this issue. Opening a new issue is also acceptable!

[stale[bot]]

As I didn't see any updates in 30 days I'm going to close this. Please see the previous comment for more information!

[najamelan]

I would like to propose that this be either reopened, or migrated to libtest. Having to scroll through lots of output in order to figure out whether any tests have failed is a really poor user experience...

[ehuss]

Sure. There is a similar issue, #2832. Cargo would need to use the structured JSON output from libtest (rust-lang/rust#46450), and somehow re-render the output. Unfortunately I don't know of any progress being made to stabilize that or how difficult it would be to use it.

[pickfire]

Is it possible to only show tests that have at least one test running when using a filter?

[FilipAndersson245]

Any update on this issue?

[ClementWalter]

new to rust here, this is a real pain when coming from (more mature ?) languages such as python of javascript. jest/mocha or pytest display a much nicer log when running a project test suite. The cargo one is barely followable and I'm never sure if the test passes without scrolling back to top.

To work on this, where should we start?

[epage]

@ClementWalter at this time, this is blocked on rust-lang/rust#46450. I am starting to experiment with a new json format and will be talking to the libs team about it to see get an idea of what the path for stablization will be.

[chevdor]

This is not near a solution but in the meantime, the following snippet should help reducing the output to a minimum:

function ctest() {
    CL=always
    cargo test --color=$CL $@ 2>/dev/null | grep --color=$CL -v 'running' | grep --color=$CL -v 'test result' | grep --color=$CL -v '^$' 
}

Then instead of:

cargo test [my options]

use:

ctest [my options]

Or even more extreme:

ctest -q [my options]

[ds-cbo]

A slightly more robust version of the helper by @chevdor which doesn't ignore tests with "running" in their name 😛

function ctest() {
  cargo test "$@" |& grep -Ev '^(running [0-9]+ tests?$|test result: ok\.|\s+Running (unittests|tests/)|$)' 
}

[solaeus]

Here's my command to filter the cargo test output down to a useful summary using ripgrep:

cargo test --no-fail-fast | rg --color never '^test|^Running [unittest|test]|^Doc-tests'

I tried to make it unambiguous so that it will work regardless of the test names. It doesn't show why tests fail, it shows a summary of all test results. I use fish but I ran this through sh and it works there as well.

If you use fish, you can add this to to config.fish to get a command that passes its arguments to cargo test:

function cargo_test_summary
    cargo test $argv | rg --color never '^test|^Running [unittest|test]|^Doc-tests'
end

Now you can run cargo_test_summary <test_name> or cargo_test_summary --workspace --no-fail-fast.

[mtalexan]

@ClementWalter at this time, this is blocked on rust-lang/rust#46450. I am starting to experiment with a new json format and will be talking to the libs team about it to see get an idea of what the path for stablization will be.

Was this blocked in some non-obvious way? The specific PR that was referenced was merged in 2018, more than 5 years before this comment saying it was a blocker. Is there some other work that needs to be completed externally for the S-blocked-externally tag?

[weihanglo]

JSON output format is not stabilized yet. See https://github.com/rust-lang/rfcs/pull/3558 for the stabilization plan, and https://github.com/rust-lang/testing-devex-team/issues/1 for the ongoing works.