cargo doesn't include test = false tests in --tests or --all-targets

[vojtechkral]

Problem

A test marked as test = false is supposed to not be run when invoking cargo test.

This works, however, not only is the test not run, it's not built or checked (when using cargo check) either, not even with --tests or --all-targets passed.

As a workaround, I'm currently using --tests \* instead of --tests to actually include all tests including test = false ones.

Steps

1. Mark a test with test = false in Cargo.toml
2. Run cargo check --tests
3. The test is not checked.
4. Run cargo check --test \*
5. The test is checked.

Possible Solution(s)

No response

Notes

No response

Version

cargo 1.77.2 (e52e36006 2024-03-26)
release: 1.77.2
commit-hash: e52e360061cacbbeac79f7f1215a7a90b6f08442
commit-date: 2024-03-26
host: aarch64-apple-darwin
libgit2: 1.7.2 (sys:0.18.2 vendored)
libcurl: 8.4.0 (sys:0.4.70+curl-8.5.0 system ssl:(SecureTransport) LibreSSL/3.3.6)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Mac OS 14.4.1 [64-bit]

[epage]

For myself, I think the --tests behavior is reasonable. I am wondering about the --all-targets behavior. We likely just alias that to --lib --bins --tests --benches --examples, overlooking cases like this. I'm assuming it would be safe to update it to include all targets.

[vojtechkral]

Yeah, test = false tests are a bit of an oddball. They sort of are and aren't tests.

Wdyt about having --tests build these "non-tests" but not run them? Edit: Alternatively they might be included in --bins instead, they sort of are bins...

The full context is here using cargo-fixture.

[weihanglo]

I'm assuming it would be safe to update it to include all targets.

Hmm… I can't see it. Changing that will include Cargo targets not supposed to build, and is a breaking change for some existing scripts relying on the current behavior.

[epage]

Yes, there is a risk that people would have test = false programs that can't compile in all situations they run cargo check --all-targets and may not even run when they do cargo test --test bad. The question is whether --all-targets behavior is a bug (which I feel like it is even if the docs don't quite say it) and whether that is enough of a corner case to be concerned with.