Support for running tests

[ibabushkin]

Given that the tool already supports running binaries and examples (and people seem to want benchmark support too), would there be interest in having the same functionality for tests? The only issue I see with this kind of functionality would be the fact that one would profile the test harness and/or multiple tests in the same binary in some cases.

Additionally, one can obviously just find the binary cargo has built and running that using flamegraph, but that's somewhat annoying.

I have some hacky code for this around, but before I clean it up and open a PR I wanted to sample what opinions the authors and maintainers have.

[jonhoo]

I think that's a great idea! There has already been a few requests from it (e.g., on Reddit), and I proposed the syntax I would love to see here. Any supported way of doing it would be awesome though :D

[ibabushkin]

Alright, I'll clean up my unwrap-fest and open a PR then. Properly isolating the profiling of the actual test code and the surrounding stuff would probably require a lot of work outside the scope of this tool, so I guess it's reasonable to leave it to the user to set up something more reasonable if accurate profiling is necessary.

[jonhoo]

I would probably do something like cargo test --no-run, and then profile cargo test. That should eliminate most of the additional stuff!

[ibabushkin]

I think there is a misunderstanding: it's rather easy to profile the running of the test only -- we just fiddle the test binary out of the target directory. We'd also have to ensure that the test has been built correctly, but that's easy enough. The issue remaining is that the test binary could run multiple tests (if they come from the same file, they end up in the same binary roughly speaking) and that there is some setup code in the binary as well. But working around that would be too much for now I think.

[spacejam]

This is partially implemented now, but can be cleaned up a bit more.

Say I want to run a bunch of tests that begin with pagecache_bug_ that live in a file called test_pagecache.rs.

With normal cargo, I would run:

cargo test pagecache_bug_

and right now I need to run:

cargo flamegraph --test test_pagecache -- pagecache_bug_

I think we can add better detection to eliminate needing to specify the binary name, and generally match the syntax @jonhoo has suggested to bring it to the ideal situation of:

cargo flamegraph test pagecache_bug_

[robinmoussu]

It seems that it should be possible to run tests, by it doesn't work for me. I have 3 tests (gpx_write, gpx_read, and read) in a file named gpx_writer_write_test_tour_generation, but when I do cargo flamegraph test gpx_writer_write_test_tour_generation, I got the following error:

several possible targets found: ["gpx_write", "gpx_read", "read"], please pass --bin <binary> or --example <example> to cargo flamegraph to choose one of them

I tried cargo flamegraph test gpx_writer_write_test_tour_generation -- gpx_write and many similar variation (with and without --test, that all gave the same error.

[gussmith23]

Can I run unittests? I have unittests scattered through my code, but I'm not sure how to run them with flamegraph.

[dtel]

Given that the tool already supports running binaries and examples (and people seem to want benchmark support too), would there be interest in having the same functionality for tests? The only issue I see with this kind of functionality would be the fact that one would profile the test harness and/or multiple tests in the same binary in some cases.

Additionally, one can obviously just find the binary cargo has built and running that using flamegraph, but that's somewhat annoying.

I have some hacky code for this around, but before I clean it up and open a PR I wanted to sample what opinions the authors and maintainers have.

This is partially implemented now, but can be cleaned up a bit more.

Say I want to run a bunch of tests that begin with pagecache_bug_ that live in a file called test_pagecache.rs.

With normal cargo, I would run:

cargo test pagecache_bug_

and right now I need to run:

cargo flamegraph --test test_pagecache -- pagecache_bug_

I think we can add better detection to eliminate needing to specify the binary name, and generally match the syntax @jonhoo has suggested to bring it to the ideal situation of:

cargo flamegraph test pagecache_bug_

Can't seem to make tests run with Flamegraph. I have tests in the tests folder in a file mytests.rs , which has a test named huge_test.

When i run cargo flamegraph --test mytest -- huge_test i get an error: could not find desired target mytests in the test targets for this crate. What i'm i missing?

[djc]

Can you run with -v to figure out how the commands generated by flamegraph might be wrong?