Handle tests that should be run within the same process

[sunshowers]

Currently, nextest runs every test in its own process. Should running tests within the same process be supported?

• [ ] How should test binaries be marked this way? package.metadata is one solution.
• [ ] How does concurrency control work? Rust's libtest doesn't support the jobserver protocol, so it's hard to communicate with it. One option is to run them serially at the end, once all the regular tests are run.
• [ ] How can test timing, stdout/stderr for individual tests, and other results be obtained? libtest's --format json is nightly-only. Maybe nextest can just give up on reporting individual statuses for these tests.

[crepererum]

I think it would be nice if this is not an "all or nothing" flag, but rather that the user can describe multiple groups that should be run within a process, e.g. by specifying:

• workspace packages: each package in the list is a group
• multiple test-name wildcards: each wildcard is a group (not sure though how to guarantee that groups don't overlap)

All tests NOT found within a group are run within their own group.

To illustrate why this is helpful: over in influxdb_iox all tests within the workspace can be run in parallel. However for end2end tests within the influxdb_iox crate start a real server process but try to reuse it for multiple tests, so we pay for the startup costs every time. The tests using the single server process could also be run in parallel though, but currently they need to be executed from the same test process (they "find" and initialize the server process using lazy_static).

[sunshowers]

Yeah, that is the use case that I was thinking of. I agree that an all-or-nothing flag would be bad; the default will always be to run each test in its own process. The same-process opt-in would be per-test-binary using package.metadata.

The main challenges here are still concurrency control and gathering data about individual tests. Currently the only way to gather data about individual tests is to use the JSON format, which is nightly-only. We could certainly add a flag to the configuration to turn on nightly-only features, or just rely on people passing in -Z unstable-features.

[sunshowers]

This is going to require somewhat advanced concurrency-related and stability-related work so is not a good first issue, but I'll mark this as "help wanted" for now. If you'd like to implement it, I'd be happy to accept a PR! We could probably go over further design intricacies once there's a PR for this.

[sunshowers]

Been thinking about this -- I've figured out a way to do this on stable Rust using the --logfile option. We can rely on two things:

1. cargo test executes tests in lexicographic order
2. data is written to logfiles as soon as the test passes

So what we can do is:

• execute the test binary with --test-threads set to min(number of tests, cpu thread count)
• instruct the test binary to write to a named pipe (fifo on unix, named pipe on Windows)
• assume that tests are being executed in lexicographic order
• as results come in over the pipe, record test success/failure and time taken
• for --success-output/--failure-output, produce combined output from the binary at the end of execution

The time taken would likely be approximate, but that's ok -- we can document that.

[sunshowers]

Update: nextest now sets a NEXTEST_RUN_ID environment variable which should help porting over some use cases to nextest.

I don't plan to work on this for free. However:

• If someone is interested and willing to implement it, I would be happy to mentor them through it. I would estimate around 100-120 hours of work for someone new to the codebase to learn about it, implement this, write proper tests for it, work through edge cases, write documentation and so on.
• If someone is interested and willing to pay me, ~~I'm open to being contracted to work on this. Please reach out at [email protected] to discuss this.~~ I'm no longer available to do this, and don't expect to be for the foreseeable future.

To be clear, this is a fundamental rework of how nextest operates. It's going to require considerations throughout the entire stack, and careful integration with every other nextest feature. This is a big, challenging project, and if you hire me to do it I will charge accordingly. I would strongly recommend considering alternative approaches first. (I'm happy to add workarounds like NEXTEST_RUN_ID.)

Edit 2023-01: I'm no longer available for contract work.

[sunshowers]

The ability to run tests with mutual exclusion (similar to the serial_test crate) is now possible with test groups.