Run multiple Minitest examples via a single ruby command from the VS Code test explorer

[thomasmarshall]

Motivation

Running multiple tests in the VS Code test explorer or via the "Run test" code lens link is currently quite slow. That's because it executes a separate command per example, even if the test runner is capable of running multiple examples at once. That means any start up cost is incurred for each example. This PR attempts to improve that situation for Minitest, by combining the examples into a single command.

Implementation

We can't use the default Minitest test reporter (or whichever custom reporter the host project is using) for this, because it doesn't have enough structured output in order to know which tests have started or finished, and it's difficult to correlate any failure/error messages to the test in the explorer. This PR introduces a custom reporter that outputs structured JSON, and overrides whatever reporter is normally in place for the host project.

This relies on Minitest conventions to find the plugin, but it also injects some code to disable other Minitest plugins except our custom reporter. This is probably not ideal, but I wasn't sure how to force Minitest to use our custom reporter, other than disabling any other ones. Perhaps there is a way to guarantee our plugin is loaded last, so it overrides any other plugins? Alternatively, maybe our custom reporter could augment the existing reporter and output JSON to a temporary log file instead stdout.

We also need some way to generate a combined test command from a series of separate examples. We're doing this by setting up some "combining" options that include the full name for each example and describe how those names should be combined into a command (in this case via a regex in passed to the --name argument). Ideally this should be extensible enough that it could be used by other test frameworks and runners—in particular I'd like to use this for testing Rails apps which often have a multi-second start up cost to running a test.

Automated Tests

I have updated the expectation tests, I'm not sure how to add automated tests for the VS Code extension part.

Manual Tests

Open a Minitest test file and click the "Run" code lens link. Try also selecting an arbitrary selection of tests in the explorer and running just those tests.

Demo

Here's a before/after demo with a fake 1s start up cost.

https://github.com/Shopify/ruby-lsp/assets/770763/0643cf92-40a5-428e-8254-4d062f26c1de

[andyw8]

It might be possible get the reporter selection upstreamed:

If TextMate, TeamCity, RubyMine or VIM presence is detected, the reporter will be automatically chosen, regardless of any reporters passed to the use! method.

https://github.com/minitest-reporters/minitest-reporters

[andyw8]

^ Ignore the above, I was forgetting minitest-reporters isn't shipped with minitest.

[andyw8]

(it's ugly, but the way RubyMine handles this is with the RM_INFO env var: https://www.jetbrains.com/help/ruby/testing-frameworks.html#supported_frameworks )

[thomasmarshall]

Is it possible to hand off the running of the tests to the LSP instead of having the VS Code extension do it directly? That way the extension wouldn't need to care about how to combine tests into a single command or how to parse/understand the test output—that would all be done in the LSP. That might make it cleaner/easier for the Ruby LSP plugins for Rails and RSpec to decide how tests are combined and what their own test reporter output looks like.

[github-actions[bot]]

This pull request is being marked as stale because there was no activity in the last 2 months

[andyw8]

@thomasmarshall thanks for your work on this, we do still plan to address it but need to prioritize some other things at the moment.