🚀 Feature: Add an option to run all tests, including skipped ones

[JavaScriptBach]

Feature Request Checklist

• [x] I have read and agree to Mocha's Code of Conduct and Contributing Guidelines
• [x] I have searched for related issues and issues with the faq label, but none matched my issue.
• [ ] I want to provide a PR to resolve this

Overview

It would be nice to have an option that could make Mocha run all tests, even skipped ones.

At my company, we skip flaky tests in our CI build to avoid blocking developers, but we'd like to set up a "quarantined" tests build that runs all tests. This allows us to check when flaky tests are no longer flaky and can be safely introduced back into the blocking tests pool.

Suggested Solution

mocha --includeSkipped

Alternatives

Sed all my test files before running Mocha, which isn't particularly sound.

Additional Info

No response

[JoshuaKGoldberg]

Hmm, but how would Mocha / a project's testing infrastructure know which tests are skipped for flakiness vs. other reasons?

This feels to me like a project-specific preference that would be better solved in project-land. For example, you could write your own function that calls it or it.skip depending on some process.env.RUN_FLAKY_TESTS:

function myIt(title, callback) {
  if (process.env.RUN_FLAKY_TESTS) {
    it(title, callback);
  } else {
    it.skip(title, callback);
  }
}

Am I missing some nuance to the request?

[JavaScriptBach]

Mocha doesn't need to know why a test is skipped. Under this model, the user asked for all tests to be run, so interpreting the results becomes the user's problem.

Your workaround is fair; it just feels weird that Mocha supports marking a test as skipped, but doesn't support other ways of interacting with it. If I wanted a test to never run, the logical thing to do would be to delete the test entirely.

[JoshuaKGoldberg]

If I wanted a test to never run, the logical thing to do would be to delete the test entirely.

Not necessarily. Folks sometimes try to keep skipped tests around to make sure they stay valid for static analysis such as linting or type-checking. A test might be skipped for a time-sensitive reason such as an external service not being ready yet or dropping support for something at a known time.

Anyway, per #5027, we're not looking to add a lot of new features. I'm going to go ahead and close this as a pretty specific use case. Thanks for filing + discussing! 🤎

[JavaScriptBach]

OK, fair enough. Thanks for considering it, and thanks for taking up the maintenance of this project. I feel like it is in good hands after seeing the work you've done in typescript-eslint.