[Feature]: An option to reduce the "Your test suite must contain at least one test" to a warning, or disable it completely

[roblframpton]

🚀 Feature Proposal

A configuration option should be implemented to configure the error thrown by Jest when it encounters empty test file, which states "Your test suite must contain at least one test". It should be possible to disable it, or reduce it to a warning. Ideally, the latter should be the default.

Motivation

Tha current behavior means that tests appear to fail, even though all test cases have actually passed. I argue that empty test files occurs commonly as an acceptable part of the development cycle. I believe it is misleading to present an error for a sitution where the tests are all passing, and that this is antithetical to TDD's Red-Green-Refactor cycle. I also believe that it is beyond the remit of Jest to present errors which are related to style rather than functionality.

Don't close this issue yet!

I'm aware that this has been raised before before, and each time it has been quickly closed as "won't do". I'd like to better articulate an argument for why Jest should make this behavior configurable (and, in my opinion, should be a warning by default). I can feel the Jest team already hovering their cursors over the close button, and I have already come to terms with the likely fate of this request - but I implore you to please hear me out! 😅

Several replies to the issues above were along the lines of, "why would you want an empty test file? That serves no purpose, therefore doing so is wrong". I do agree with this - but NOT at the point in the development cycle where Jest is usually executed, which I will expand on later. I do not think it is Jest's responsibility present such an error. I believe that the downside of having this error, which is a lack of clarity, particularly when performing TDD, outweighs the benefit of having a clean codebase.

Some examples

I would describe myself as a competent developer, and yet I encounter this message frequently during day-to-day development. I have never encountered a situation where it improved my workflow. Instead, it gave me unnecessary errors to sift through when my tests were, in fact, passing.

Consider this example: I create a new file called service.test.js, start writing my first test, and then suddenly realize there's some other functionality I need to make changes to first. Not an uncommon situation in my experience - after all, as Agile developers we promote the emergence of designs as we go along. I'm halfway through writing that test so I quickly comment it out, go make some other changes, run my tests and... Error 💥. Jest tells me my test run failed. This message hasn't helped me in any way, it's just filled my console with an error which didn't actually reflect the state of my tests, momentarily raising my stress levels and taking more valuable minutes from my already heavily compromised lifespan.

Or, perhaps I'm thinking about functionality I'm planning to implement next, so I create a couple of files, service.get.test.js and service.post.test.js. It just helps me organize my thoughts, and I don't think Jest should really have an opinion on how I go about planning my work. I start on one of the files, run my tests to check my progress... Error 💥. This hasn't helped me - I know there's an empty file, I'm going to add tests there in about 5 minutes, I just haven't got there yet.

Now maybe I've changed the interface of one of my methods, and a whole load of tests are broken as a result. I'll fix them in a minute, but I need to make another change first. Since these tests are now full of compile errors I can't use test.skip, so I'll just comment out the entire file for a moment. I'll put them back and fix them once I make this other change. I make that change, run my tests and... Error 💥. "Test Suites: 1 failed", says Jest. I beg to bloody differ. All my tests passed. I made a code style infraction, that isn't a test failure!

Yet another example, similar to one of the previous issues I linked, might be that I have a template for starting new projects in my team. It contains my team's folder structure, the package.json, maybe a handful of JS files which we always have by convention, and a couple of empty test files to show team members how we structure our tests. I say, "here's our template to get you started, oh, and here's how you run the tests"... Error 💥. And that'll keep happening until each file has at least one test. It hasn't helped us - we know about those placeholder files, they're there to guide the team, so we want them there, we just haven't got to them yet. A warning would suffice.

Conclusion / TL;DR

My point is that Jest presents this error when my code is very much work-in-progress. Now, I know that some will argue that "you shouldn't have empty test files! It's good that Jest catches them". But I think this is ignoring the role that tests play in development, and the point at which they are run. In an Agile team using TDD, running tests is part of the low-level development cycle. It's something we do repeatedly after every increment of code is written. We should expect half-written, non-conventional code at this point, and Jest isn't helping by expecting a clean codebase at this point in the cycle. Clean code comes a little later - for example, I wouldn't commit empty test files to source control - but Jest gets executed far earlier and more frequently in the cycle than a commit.

At best, this is behavior annoying. But at worst, it's downright misleading. If I don't spot the actual error message (easily done at 4pm on a Friday), I may think that some test which I'm trying to fix is still failing. There's a reason that Jest highlights test results with big red or green highlights, which is clarity. I'm supposed to glance at the console and immediately see if I'm in the red or the green stage of my red-green-refactor cycle. But instead, I'm always in red, despite my tests all passing. This is not in the spirit of TDD.

Taking that point further - I would strongly argue that dissuading the user from having empty test files is out of the scope for a test runner. Jest is not a linter. I have other (highly configurable) tools which I use to warn about code which is untidy, bad practice, or breaks conventions, and I run these at appropriate times (e.g. before a commit). Why has Jest decided to be so opinionated about something which is a matter of good practice and convention rather than functionality? Or, looking from another angle - why does it stop there? If Jest cares about these things, why doesn't it tell me off for running a test where the name is an empty string (which it does allow, and seems worse practice to me than an empty file!). It doesn't tell me off for having an empty function body. It doesn't care about file name formats. So why does it get involved with this one particular issue of style?

To me, it seems clear that this should never be presented as a test failure. A warning seems like a good compromise. It helps guide the user towards good style, but crucially, it does not present a red test failure when there are empty files, which I believe is plain wrong.

Thanks for reading! If this proposal actually gets accepted, I will immediately buy myself a lottery ticket 😁

Actual TL;DR:

Empty files shouldn't be encouraged but they definitely aren't test failures, and presenting them as such is misleading for the TDD cycle, during which we might reasonably expect empty files as the code is work-in-progress. A warning would be sufficient.

[marr]

:+1: at the least, the --passWithNoTests should prevent this error.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity. Please open a new issue if the issue is still relevant, linking to this one.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity. Please open a new issue if the issue is still relevant, linking to this one.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.