Do not fail on 'Jest: Coverage data for ... was not found.' or make `coverageThreshold` configurable on the cli

[Globegitter]

🚀 Feature Proposal

We are using one jest.config.js per application and using Bazel as the build/testing tool. The setup there works that each test file will be invoked in isolation/sandboxed via jest. So as soon as I have coverageThreshold: { './some/path/': {...} } some of these test will fail with Jest: Coverage data for ... was not found. Now I could point them all to their individual config, but then I would have to create many many more file which all would be identical apart from the coverageThreshold.

So it would be great to either get a passWithNoCoverage flag or similar, or make coverageThreshold configurable on the cli, so I could just pass that in for each individual jest invocations

Motivation

So I do not have to have duplicate jest.config files

Example

either:

jest --passWithNoCoverage --config=path/to/config.js --runTestsByPath=path/to/test.js

or

jest --coverageThreshold='{\"global\":{\"statements\":90}' --config=path/to/config.js --runTestsByPath=path/to/test.js

Pitch

One of the proposals might even be a bug report, see: https://github.com/facebook/jest/issues/7509 and the addition of the new flag would imo be a good fit to complement the already existing --passWithNoTests flag.

[Globegitter]

It seems it is possible to set --coverageThreshold after all, just in a very specific way. While --passWithNoCoverage would still be a nice addition at least there is a reasonable workaround possible.

[tiffafoo]

Have the same query! Trying to only run changed files and it'll fail with Jest: Coverage data for ... was not found. when there's a coverage threshold in the config. I'll look into --coverageThreshold although it'll be a very ugly script :P.

[Edit]: My solution is to have

jest --coverage --changedSince=master --passWithNoTests --coverageThreshold='$(cat coverage.json)'

with coverage.json being an object with my thresholds

[thymikee]

Hey! I don't think adding a new config option is a good solution to that. You can run your tests without coverage using --no-coverage flag.

Supporting coverageThreshold through CLI is definitely an option.

I think it would make sense to pass the test with coverage if there's no coverage data found. We still need to keep the message though. cc @jeysal @SimenB

[jebentier]

The --no-coverage flag isn't an option for my use case because what I'm looking to do is be able to run a subset of my test suite, with coverage and the coverageThresholds set in the main configuration of the test framework, so that I can see if for my subset of tests in the suite, I am going to be decreasing or breaking coverage.

I think this use case is completely valid in that locally you don't want to have to run the full test suite when you're developing in a specific area, and you also want your tests to run as close to CI as possible so that you aren't surprised if CI breaks when you push up to remote.

[smably]

Our use case is: 1) coverage threshold set, and 2) prepush hook that runs Jest with --changedSince=master to reduce test execution time, since the prepush hook is about getting quick feedback before a full CI run, and 3) not an option for us to run with --no-coverage because meeting our coverage threshold is important if tests are run.

However, I would expect --passWithNoTests to also imply that Jest should pass with no coverage, since there's no way that a coverage threshold could ever be met if no tests ran.

[stefaniehansen]

We split our test cases so we can parallelize and have faster test runs in CI using --testPathPattern. We've been forced to remove coverage during CI because we cannot specify coverage thresholds for the tests that we specify alone. It would be great to be able to specify coverage thresholds for a subset of tests so we can get coverage data from our CI runs again.

[github-actions[bot]]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 14 days.

[smably]

Absolutely still an issue.

[schmkr]

Chiming in here, with different example.

When using Nx, it comes with a Jest preset in the root of the project where you can define thresholds for certain glob patterns (e.g. all files matching '**/*.effects.ts'). That preset is then used by all the apps and libs Nx generated. But some of the apps and libs don't have files matching that pattern. We work around that now, by setting coverageThreshold: {}, in the jest.config.ts file of the specific app, but that would result in also losing all other patterns from the preset.

Having a configuration option to not error on missing coverage data for some of the patterns defined, would be very helpful for this.

[JakeTompkins]

I'd like this too. I have a threshold configuration that contains a few different globs, but those globs don't always have a match when I run tests on only staged files. I'd like it to be smart enough to say that if there aren't any staged files matching the glob, then the threshold should be ignored.

[jebentier]

@thymikee you closed the PR looking for discussion, but there’s been no response to the valid use cases here requesting to have a coverage report without failing tests. Any chance a discussion around this can be had? It seems that there are plenty of cases where gaining visibility with the coverage report is desired but failing the test run strictly because of missing coverage is not what’s desired. This is why in most testing frameworks, coverage has been decoupled from the test framework itself, allowing the developers more control over what will fail a test run, and when a failure happens, if it was due to a failing test or missing coverage.

[scamden]

100% this PR is not that same thing as running with --no-coverage. arguably this should be the default behavior?? if i specify the tests down to a certain set i wouldn't expect untested files to still fail for coverage reasons.

[github-actions[bot]]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 30 days.

[scamden]

Not stale, please fix this

[zabdalimov]

I was desperate to find a workaround to ignore Jest: Coverage data for ... was not found. error since nothing like --passWithNoCoverage exists.

I tried adding, extending and overriding default jest reporters using reporters and coverageReporters fields in jest configuration, but nothing worked.

In the end I decided I'll go as dirty as needed and wrote a bash script which calls jest, writes all output to a file, and then using a couple of greps evaluates whether this error was encountered and returns exit code 0 if there are no real test failures:

#!/bin/bash

TEST_OUTPUT_LOG_NAME="test-output.log"

# `$@` passes any additional params given to the script
# `2>&1 | tee` saves all output to a file as well as printing it in command line
yarn jest "$@" 2>&1 | tee $TEST_OUTPUT_LOG_NAME

EXIT_CODE_ERROR=$(cat $TEST_OUTPUT_LOG_NAME | grep 'error Command failed with exit code \d.')
TEST_FAILED_ERRORS=$(cat $TEST_OUTPUT_LOG_NAME | grep 'FAIL')
COVERAGE_DATA_ERROR=$(cat $TEST_OUTPUT_LOG_NAME | grep 'Jest: Coverage data for .* was not found')

rm $TEST_OUTPUT_LOG_NAME

if [ "$EXIT_CODE_ERROR" ]; then
    if [ "$TEST_FAILED_ERRORS" ]; then
        exit 1
    elif [ "$COVERAGE_DATA_ERROR" ]; then
        echo "Encountered coverage data not found error, will ignore it and exit with code 0 instead"
        exit 0
    else
        exit 1
    fi
fi

exit 0

It is very much unreliable and questionable workaround, but I couldn't find any other way to do this.

[EsunR]

Please pay attention to this issue that has been public for a long time 🙏 When I want to perform incremental checks while submitting code, if the files for incremental testing do not hit the directories with set coverage thresholds, an error is thrown. This frustrates me, and I have to write an extra script to dynamically change the coverageThreshold based on the input test files.

[pritam-patil]

For me, I had modular coverages recorded in my jest config which was causing the issue. I only kept global coverageThreshold. Jest didn't complain after running the coverage again