Coverage Testing Does Not Work For Nested Functions (Closures)

[seanlaw]

Summary

I have series of tests where the assertion needs to be executed inside of a closure (nested) test function since it needs to utilize a separate test framework in order for setup/teardown to be handled elegantly. However, it doesn't look like closures/nested test functions are supported for coverage testing.

Expected vs actual result

Something like this seems to work fine and is detected by coverage:

@pytest.fixture(scope="module")
def dask_client():
    cluster = LocalCluster(n_workers=2, threads_per_worker=2)
    client = Client(cluster)
    yield client
    # teardown
    client.close()
    cluster.close()

test_data = [np.random.randint(1000)]

@pytest.mark.filterwarnings("ignore:\\s+Port 8787 is already in use:UserWarning")
@pytest.mark.parametrize("x", test_data)
def test_some_func(x, dask_client):
    y = some_func(dask_client, x)
    ...
    npt.assert_almost_equal(the_correct_answer, y)

Unfortunately, the pytest fixture is flaky and it isn't tearing down the dask_client properly which leads to computational resources not being released and ultimately consumes all CPUs as more tests are ran. It was recommended that we do this instead:

test_data = [np.random.randint(1000)]

@pytest.mark.filterwarnings("ignore:\\s+Port 8787 is already in use:UserWarning")
@pytest.mark.parametrize("x", test_data)
def test_some_func(x):
    @gen_cluster(client=True)
    def test(c, s, a, b):
        y = some_func(c, x)  # Coverage can't seem to see this function call
        ...
        npt.assert_almost_equal(the_correct_answer, y)

Here @gen_cluster replaces the pytest fixture above and handles setup/teardown of the dask_client (in this case, the dask_client is retrieved as c. The test actually passes but coverage does not detect it.

Normally, @gen_cluster is used to decorate the test function:

@gen_cluster(client=True)
def test_some(c, s, a, b):
    y = some_func(c)  # Note that x is missing here
    ...
    npt.assert_almost_equal(the_correct_answer, y)

Unfortunately, we also need to use pytest.mark to pass test_data and pytest.mark is incompatible with @gen_cluster so this will not work:

@pytest.mark.filterwarnings("ignore:\\s+Port 8787 is already in use:UserWarning")
@pytest.mark.parametrize("x", test_data)
@gen_cluster(client=True)
def test_some(c, s, a, b):
    y = some_func(c)  # Note that x is missing here
    ...
    npt.assert_almost_equal(the_correct_answer, y)

Versions

coverage               5.0
pytest                 5.3.4
pytest-cov             2.8.1
python                 3.7.1

[ionelmc]

You should also mention what version of coverage and python.

[seanlaw]

You should also mention what version of coverage and python.

Done. Thank you!

[seanlaw]

In case it matters, I will need the solution to work for Python 3.6 as well