Add coverage reporting for Numba JIT-compiled functions

[vc1492a]

Any functions with the @jit decorator are not read by pytest-cov as having been executed despite them running successfully within the unit tests, as they are compiled by numba as C and thus are not Python functions which are able to be evaluated by pytest-cov.

This occurs even when setting the environment variable to disable numba execution, e.g. NUMBA_DISABLE_JIT = "1".

[vc1492a]

Resolved by disabling Numba prior to pytest execution. Turns out while you can set environmental variables in a Python script, the updated values are only accessible from new processes. Although it's possible to spawn a new child process from within a Python script, this can become a little messy with the test and instead Numba is disabled in order to provide accurate coverage reporting.

I however believe the successful integration of unit tests with Numba would provide the appropriate set of tests for development, so I will be renaming this issue and reassigning its label. In development, unit tests should be executed successfully both with Numba enabled and disabled (only the coverage reporting is affected).

[newskooler]

Is there a plan to add support for this, such that pytest-cov can detect coverage with @jit functions too? It would be really convenient.

P.S. If not, at least a good warning should exist, I guess.

[vc1492a]

Thanks for asking @snenkov! At this time, I do not have a solution in mind to support coverage reporting of JIT-enabled functions within the code base. I read through the two issues referenced above (from other repositories), and it looks like a solution has yet to be found / developed. If you have any suggestions, please let me know!

I agree however that a warning during testing would at least be nice. We can include a warning as part of the next release.

[newskooler]

Thank for the reply @vc1492a . May I ask, how do you handle such function in your unit tests? Do you just test accepting that it will not reflect in coverage? Or do you let coverage ignore it? What's your approach?

I guess when removing the @njit decorator it still performs the function tests and it's reflected in the coverage, but that's a manual fix to the problem (not really a fix).

[vc1492a]

See the unit tests within the repository, line 24 of tests/test_loop.py which contains a flag to enable or disable the use of numba in the unit tests. Numba is installed within the local testing environment (not within the continuous integration environment as reflected in requirements_ci.txt), and that line value set to True when desiring to test the code base with numba. This reduces the coverage reported from 99% to 94%, but still executes each unit test and provides an indication of pass / failure.

We're able to run tests with Numba enabled or disabled locally, but for Travis CI we simply run the testing suite with numba disabled (which also reflects how most users will use the PyNomaly software).

[s-weigand]

Did anyone try simply monkey patching the @jit decorator with a decorator which leaves the function unchanged for the unit test of that given function, and know if that works?

[samyhocine]

Hi, any news on that ? Would be really helpful to be able to do it without disabling numba :/