Issues when using pytest-cov for project with sklearn

[mdscruggs]

My repo contains 3 Python projects in a "projects" dir, and I'm running pytest --cov=packageA projects/packageA/tests (for each of the 3 projects). The 2nd project in the test sequence (so the 2nd pytest --cov run, out of 3) causes some lingering process-specific .coverage data files, which seems to impact the 3rd project's pytest run.

I've documented the issue with steps to reproduce in this repo (some details there about why this issue is here and not with coverage itself): https://github.com/mdscruggs/pytest-coverage-sklearn-issue

Happy to answer questions and assist.

> coverage debug config data sys

-- config ----------------------------------------------------
 attempted_config_files: .coveragerc
                         setup.cfg
                         tox.ini
           config_files: -none-
               _include: None
                  _omit: None
                 branch: False
            concurrency: None
            cover_pylib: False
              data_file: .coverage
                  debug: -none-
       disable_warnings: -none-
                   note: None
               parallel: False
                plugins: -none-
                 source: None
            run_include: None
               run_omit: None
                  timid: False
           exclude_list: #\s*(pragma|PRAGMA)[:\s]?\s*(no|NO)\s*(cover|COVER)
             fail_under: 0.0
          ignore_errors: False
         report_include: None
            report_omit: None
    partial_always_list: while (True|1|False|0):
                         if (True|1|False|0):
           partial_list: #\s*(pragma|PRAGMA)[:\s]?\s*(no|NO)\s*(branch|BRANCH)
              precision: 0
           show_missing: False
           skip_covered: False
              extra_css: None
               html_dir: htmlcov
             html_title: Coverage report
             xml_output: coverage.xml
      xml_package_depth: 99
                  paths: {}
         plugin_options: {}
ds: None
-- data ------------------------------------------------------
path: /Users/user/repos/coverage-sklearn/.coverage
has_arcs: False

2 files:
/Users/user/repos/coverage-sklearn/mycode.py: 2 lines
/Users/user/repos/coverage-sklearn/othercode.py: 1 lines
-- sys -------------------------------------------------------
              version: 4.5.3
             coverage: /Users/user/repos/coverage-sklearn/venv/lib/python3.6/site-packages/coverage/__init__.py
          cover_paths: /Users/user/repos/coverage-sklearn/venv/lib/python3.6/site-packages/coverage
          pylib_paths: /Users/user/anaconda3/lib/python3.6
               tracer: CTracer
 plugins.file_tracers: -none-
  plugins.configurers: -none-
         config_files: .coveragerc
                       setup.cfg
                       tox.ini
         configs_read: -none-
            data_path: /Users/user/repos/coverage-sklearn/.coverage
               python: 3.6.5 |Anaconda custom (64-bit)| (default, Apr 26 2018, 08:42:37) [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]
             platform: Darwin-17.7.0-x86_64-i386-64bit
       implementation: CPython
           executable: /Users/user/repos/coverage-sklearn/venv/bin/python3
                  cwd: /Users/user/repos/coverage-sklearn
                 path: 
                       /Users/user/anaconda3/lib/python36.zip
                       /Users/user/anaconda3/lib/python3.6
                       /Users/user/anaconda3/lib/python3.6/lib-dynload
                       /Users/user/repos/coverage-sklearn/venv/lib/python3.6/site-packages
          environment: PYSPARK_DRIVER_PYTHON_OPTS = -m jupyter notebook
         command_line: /Users/user/repos/coverage-sklearn/venv/bin/coverage debug config data sys
         source_match: -none-
    source_pkgs_match: -none-
        include_match: -none-
           omit_match: -none-
          cover_match: /Users/user/repos/coverage-sklearn/venv/lib/python3.6/site-packages/coverage
          pylib_match: /Users/user/anaconda3/lib/python3.6

[ionelmc]

@mdscruggs just to check: have you tried the master of pytest-cov? Cause the master has quite some changes around subprocess handling ...

[mdscruggs]

I just tried per your suggestion @ionelmc and the behavior + results appears the same.

[ionelmc]

So it appears that the lingering process starts with this:

execve("/tmp/pytest-coverage-sklearn-issue/venv/bin/python3", ["/tmp/pytest-coverage-sklearn-issue/venv/bin/python3", "-Wignore:A true SSLContext object is not available", "-c", "from multiprocessing.semaphore_tracker import main;main(8)"]

Obviously coveragepy alone won't track this process thus the assessment that pytest-cov does cleanup that coveragepy does is wrong. Coveragepy doesn't even know about that process.

~~Anyway, the situation is a bit tricky - pytest-cov has api for extra cleanup handler from a signal (pytest_cov.embed.cleanup_on_signal) however it cannot be used cause there's no place to put that code. We could have an env var to enable the handler I guess ...~~ - actually I doubt anyone tries to shutdown/sigterm that semaphore_tracker ...

An alternative solution is to make sure the semaphore_tracker is shut down before the test is done.

[ionelmc]

You can see the difference by measuring coverage on multiprocessing.semaphore_tracker.

pytest-cov could specifically handle cleanup for this semaphore_tracker but then again, maybe a more generic solution would be more useful. What if pytest-cov would wait for all the subprocesses to exit? That could get test suites stuck tho ...

[ionelmc]

@schlamar or @blueyed please share your opinion. These are the options:

• Implement specific handling for multiprocessing.semaphore_tracker. Will probably involve gross monkeypatching.
• Implement generic subprocess waiting. Can get tests stuck when users forget to shutdown subprocesses.