codecov-python icon indicating copy to clipboard operation
codecov-python copied to clipboard

Make test suite independent of .git directory

Open yzgyyang opened this issue 3 years ago • 2 comments

By simply removing the .git directory, tests will begin to fail (I got "12 failed, 38 passed, 20 skipped in 3.19s"). It seems like the suite is dependent on the .git directory of the source.

=============================================== short test summary info ================================================
FAILED tests/test.py::TestUploader::test_bowerrc - AttributeError: 'FileNotFoundError' object has no attribute 'retur...
FAILED tests/test.py::TestUploader::test_bowerrc_none - AttributeError: 'FileNotFoundError' object has no attribute '...
FAILED tests/test.py::TestUploader::test_disable_search - AttributeError: 'FileNotFoundError' object has no attribute...
FAILED tests/test.py::TestUploader::test_discovers - AttributeError: 'FileNotFoundError' object has no attribute 'ret...
FAILED tests/test.py::TestUploader::test_include_env - AttributeError: 'FileNotFoundError' object has no attribute 'r...
FAILED tests/test.py::TestUploader::test_none_found - AttributeError: 'FileNotFoundError' object has no attribute 're...
FAILED tests/test.py::TestUploader::test_not_jacoco - AttributeError: 'FileNotFoundError' object has no attribute 're...
FAILED tests/test.py::TestUploader::test_prefix - AttributeError: 'FileNotFoundError' object has no attribute 'return...
FAILED tests/test.py::TestUploader::test_read_token_file - AttributeError: 'FileNotFoundError' object has no attribut...
FAILED tests/test.py::TestUploader::test_returns_none - SystemExit: 0
FAILED tests/test.py::TestUploader::test_run_coverage_fails - AttributeError: 'FileNotFoundError' object has no attri...
FAILED tests/test.py::TestUploader::test_send - AttributeError: 'FileNotFoundError' object has no attribute 'returncode'
================================= 12 failed, 38 passed, 20 skipped, 1 warning in 8.09s =================================

Use Case: This happens if one downloads a release tarball (which doesn't include .git) and try to run the test suite. The situation happens when we try to include codecov in the FreeBSD Ports Collection - according to the FreeBSD Policy [1]: Python ports must include a do-test target if the package includes tests. We are currently removing failed tests manually, but would love to have this issue fixed upstream, so that the test suite will no longer depend on the .git directory.

This is essentially the same problem reported in #100.

[1] https://wiki.freebsd.org/Python/PortsPolicy#Tests

yzgyyang avatar Nov 24 '20 11:11 yzgyyang

@yzgyyang thanks for this. Do you happen to be running this in a public repository, and if so, can you link to a CI build?

thomasrockhu avatar Nov 26 '20 21:11 thomasrockhu

@yzgyyang thanks for this. Do you happen to be running this in a public repository, and if so, can you link to a CI build?

I don't have the CI build results on hand, but essentially by the following steps:

  • Download and unzip https://github.com/codecov/codecov-python/archive/v2.1.10.zip
  • Install test dependencies and run pytest tests/test.py (Since releases don't ship with the .git directory)

Or:

  • Clone the repo
  • Remove .git directory
  • Install test dependencies and run pytest tests/test.py

Since I am trying to commit the latest version to FreeBSD Ports 1, I am essentially using a patch to disable all tests that seem to fail when removed .git. 2

Haven't really dig into the tests, but IMO the tests shouldn't be dependent on the actual .git directory of the repo (maybe using a mocked one that ships with the test suite instead?).

yzgyyang avatar Nov 27 '20 03:11 yzgyyang