parameterized icon indicating copy to clipboard operation
parameterized copied to clipboard
verified publisher icon wolever

0.8.1: pytest based test suite is failing

Open kloczek opened this issue 4 years ago • 5 comments 

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-parameterized-0.8.1-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-parameterized-0.8.1-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/python3 -Bm pytest -ra parameterized/test.py
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.9, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
rootdir: /home/tkloczko/rpmbuild/BUILD/parameterized-0.8.1
plugins: forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, asyncio-0.14.0, expect-1.1.0, cov-2.11.1, mock-3.5.1, httpbin-1.0.0, xdist-2.2.1, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, pyfakefs-4.4.0, freezegun-0.4.2, flaky-3.7.0, cases-3.4.6, hypothesis-6.10.0
collected 59 items

parameterized/test.py ...........xx......x......xx....................x.s..xxxx..E                                                                                   [100%]

================================================================================== ERRORS ==================================================================================
___________________________________________________ ERROR at teardown of TestUnicodeDocstring.test_with_docstring_1_v_l_ ___________________________________________________

    def tearDownModule():
        missing = sorted(list(missing_tests))
>       assert_equal(missing, [])

parameterized/test.py:371:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib64/python3.8/unittest/case.py:912: in assertEqual
    assertion_func(first, second, msg=msg)
/usr/lib64/python3.8/unittest/case.py:1118: in assertListEqual
    self.assertSequenceEqual(list1, list2, msg, seq_type=list)
/usr/lib64/python3.8/unittest/case.py:1100: in assertSequenceEqual
    self.fail(msg)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <nose.tools.trivial.Dummy testMethod=nop>
msg = 'Lists differ: ["test_instance_method(\'foo0\', bar=None)",[461 chars]o\')"] != []\n\nFirst list contains 12 additiona...1, \'umask\', \'getpid\')",\n-  "test_patch_no_expand(42, 51, \'umask\')",\n-  "test_wrapped_iterable_input(\'foo\')"]'

    def fail(self, msg=None):
        """Fail immediately, with the given message."""
>       raise self.failureException(msg)
E       AssertionError: Lists differ: ["test_instance_method('foo0', bar=None)",[461 chars]o')"] != []
E
E       First list contains 12 additional elements.
E       First extra element 0:
E       "test_instance_method('foo0', bar=None)"
E
E       + []
E       - ["test_instance_method('foo0', bar=None)",
E       -  "test_instance_method('foo1', bar=None)",
E       -  "test_instance_method('foo2', bar=42)",
E       -  'test_instance_method(42, bar=None)',
E       -  "test_mock_patch_standalone_function(42, 'umask')",
E       -  "test_naked_function('foo0', bar=None)",
E       -  "test_naked_function('foo1', bar=None)",
E       -  "test_naked_function('foo2', bar=42)",
E       -  'test_naked_function(42, bar=None)',
E       -  "test_patch_class_no_expand(42, 51, 'umask', 'getpid')",
E       -  "test_patch_no_expand(42, 51, 'umask')",
E       -  "test_wrapped_iterable_input('foo')"]

/usr/lib64/python3.8/unittest/case.py:753: AssertionError
============================================================================= warnings summary =============================================================================
parameterized/test.py:55
  parameterized/test.py:55: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_naked_function will be ignored
    @parameterized(test_params)

parameterized/test.py:201
  parameterized/test.py:201: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_mock_patch_standalone_function will be ignored
    @parameterized([(42, )])

parameterized/test.py:356
  parameterized/test.py:356: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_wrapped_iterable_input will be ignored
    @parameterized(lambda: iter(["foo"]))

parameterized/test.py:401
  parameterized/test.py:401: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_parameterized_argument_value_pairs will be ignored
    @parameterized([

parameterized/test.py:424
  parameterized/test.py:424: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_short_repr will be ignored
    @parameterized([

parameterized/test.py:433
  parameterized/test.py:433: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_with_docstring will be ignored
    @parameterized([

parameterized/test.py:443
  parameterized/test.py:443: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_cases_over_10 will be ignored
    @parameterized(cases_over_10)

parameterized/test.py:68
  parameterized/test.py:68: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_instance_method will be ignored
    @parameterized(test_params)

parameterized/test.py:146
  parameterized/test.py:146: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_patch_class_no_expand will be ignored
    @parameterized([(42, 51)])

parameterized/test.py:190
  parameterized/test.py:190: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_patch_no_expand will be ignored
    @parameterized([(42, 51)])

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================================= short test summary info ==========================================================================
SKIPPED [1] ../../../../../usr/lib/python3.8/site-packages/_pytest/unittest.py:355: Py3 doesn't have old-style classes
XFAIL parameterized/test.py::test_naked_function
  reason: [NOTRUN] yield tests were removed in pytest 4.0 - test_naked_function will be ignored
XFAIL parameterized/test.py::TestParameterized::test_instance_method
  reason: [NOTRUN] yield tests were removed in pytest 4.0 - test_instance_method will be ignored
XFAIL parameterized/test.py::TestParameterizedExpandWithNoExpand::test_patch_class_no_expand
  reason: [NOTRUN] yield tests were removed in pytest 4.0 - test_patch_class_no_expand will be ignored
XFAIL parameterized/test.py::TestParameterizedExpandWithNoMockPatchForClassNoExpand::test_patch_no_expand
  reason: [NOTRUN] yield tests were removed in pytest 4.0 - test_patch_no_expand will be ignored
XFAIL parameterized/test.py::test_mock_patch_standalone_function
  reason: [NOTRUN] yield tests were removed in pytest 4.0 - test_mock_patch_standalone_function will be ignored
XFAIL parameterized/test.py::test_wrapped_iterable_input
  reason: [NOTRUN] yield tests were removed in pytest 4.0 - test_wrapped_iterable_input will be ignored
XFAIL parameterized/test.py::test_parameterized_argument_value_pairs
  reason: [NOTRUN] yield tests were removed in pytest 4.0 - test_parameterized_argument_value_pairs will be ignored
XFAIL parameterized/test.py::test_short_repr
  reason: [NOTRUN] yield tests were removed in pytest 4.0 - test_short_repr will be ignored
XFAIL parameterized/test.py::test_with_docstring
  reason: [NOTRUN] yield tests were removed in pytest 4.0 - test_with_docstring will be ignored
XFAIL parameterized/test.py::test_cases_over_10
  reason: [NOTRUN] yield tests were removed in pytest 4.0 - test_cases_over_10 will be ignored
ERROR parameterized/test.py::TestUnicodeDocstring::test_with_docstring_1_v_l_ - AssertionError: Lists differ: ["test_instance_method('foo0', bar=None)",[461 chars]o')"] ...
===================================================== 48 passed, 1 skipped, 10 xfailed, 10 warnings, 1 error in 0.98s ======================================================

kloczek avatar Apr 25 '21 20:04 kloczek

Looks like test suite is using nose which should not be used with python 3.x https://nose.readthedocs.io/en/latest/ So that issue needs to be resolved first :)

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-parameterized-0.8.1-2.1.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-parameterized-0.8.1-2.1.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra parameterized/test.py --deselect parameterized/test.py::TestUnicodeDocstring::test_with_docstring_0_value1 --deselect parameterized/test.py::TestUnicodeDocstring::test_with_docstring_1_v_l_
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/parameterized-0.8.1
collected 0 items / 1 error

================================================================================== ERRORS ==================================================================================
__________________________________________________________________ ERROR collecting parameterized/test.py __________________________________________________________________
ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/parameterized-0.8.1/parameterized/test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib64/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
parameterized/test.py:6: in <module>
    from nose.tools import assert_equal, assert_raises
E   ModuleNotFoundError: No module named 'nose'
========================================================================= short test summary info ==========================================================================
ERROR parameterized/test.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================= 1 error in 0.17s =============================================================================

kloczek avatar Jan 02 '22 10:01 kloczek 

paul@mulciber ~/packages/bugs/python-parameterized $ python3.9 -m pytest -v --pyargs parameterized.test

The test has 2 items, but results in the test being run 3 times, where the third time fails because it wasn't expected AFAICT.

parameterized/test.py::TestUnicodeDocstring::test_with_docstring_0_value1 PASSED      [ 98%]
parameterized/test.py::TestUnicodeDocstring::test_with_docstring_1_v_l_ PASSED        [100%]
parameterized/test.py::TestUnicodeDocstring::test_with_docstring_1_v_l_ ERROR         [100%]

paulgevers avatar Jan 31 '22 18:01 paulgevers 

paul@mulciber ~/packages/bugs/python-parameterized $ python3.9 -m pytest -v --pyargs parameterized.test

On execution python -m foo python automatically adds upfront to sys.path curent direstory. pytest shound not be used that way and this is why is provided /usr/nin/pytest script.

kloczek avatar Feb 03 '22 12:02 kloczek

/usr/bin/pytest-3 fails in the same way:

paul@mulciber ~/packages/bugs/python-parameterized.team $ pytest-3 parameterized/test.py 
==================================== test session starts ====================================
platform linux -- Python 3.9.10, pytest-6.2.5, py-1.10.0, pluggy-0.13.0
rootdir: /home/paul/packages/bugs/python-parameterized.team
collected 59 items                                                                          

parameterized/test.py ...........xx......x......xx....................x.s..xxxx..E    [100%]

========================================== ERRORS ===========================================
___________ ERROR at teardown of TestUnicodeDocstring.test_with_docstring_1_v_l_ ____________

    def tearDownModule():
        missing = sorted(list(missing_tests))
>       assert_equal(missing, [])

parameterized/test.py:371: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.9/unittest/case.py:837: in assertEqual
    assertion_func(first, second, msg=msg)
/usr/lib/python3.9/unittest/case.py:1043: in assertListEqual
    self.assertSequenceEqual(list1, list2, msg, seq_type=list)
/usr/lib/python3.9/unittest/case.py:1025: in assertSequenceEqual
    self.fail(msg)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <nose.tools.trivial.Dummy testMethod=nop>
msg = 'Lists differ: ["test_instance_method(\'foo0\', bar=None)",[461 chars]o\')"] != []\n\nFirst list contains 12 additiona...1, \'umask\', \'getpid\')",\n-  "test_patch_no_expand(42, 51, \'umask\')",\n-  "test_wrapped_iterable_input(\'foo\')"]'

    def fail(self, msg=None):
        """Fail immediately, with the given message."""
>       raise self.failureException(msg)
E       AssertionError: Lists differ: ["test_instance_method('foo0', bar=None)",[461 chars]o')"] != []
E       
E       First list contains 12 additional elements.
E       First extra element 0:
E       "test_instance_method('foo0', bar=None)"
E       
E       + []
E       - ["test_instance_method('foo0', bar=None)",
E       -  "test_instance_method('foo1', bar=None)",
E       -  "test_instance_method('foo2', bar=42)",
E       -  'test_instance_method(42, bar=None)',
E       -  "test_mock_patch_standalone_function(42, 'umask')",
E       -  "test_naked_function('foo0', bar=None)",
E       -  "test_naked_function('foo1', bar=None)",
E       -  "test_naked_function('foo2', bar=42)",
E       -  'test_naked_function(42, bar=None)',
E       -  "test_patch_class_no_expand(42, 51, 'umask', 'getpid')",
E       -  "test_patch_no_expand(42, 51, 'umask')",
E       -  "test_wrapped_iterable_input('foo')"]

/usr/lib/python3.9/unittest/case.py:676: AssertionError

paulgevers avatar Feb 03 '22 13:02 paulgevers

Also, please consider to use unittest.mock instead of mock, as a fallback at least.

pgajdos avatar Apr 07 '22 12:04 pgajdos