yamllint
yamllint copied to clipboard
Published
20 hours ago •
adrienverge
adrienverge
4 tests fail to run pas on MacOS
There are 3 tests that fail on MacOS, and I even created https://github.com/adrienverge/yamllint/pull/303 to prove that would reproduce on Travis, if we would run at least one job using MacOS.
The good part is that apparently only the tests are failing as I am using the tool daily without problems.
======================================================================
FAIL: test_locale_accents (tests.rules.test_key_ordering.KeyOrderingTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/ssbarnea/c/os/yamllint/tests/rules/test_key_ordering.py", line 142, in test_locale_accents
self.check('---n'
File "/Users/ssbarnea/c/os/yamllint/tests/common.py", line 53, in check
self.assertEqual(real_problems, expected_problems)
AssertionError: Lists differ: [4:1: wrong ordering of key "hais" in mapping (key-ordering)] != []
First list contains 1 additional elements.
First extra element 0:
4:1: wrong ordering of key "hais" in mapping (key-ordering)
- [4:1: wrong ordering of key "hais" in mapping (key-ordering)]
+ []
======================================================================
FAIL: test_locale_case (tests.rules.test_key_ordering.KeyOrderingTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/ssbarnea/c/os/yamllint/tests/rules/test_key_ordering.py", line 123, in test_locale_case
self.check('---n'
File "/Users/ssbarnea/c/os/yamllint/tests/common.py", line 53, in check
self.assertEqual(real_problems, expected_problems)
AssertionError: Lists differ: [3:1: wrong ordering of key "T-shirt" in m[82 chars]ing)] != []
First list contains 2 additional elements.
First extra element 0:
3:1: wrong ordering of key "T-shirt" in mapping (key-ordering)
+ []
- [3:1: wrong ordering of key "T-shirt" in mapping (key-ordering),
- 5:1: wrong ordering of key "T-shirts" in mapping (key-ordering)]
======================================================================
FAIL: test_run_with_locale (tests.test_cli.CommandLineTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/ssbarnea/c/os/yamllint/tests/test_cli.py", line 376, in test_run_with_locale
self.assertEqual(ctx.returncode, 0)
AssertionError: 1 != 0
======================================================================
FAIL: test_run_with_user_global_config_file (tests.test_cli.CommandLineTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/ssbarnea/c/os/yamllint/tests/test_cli.py", line 317, in test_run_with_user_global_config_file
self.assertEqual(ctx.returncode, 0)
AssertionError: 1 != 0
Closed-source platforms are supported on "best-effort", so if you have a ready-to-go solution (not a draft PR, please), perfect :ok_hand:
Otherwise, I suggest installing GNU/Linux on your Mac :+1:
I am planning to have a look on that. BTW, I have a strong feeling the issue may be reproduced with any BSD based system, and that is open-source too ;)
3 tests are failing on OpenBSD 6.7 amd64 on latest stable yamllint version:
===> Regression tests for py3-yamllint-1.24.2
============================= test session starts ==============================
platform openbsd6 -- Python 3.7.7, pytest-4.4.0, py-1.8.0, pluggy-0.11.0
rootdir: /usr/ports/pobj/py-yamllint-1.24.2-python3/yamllint-1.24.2
collected 350 items
tests/test_cli.py .......................F.. [ 7%]
tests/test_config.py .................. [ 12%]
tests/test_linter.py ....... [ 14%]
tests/test_module.py .... [ 15%]
tests/test_parser.py ... [ 16%]
tests/test_spec_examples.py ............................................ [ 29%]
................................................. [ 43%]
tests/test_syntax_errors.py ..... [ 44%]
tests/test_yamllint_directives.py ........... [ 47%]
tests/rules/test_braces.py ........ [ 50%]
tests/rules/test_brackets.py ........ [ 52%]
tests/rules/test_colons.py .......... [ 55%]
tests/rules/test_commas.py ........ [ 57%]
tests/rules/test_comments.py ............ [ 60%]
tests/rules/test_comments_indentation.py ...... [ 62%]
tests/rules/test_common.py . [ 62%]
tests/rules/test_document_end.py .... [ 64%]
tests/rules/test_document_start.py ..... [ 65%]
tests/rules/test_empty_lines.py ........ [ 67%]
tests/rules/test_empty_values.py ................. [ 72%]
tests/rules/test_hyphens.py ... [ 73%]
tests/rules/test_indentation.py ........................................ [ 84%]
.... [ 86%]
tests/rules/test_key_duplicates.py ... [ 86%]
tests/rules/test_key_ordering.py ......FF. [ 89%]
tests/rules/test_line_length.py ........ [ 91%]
tests/rules/test_new_line_at_end_of_file.py .. [ 92%]
tests/rules/test_new_lines.py .... [ 93%]
tests/rules/test_octal_values.py ... [ 94%]
tests/rules/test_quoted_strings.py .......... [ 97%]
tests/rules/test_trailing_spaces.py ... [ 98%]
tests/rules/test_truthy.py ....... [100%]
=================================== FAILURES ===================================
___________________ CommandLineTestCase.test_run_with_locale ___________________
self = <tests.test_cli.CommandLineTestCase testMethod=test_run_with_locale>
def test_run_with_locale(self):
# check for availability of locale, otherwise skip the test
# reset to default before running the test,
# as the first two runs don't use setlocale()
try:
locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')
except locale.Error:
self.skipTest('locale en_US.UTF-8 not available')
locale.setlocale(locale.LC_ALL, (None, None))
# C + en.yaml should fail
with RunContext(self) as ctx:
cli.run(('-d', 'rules: { key-ordering: enable }',
os.path.join(self.wd, 'en.yaml')))
self.assertEqual(ctx.returncode, 1)
# C + c.yaml should pass
with RunContext(self) as ctx:
cli.run(('-d', 'rules: { key-ordering: enable }',
os.path.join(self.wd, 'c.yaml')))
self.assertEqual(ctx.returncode, 0)
# the next two runs use setlocale() inside,
# so we need to clean up afterwards
self.addCleanup(locale.setlocale, locale.LC_ALL, (None, None))
# en_US + en.yaml should pass
with RunContext(self) as ctx:
cli.run(('-d', 'locale: en_US.UTF-8\n'
'rules: { key-ordering: enable }',
os.path.join(self.wd, 'en.yaml')))
> self.assertEqual(ctx.returncode, 0)
E AssertionError: 1 != 0
tests/test_cli.py:364: AssertionError
___________________ KeyOrderingTestCase.test_locale_accents ____________________
self = <tests.rules.test_key_ordering.KeyOrderingTestCase testMethod=test_locale_accents>
def test_locale_accents(self):
self.addCleanup(locale.setlocale, locale.LC_ALL, (None, None))
try:
locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')
except locale.Error:
self.skipTest('locale en_US.UTF-8 not available')
conf = ('key-ordering: enable')
self.check('---\n'
'hair: true\n'
'haïr: true\n'
'hais: true\n'
> 'haïssable: true\n', conf)
tests/rules/test_key_ordering.py:146:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/common.py:53: in check
self.assertEqual(real_problems, expected_problems)
E AssertionError: Lists differ: [4:1: wrong ordering of key "hais" in mapping (key-ordering)] != []
E
E First list contains 1 additional elements.
E First extra element 0:
E 4:1: wrong ordering of key "hais" in mapping (key-ordering)
E
E - [4:1: wrong ordering of key "hais" in mapping (key-ordering)]
E + []
_____________________ KeyOrderingTestCase.test_locale_case _____________________
self = <tests.rules.test_key_ordering.KeyOrderingTestCase testMethod=test_locale_case>
def test_locale_case(self):
self.addCleanup(locale.setlocale, locale.LC_ALL, (None, None))
try:
locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')
except locale.Error:
self.skipTest('locale en_US.UTF-8 not available')
conf = ('key-ordering: enable')
self.check('---\n'
't-shirt: 1\n'
'T-shirt: 2\n'
't-shirts: 3\n'
> 'T-shirts: 4\n', conf)
tests/rules/test_key_ordering.py:127:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/common.py:53: in check
self.assertEqual(real_problems, expected_problems)
E AssertionError: Lists differ: [3:1: wrong ordering of key "T-shirt" in m[82 chars]ing)] != []
E
E First list contains 2 additional elements.
E First extra element 0:
E 3:1: wrong ordering of key "T-shirt" in mapping (key-ordering)
E
E + []
E - [3:1: wrong ordering of key "T-shirt" in mapping (key-ordering),
E - 5:1: wrong ordering of key "T-shirts" in mapping (key-ordering)]
=============================== warnings summary ===============================
tests/test_cli.py::CommandLineTestCase::test_run_non_existing_file
/usr/ports/pobj/py-yamllint-1.24.2-python3/yamllint-1.24.2/tests/test_cli.py:386: DeprecationWarning: Please use assertRegex instead.
self.assertRegexpMatches(ctx.stderr, r'No such file or directory')
tests/test_cli.py::CommandLineTestCase::test_run_version
/usr/ports/pobj/py-yamllint-1.24.2-python3/yamllint-1.24.2/tests/test_cli.py:377: DeprecationWarning: Please use assertRegex instead.
self.assertRegexpMatches(ctx.stdout + ctx.stderr, r'yamllint \d+\.\d+')
tests/test_cli.py::CommandLineTestCase::test_run_with_bad_arguments
/usr/ports/pobj/py-yamllint-1.24.2-python3/yamllint-1.24.2/tests/test_cli.py:237: DeprecationWarning: Please use assertRegex instead.
self.assertRegexpMatches(ctx.stderr, r'^usage')
tests/test_cli.py::CommandLineTestCase::test_run_with_bad_arguments
/usr/ports/pobj/py-yamllint-1.24.2-python3/yamllint-1.24.2/tests/test_cli.py:243: DeprecationWarning: Please use assertRegex instead.
self.assertRegexpMatches(ctx.stderr, r'^usage')
tests/test_cli.py::CommandLineTestCase::test_run_with_bad_arguments
/usr/ports/pobj/py-yamllint-1.24.2-python3/yamllint-1.24.2/tests/test_cli.py:251: DeprecationWarning: Please use assertRegex instead.
r'^yamllint: error: argument -d\/--config-data: '
tests/test_cli.py::CommandLineTestCase::test_run_with_bad_arguments
/usr/ports/pobj/py-yamllint-1.24.2-python3/yamllint-1.24.2/tests/test_cli.py:260: DeprecationWarning: Please use assertRegex instead.
self.assertRegexpMatches(ctx.stderr, r'^usage')
tests/test_cli.py::CommandLineTestCase::test_run_with_bad_config
/usr/ports/pobj/py-yamllint-1.24.2-python3/yamllint-1.24.2/tests/test_cli.py:267: DeprecationWarning: Please use assertRegex instead.
self.assertRegexpMatches(ctx.stderr, r'^invalid config: no such rule')
tests/test_cli.py::CommandLineTestCase::test_run_with_empty_config
/usr/ports/pobj/py-yamllint-1.24.2-python3/yamllint-1.24.2/tests/test_cli.py:274: DeprecationWarning: Please use assertRegex instead.
self.assertRegexpMatches(ctx.stderr, r'^invalid config: not a dict')
tests/test_config.py::SimpleConfigTestCase::test_unknown_option
/usr/ports/pobj/py-yamllint-1.24.2-python3/yamllint-1.24.2/tests/test_config.py:72: DeprecationWarning: Please use assertRaisesRegex instead.
'invalid config: unknown option "abcdef" for rule "colons"'):
tests/test_config.py::SimpleConfigTestCase::test_unknown_rule
/usr/ports/pobj/py-yamllint-1.24.2-python3/yamllint-1.24.2/tests/test_config.py:53: DeprecationWarning: Please use assertRaisesRegex instead.
'invalid config: no such rule: "this-one-does-not-exist"'):
tests/test_config.py::SimpleConfigTestCase::test_yes_no_for_booleans
/usr/ports/pobj/py-yamllint-1.24.2-python3/yamllint-1.24.2/tests/test_config.py:110: DeprecationWarning: Please use assertRaisesRegex instead.
'invalid config: option "indent-sequences" of "indentation" '
tests/test_module.py::ModuleTestCase::test_run_module_no_args
/usr/ports/pobj/py-yamllint-1.24.2-python3/yamllint-1.24.2/tests/test_module.py:51: DeprecationWarning: Please use assertRegex instead.
r'^usage: yamllint')
tests/test_module.py::ModuleTestCase::test_run_module_on_bad_dir
/usr/ports/pobj/py-yamllint-1.24.2-python3/yamllint-1.24.2/tests/test_module.py:59: DeprecationWarning: Please use assertRegex instead.
r'No such file or directory')
-- Docs: https://docs.pytest.org/en/latest/warnings.html
Apparently neither travis or github-actions support OpenBSD/FreeBSD but they do support macos, which is quite close from out point of view. Apparently the errors you found match mine. I could try to run the tests in my router or spawn a VM, but I am confident I will get similar failure related to how locales work.
Just got the same test failures for 1.26.0 building on Alpinelinux (musl) ...looks missing locales
**EDIT** only 3 tests and log
.................................................................................................................................................FF................................................................F................................................................................................................................................
======================================================================
FAIL: test_locale_accents (tests.rules.test_key_ordering.KeyOrderingTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/builds/andypost/aports/community/yamllint/src/yamllint-1.26.0/tests/rules/test_key_ordering.py", line 142, in test_locale_accents
self.check('---\n'
File "/builds/andypost/aports/community/yamllint/src/yamllint-1.26.0/tests/common.py", line 53, in check
self.assertEqual(real_problems, expected_problems)
AssertionError: Lists differ: [4:1: wrong ordering of key "hais" in mapping (key-ordering)] != []
First list contains 1 additional elements.
First extra element 0:
4:1: wrong ordering of key "hais" in mapping (key-ordering)
- [4:1: wrong ordering of key "hais" in mapping (key-ordering)]
+ []
======================================================================
FAIL: test_locale_case (tests.rules.test_key_ordering.KeyOrderingTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/builds/andypost/aports/community/yamllint/src/yamllint-1.26.0/tests/rules/test_key_ordering.py", line 123, in test_locale_case
self.check('---\n'
File "/builds/andypost/aports/community/yamllint/src/yamllint-1.26.0/tests/common.py", line 53, in check
self.assertEqual(real_problems, expected_problems)
AssertionError: Lists differ: [3:1: wrong ordering of key "T-shirt" in m[82 chars]ing)] != []
First list contains 2 additional elements.
First extra element 0:
3:1: wrong ordering of key "T-shirt" in mapping (key-ordering)
+ []
- [3:1: wrong ordering of key "T-shirt" in mapping (key-ordering),
- 5:1: wrong ordering of key "T-shirts" in mapping (key-ordering)]
======================================================================
FAIL: test_run_with_locale (tests.test_cli.CommandLineTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/builds/andypost/aports/community/yamllint/src/yamllint-1.26.0/tests/test_cli.py", line 373, in test_run_with_locale
self.assertEqual(ctx.returncode, 0)
AssertionError: 1 != 0
----------------------------------------------------------------------
Ran 356 tests in 11.193s
FAILED (failures=3)
@andypost You are right, these failures are related to different default locale settings on these platforms, not misconfigured ones.
Sadly, due to the maintainer lack of interest in supporting anything else than pure-Linux (see his "joke"), I doubt others will be motivated to contribute and improve the codebase and the test coverage. Practical proposals about testing on these platforms were just ignored.
As of today the project CI still used Travis which is close to dead. In fact this is the only project I know that did not switch to something else, usually github actions. Travis will only get worse and will not add support for new platforms.
Don't get me wrong, I do appreciate @adrienverge work on this and I do not expect him to do that kind of work. My only expectation is to get some positive and constructive messages from him that would encourage others to make changes. In the end, he is the one that can approve or reject a PR.
Thank you for elaborated answer, I used just to add patch to skip this test methods. I'm not much deep in python internals to find out how it using locales to figure out what's missing in musl but locales often are issue. Related docs https://wiki.musl-libc.org/functional-differences-from-glibc.html#Character-sets-and-locale