New falsifying example for test_isort_is_idempotent

[bnavigator]

The current openSUSE rpm build of isort 5.12 finds this failing example with a high probability:

[  147s] ============================= test session starts ==============================
[  147s] platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.0.0 -- /usr/bin/python3.11
[  147s] cachedir: .pytest_cache
[  147s] hypothesis profile 'obs' -> deadline=timedelta(milliseconds=5000), suppress_health_check=[HealthCheck.too_slow], database=DirectoryBasedExampleDatabase('/home/abuild/rpmbuild/BUILD/isort-5.12.0/.hypothesis/examples')
[  147s] rootdir: /home/abuild/rpmbuild/BUILD/isort-5.12.0
[  147s] plugins: hypothesis-6.82.4, mock-3.11.1, pylama-8.4.1
[  151s] collecting ... collected 532 items / 2 deselected / 530 selected
[  151s] 
[ 4951s] tests/integration/test_hypothesmith.py::test_isort_is_idempotent FAILED  [  0%]
...
[ 4975s] ___________________________ test_isort_is_idempotent ___________________________
[ 4975s] 
[ 4975s]     @hypothesis.example("import A\nimportA\r\n\n", isort.Config(), False)
[ 4975s] >   @hypothesis.given(
[ 4975s]         source_code=st.lists(
[ 4975s]             from_grammar(auto_target=False)
[ 4975s]             | from_node(auto_target=False)
[ 4975s]             | from_node(libcst.Import, auto_target=False)
[ 4975s]             | from_node(libcst.ImportFrom, auto_target=False),
[ 4975s]             min_size=1,
[ 4975s]             max_size=10,
[ 4975s]         ).map("\n".join),
[ 4975s]         config=st.builds(isort.Config),
[ 4975s]         disregard_skip=st.booleans(),
[ 4975s]     )
[ 4975s] 
[ 4975s] tests/integration/test_hypothesmith.py:76: 
[ 4975s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[ 4975s] 
[ 4975s] source_code = 'from A.\\\n\x0cA import A'
[ 4975s] config = Config(py_version='py3', force_to_top=frozenset(), skip=frozenset({'.pants.d', '_build', '.mypy_cache', '.svn', '.eggs...ge}', format_success='{success}: {message}', sort_order='natural', sort_reexports=False, split_on_trailing_comma=False)
[ 4975s] disregard_skip = False
[ 4975s] 
[ 4975s]     @hypothesis.example("import A\nimportA\r\n\n", isort.Config(), False)
[ 4975s]     @hypothesis.given(
[ 4975s]         source_code=st.lists(
[ 4975s]             from_grammar(auto_target=False)
[ 4975s]             | from_node(auto_target=False)
[ 4975s]             | from_node(libcst.Import, auto_target=False)
[ 4975s]             | from_node(libcst.ImportFrom, auto_target=False),
[ 4975s]             min_size=1,
[ 4975s]             max_size=10,
[ 4975s]         ).map("\n".join),
[ 4975s]         config=st.builds(isort.Config),
[ 4975s]         disregard_skip=st.booleans(),
[ 4975s]     )
[ 4975s]     @hypothesis.seed(235738473415671197623909623354096762459)
[ 4975s]     @hypothesis.settings(
[ 4975s]         suppress_health_check=[hypothesis.HealthCheck.too_slow, hypothesis.HealthCheck.filter_too_much]
[ 4975s]     )
[ 4975s]     def test_isort_is_idempotent(source_code: str, config: isort.Config, disregard_skip: bool) -> None:
[ 4975s]         # NOTE: if this test finds a bug, please notify @Zac-HD so that it can be added to the
[ 4975s]         #       Hypothesmith trophy case.  This really helps with research impact evaluations!
[ 4975s]         _record_targets(source_code)
[ 4975s]         result = isort.code(source_code, config=config, disregard_skip=disregard_skip)
[ 4975s] >       assert result == isort.code(result, config=config, disregard_skip=disregard_skip)
[ 4975s] E       AssertionError: assert 'from A.\\\nA import A\n' == 'from A.A import A\n'
[ 4975s] E         + from A.\
[ 4975s] E         - from A.A import A
[ 4975s] E         ? ----- --
[ 4975s] E         + A import A
[ 4975s] E       Falsifying example: test_isort_is_idempotent(
[ 4975s] E           source_code='from A.\\\n\x0cA import A',
[ 4975s] E           config=Config(),
[ 4975s] E           disregard_skip=False,
[ 4975s] E       )
[ 4975s] E       Highest target scores:
[ 4975s] E                     10  (label='total number of import nodes')
[ 4975s] E                     29  (label='number of unique ast node types')
[ 4975s] 
[ 4975s] tests/integration/test_hypothesmith.py:97: AssertionError

Wondering about the super long runtime of pytest until the failure is reported. :thinking:

It looks like the failure happens since hypothesis got updated from 6.75.3 to 6.82.4

[bnavigator]

Ping @Zac-HD

[bnavigator]

From HEAD Mon Sep 17 00:00:00 2001
From: Hypothesis 6.82.7 <[email protected]>
Date: Tue, 29 Aug 2023 19:15:37
Subject: [PATCH] Hypothesis: add explicit examples

---
--- tests/integration/test_hypothesmith.py
+++ tests/integration/test_hypothesmith.py
@@ -89,6 +89,9 @@
 @hypothesis.settings(
     suppress_health_check=[hypothesis.HealthCheck.too_slow, hypothesis.HealthCheck.filter_too_much]
 )
[email protected](
+    source_code="from A.\\\n\x0cA import A", config=Config(), disregard_skip=False
+).via("discovered failure")
 def test_isort_is_idempotent(source_code: str, config: isort.Config, disregard_skip: bool) -> None:
     # NOTE: if this test finds a bug, please notify @Zac-HD so that it can be added to the
     #       Hypothesmith trophy case.  This really helps with research impact evaluations!

[Zac-HD]

See also https://github.com/Instagram/LibCST/issues/446 - \x0c is form-feed, and supported as leading whitespace in indentation for historical reasons in Python (I think it resets the column count in Emacs?). Presumably recent changes to Hypothesis made us more likely to generate this and trigger a latent failure.