Certain codemod decorator matchers crash in parallel_exec_transform_with_prettyprint

[nacl]

Hi there!

When writing some codemods, I observed that some of them crash strangely in codemod.parallel_exec_transform_with_prettyprint with a non-1 jobs argument when the number of files is (oddly) 5 or more. No failures seem occur when there are less than 5 files, regardless of the jobs argument.

A minimal example can be found here, based on the current end-to-end test in libcst/tests/test_e2e.py. There are two failures I observed, as mentioned in the comments in the file. Reproduction is simple as running the script.

I've observed this fail in python3.9 and python3.11, with libcst at current top-of-tree (f668e88) on macOS 13.1 arm64. It also reproduces with python 3.9 and libcst 0.4.9 on macOS and Linux arm64, as downloaded from PyPI.

I did not test if this occurs when used the matches are configured using something other than decorators. It might not.

[nacl]

The "less than five" observation seems to be happening because parallel_exec_transform_with_prettyprint batches operations into groups of four.

[smackesey]

I've observed something similar on macOS (Ventura 13.3.1 w/ python 3.10.10). This dead simple codemod fails for me using the default settings. It works fine with -j1:

class GrapheneToStrawberryTransformer(VisitorBasedCodemodCommand):

    @m.call_if_inside(m.ClassDef(name=m.Name("Meta")))
    def visit_Name(self, node):
        print(node)

Here's a backtrace from a file it fails on (they all look like this):

Codemodding /Users/smackesey/stm/code/elementl/oss/python_modules/dagster-graphql/dagster_graphql/schema/roots/subscription.py
Traceback (most recent call last):
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/codemod/_cli.py", line 279, in _execute_transform
    output_tree = transformer.transform_module(input_tree)
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/codemod/_command.py", line 71, in transform_module
    tree = super().transform_module(tree)
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/codemod/_codemod.py", line 108, in transform_module
    return self.transform_module_impl(tree_with_metadata)
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/codemod/_visitor.py", line 32, in transform_module_impl
    return tree.visit(self)
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/_nodes/module.py", line 90, in visit
    result = super(Module, self).visit(visitor)
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/_nodes/base.py", line 228, in visit
    _CSTNodeSelfT, self._visit_and_replace_children(visitor)
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/_nodes/module.py", line 74, in _visit_and_replace_children
    body=visit_body_sequence(self, "body", self.body, visitor),
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/_nodes/internal.py", line 227, in visit_body_sequence
    return tuple(visit_body_iterable(parent, fieldname, children, visitor))
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/_nodes/internal.py", line 193, in visit_body_iterable
    new_child = child.visit(visitor)
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/_nodes/base.py", line 228, in visit
    _CSTNodeSelfT, self._visit_and_replace_children(visitor)
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/_nodes/statement.py", line 443, in _visit_and_replace_children
    body=visit_sequence(self, "body", self.body, visitor),
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/_nodes/internal.py", line 177, in visit_sequence
    return tuple(visit_iterable(parent, fieldname, children, visitor))
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/_nodes/internal.py", line 159, in visit_iterable
    new_child = child.visit(visitor)
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/_nodes/base.py", line 228, in visit
    _CSTNodeSelfT, self._visit_and_replace_children(visitor)
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/_nodes/statement.py", line 1251, in _visit_and_replace_children
    names=visit_sequence(self, "names", self.names, visitor),
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/_nodes/internal.py", line 177, in visit_sequence
    return tuple(visit_iterable(parent, fieldname, children, visitor))
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/_nodes/internal.py", line 159, in visit_iterable
    new_child = child.visit(visitor)
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/_nodes/base.py", line 228, in visit
    _CSTNodeSelfT, self._visit_and_replace_children(visitor)
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/_nodes/statement.py", line 1170, in _visit_and_replace_children
    name=visit_required(self, "name", self.name, visitor),
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/_nodes/internal.py", line 81, in visit_required
    result = node.visit(visitor)
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/_nodes/base.py", line 219, in visit
    should_visit_children = visitor.on_visit(self)
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/matchers/_visitors.py", line 504, in on_visit
    if not _should_allow_visit(
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/matchers/_visitors.py", line 430, in _should_allow_visit
    return _all_positive_matchers_true(
  File "/Users/smackesey/stm/code/elementl/oss/.venv-3.10.10/lib/python3.10/site-packages/libcst/matchers/_visitors.py", line 409, in _all_positive_matchers_true
    if all_matchers[matcher] is None:
KeyError: ClassDef(name=Name(value='Meta', lpar=DoNotCare(), rpar=DoNotCare(), metadata=DoNotCare()), body=DoNotCare(), bases=DoNotCare(), keywords=DoNotCare(), decorators=DoNotCare(), lpar=DoNotCare(), rpar=DoNotCare(), leading_lines=DoNotCare(), lines_after_decorators=DoNotCare(), whitespace_after_class=DoNotCare(), whitespace_after_name=DoNotCare(), whitespace_before_colon=DoNotCare(), metadata=DoNotCare())

Failed to codemod /Users/smackesey/stm/code/elementl/oss/python_modules/dagster-graphql/dagster_graphql/schema/roots/subscription.py

FWIW this reminds me of a similar issue in pylint, for a long time it would crash using the default parallelism on macOS.

[jmhodges-color]

Seeing this as well! Same machine types

[jmhodges-color]

(And did some digging and, yep, as expected the --jobs 1 setting will, of course, avoids the use of multiprocessing entirely and uses a DummyPool instead of the usual Pool.)

[jmhodges-color]

oh, for clarity: This is still happening on 1.0.0.

[zsol]

I think this will be fixed by #1204.