MatchIfTrue lambdas used in decorator matchers in VisitorBasedCodemodCommand hit pickle errors on large codebase

[jmhodges-color]

In a subclass of VisitorBasedCodemodCommand, I have code that creates a @matchers.visit with a MatchIfTrue lambda that throws a lambda pickling error when the codemod is run on a sufficiently large codebase. Running it on a file at a time is fine, because the multiprocessing work doesn't kick in.

Probably it deserves a call out in the docs of MatchIfTrue to use named functions instead of lambdas. I'm not sure what other solutions are available.

The command in question:

def func_def_has_request_param_matcher() -> m.FunctionDef:
    return m.FunctionDef(
        params=m.Parameters(
            params=m.MatchIfTrue(
                lambda params: any(p.name.value == "request" for p in params)
            ),
        )
    )

class ReproCommand(VisitorBasedCodemodCommand):
    DESCRIPTION = (
        "Do cool thing."
    )

    @staticmethod
    def add_args(_arg_parser: argparse.ArgumentParser) -> None:
        return

    @m.visit(func_def_has_request_param_matcher())
    def _mark_func_def_includes_request_param(self, node: cst.FunctionDef) -> None:
        self.context.scratch["cool-scratch"] = node

The error in question

$ python -m libcst.tool codemod commands.repro.ReproCommand src Calculating full-repo metadata... Executing codemod... Traceback (most recent call last): File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Users/jeff.hodges/src/github.com/color/color/local/virtualenv3/lib/python3.8/site-packages/libcst/tool.py", line 839, in main(os.environ.get("LIBCST_TOOL_COMMAND_NAME", "libcst.tool"), sys.argv[1:]) File "/Users/jeff.hodges/src/github.com/color/color/local/virtualenv3/lib/python3.8/site-packages/libcst/tool.py", line 834, in main return lookup.get(args.action or None, _invalid_command)(proc_name, command_args) File "/Users/jeff.hodges/src/github.com/color/color/local/virtualenv3/lib/python3.8/site-packages/libcst/tool.py", line 572, in _codemod_impl result = parallel_exec_transform_with_prettyprint( File "/Users/jeff.hodges/src/github.com/color/color/local/virtualenv3/lib/python3.8/site-packages/libcst/codemod/_cli.py", line 638, in parallel_exec_transform_with_prettyprint for result in p.imap_unordered( File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/multiprocessing/pool.py", line 448, in return (item for chunk in result for item in chunk) File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/multiprocessing/pool.py", line 868, in next raise value File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/multiprocessing/pool.py", line 537, in _handle_tasks put(task) File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps cls(buf, protocol).dump(obj) AttributeError: Can't pickle local object 'func_def_has_request_param_matcher..'

Inlining the matcher creation gets a slightly different error

$ python -m libcst.tool codemod commands.repro.ReproCommand src Calculating full-repo metadata... Executing codemod... Traceback (most recent call last): File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Users/jeff.hodges/src/github.com/color/color/local/virtualenv3/lib/python3.8/site-packages/libcst/tool.py", line 839, in main(os.environ.get("LIBCST_TOOL_COMMAND_NAME", "libcst.tool"), sys.argv[1:]) File "/Users/jeff.hodges/src/github.com/color/color/local/virtualenv3/lib/python3.8/site-packages/libcst/tool.py", line 834, in main return lookup.get(args.action or None, _invalid_command)(proc_name, command_args) File "/Users/jeff.hodges/src/github.com/color/color/local/virtualenv3/lib/python3.8/site-packages/libcst/tool.py", line 572, in _codemod_impl result = parallel_exec_transform_with_prettyprint( File "/Users/jeff.hodges/src/github.com/color/color/local/virtualenv3/lib/python3.8/site-packages/libcst/codemod/_cli.py", line 638, in parallel_exec_transform_with_prettyprint for result in p.imap_unordered( File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/multiprocessing/pool.py", line 448, in return (item for chunk in result for item in chunk) File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/multiprocessing/pool.py", line 868, in next raise value File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/multiprocessing/pool.py", line 537, in _handle_tasks put(task) File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps cls(buf, protocol).dump(obj) _pickle.PicklingError: Can't pickle <function ReproCommand. at 0x10205b1f0>: attribute lookup ReproCommand. on projects.codemods.pycodemods.commands.repro failed

(Tangentially, if there's a better way to do this "match if any Node in this Sequence matches this pattern", I'm down to be told what that is)