Mypy error of type "assignment" only discovered during validation run

[obi1kenobi]

Encountered a mypy error that is unexpectedly not raised in the minimal mypy run, and is not found and suppressed by the config generated after the strictest mypy run. This should not be possible.

Here is the error output:

Validation failed due to unexpected error(s):
[MypyError(file_path='arviz/data/io_pyjags.py', line_number=28, error_code='assignment', message='error: Incompatible default for argument "save_warmup" (default has type "None", argument has type "bool")')]
Traceback (most recent call last):
  File ".../arviz/venv/bin/typing_copilot", line 8, in <module>
    sys.exit(cli())
  File ".../arviz/venv/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File ".../arviz/venv/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File ".../arviz/venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ".../arviz/venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ".../arviz/venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File ".../arviz/venv/lib/python3.8/site-packages/typing_copilot/cli.py", line 254, in init
    unused_ignore_errors = _get_unused_ignore_errors_from_validation_run(final_config)
  File ".../arviz/venv/lib/python3.8/site-packages/typing_copilot/cli.py", line 136, in _get_unused_ignore_errors_from_validation_run
    raise AssertionError(
AssertionError: Validation failed: mypy reported 1 unexpected error(s). Please submit the produced logs so we can update typing-copilot to fix this issue. Apologies for the inconvenience, and thank you for supporting typing-copilot.

To reproduce:

• clone the arviz project repo and install its dev dependencies: https://github.com/arviz-devs/arviz
• ensure that arviz itself is installed in the virtualenv you are using
• use the commit corresponding to wherever this PR merged: https://github.com/arviz-devs/arviz/pull/1498
• undo the change applied by this portion of the PR: https://github.com/arviz-devs/arviz/pull/1498/commits/64913759a2bd26ca40f0aedfdd18e9f847fbb2f1

It is not yet clear what the underlying reason is. It might be due to unexpected behavior in mypy that we'll have to work around, or it might be due to a missing suppression in the strictest run analysis code.

[obi1kenobi]

After more investigation, it appears that this is due to an unfortunate behavior in mypy, where:

• the error is not caught in the laxest mypy configuration, however
• it is a real error, and is inappropriate to suppress.

In a pinch, it could be suppressed by using a disable_error_code = assignment configuration setting (disable_error_code = assignment,union-attr as an example of multiple error codes), but that setting is only supported at top level and would disable the check for the entire project. That is generally undesirable since it will allow buggy code to be checked elsewhere into the project, and typing_copilot should not apply it automatically.

Instead, we'll need to improve the error message text here, explaining the situation to the user and directing them toward the appropriate next steps.