1.16 changed behavior of redefined variable to None

[eli-schwartz]

https://github.com/mesonbuild/meson/blob/master/mesonbuild/cmake/fileapi.py

This file started triggering:

mesonbuild/cmake/fileapi.py:81:20: error: "None" has no attribute "__iter__" (not iterable)  [attr-defined]
mesonbuild/cmake/fileapi.py:82:20: error: Value of type "None" is not indexable  [index]
mesonbuild/cmake/fileapi.py:84:36: error: Value of type "None" is not indexable  [index]

due to reuse of for i in ......:

Reduction:

import typing as T
from pathlib import Path

for i in Path().iterdir():
    T.reveal_type(i)
    break

data = T.cast('T.Dict[str, T.Any]', {'foo': [{'kind': 1, 'b': 'two'}]})
for i in data['foo']:
    T.reveal_type(i)
    assert isinstance(i, dict)
    T.reveal_type(i)
    assert 'kind' in i

With mypy 1.16.0

$ mypy --no-strict-optional --warn-unreachable fileapi.py 
fileapi.py:5: note: Revealed type is "pathlib.Path"
fileapi.py:10: note: Revealed type is "Any"
fileapi.py:12: note: Revealed type is "None"
fileapi.py:13: error: "None" has no attribute "__iter__" (not iterable)  [attr-defined]

It starts off as Path, gets redefined as Any (which works?), but then asserting the Any is a dict results in it migrating to None. With mypy 1.15, it never redefined to Any at all, so it became unreachable:

$ pip install mypy==1.15.* 
$ mypy --no-strict-optional --warn-unreachable fileapi.py 
fileapi.py:5: note: Revealed type is "pathlib.Path"
fileapi.py:10: note: Revealed type is "pathlib.Path"
fileapi.py:11: error: Subclass of "Path" and "dict[Any, Any]" cannot exist: would have incompatible method signatures  [unreachable]
fileapi.py:12: error: Statement is unreachable  [unreachable]

I can't really say either behavior is useful. I want variables to be block-scoped and reset their expected value. But at least the 1.15 behavior was comprehensible to me. The new behavior only works if I delete the assert, which seems terribly counterproductive.

[brianschubert]

Bisects to b50f3a1a44038b5f6304f77263f6e08c157f9aa8 (#18538), cc @ilevkivskyi

Without --no-strict-optional, i is narrowed to Never:

$ mypy --version
mypy 1.16.0+dev.b50f3a1a44038b5f6304f77263f6e08c157f9aa8 (compiled: no)

$ mypy --no-strict-optional fileapi.py
fileapi.py:5: note: Revealed type is "pathlib.Path"
fileapi.py:10: note: Revealed type is "Any"
fileapi.py:12: note: Revealed type is "None"
fileapi.py:13: error: "None" has no attribute "__iter__" (not iterable)  [attr-defined]
Found 1 error in 1 file (checked 1 source file)

$ mypy --no-strict-optional fileapi.py
fileapi.py.:5: note: Revealed type is "pathlib.Path"
fileapi.py:10: note: Revealed type is "Any"
fileapi.py:12: note: Revealed type is "Never"
fileapi.py:13: error: "Never" has no attribute "__iter__" (not iterable)  [attr-defined]
Found 1 error in 1 file (checked 1 source file)

After #18972, the behavior improves without --no-strict-optional

$ mypy --version
mypy 1.16.0+dev.c724a6a806655f94d0c705a7121e3d671eced96d (compiled: no)

$ mypy fileapi.py                        
fileapi.py:5: note: Revealed type is "pathlib.Path"
fileapi.py:10: note: Revealed type is "Any"
fileapi.py:12: note: Revealed type is "builtins.dict[Any, Any]"

$ mypy --no-strict-optional fileapi.py
fileapi.py:5: note: Revealed type is "pathlib.Path"
fileapi.py:10: note: Revealed type is "Any"
fileapi.py:12: note: Revealed type is "None"
fileapi.py:13: error: "None" has no attribute "__iter__" (not iterable)  [attr-defined]
Found 1 error in 1 file (checked 1 source file)

[hauntsaninja]

I guess this branch https://github.com/python/mypy/pull/18972/files#diff-42eda644ce5003e33ea0766d7601fcfb4934ad106429abfe115954a1fccddd07R6301 could also check for no strict optional and NoneType

@eli-schwartz your best unblock might be trying out --allow-redefinition-new in 1.16

[eli-schwartz]

I'm not in an urgent hurry, we are still tangled up in python 3.7 compat :) but will probably jump straight to >=3.10 once November rolls around. Just locally exploring in the interest of being better prepared for November, actually.

[ilevkivskyi]

Yeah, the fact that old behavior worked "better" for variable redefinition is a pure coincidence. If you really want variable redefinition semantics you can try either --allow-redefinition (older but more restricted), or --allow-redefinition-new (newer/less tested but more flexible/intuitive). Also --no-strict-optional in 2025? :-) Seriously however I strongly recommend migrating away from this flag (even if it is hard), many code paths in mypy are not tested with this flag anymore, as it is considered more of a legacy thing.

@hauntsaninja

could also check for no strict optional and NoneType

Maybe, but I am worried this may trigger some weird behavior with "genuine" None, i.e. not one inferred as a "bottom" type.

[eli-schwartz]

Every once in a while I do some work on on fixing implicit None issues. It's definitely something I want to fix, just rather low-priority for me. :(

(Quite some time ago the original efforts to add type checking made the decision that it would be easier to get things up and running by ignoring such issues "for now". Ha.)

[hauntsaninja]

Given allowing redefinition is the "right" fix and my proposed change is a little worrisome (as per Ivan's comment), I think this is a won't fix