Try block handling assumes all statements could have succeeded

[aiudirog]

trafficstars

Bug Report

When a try block is used to attempt to convert/consolidate the type of a variable, the except block assumes the conversion may or may not have happened. For try blocks with multiple statements, this is the correct behavior for every line but the final one as mypy can't know which of those lines succeeded. However, I would expect mypy to understand that the last/only line can't have executed.

To Reproduce

Minimal example similar to where I encountered the issue:

# file: type_test.py
import sys
from importlib import import_module
from types import ModuleType


def ensure_module(module: str | ModuleType) -> ModuleType:
    if isinstance(module, str):
        try:
            module = import_module(module)
        except ImportError:
            sys.path.append('/path/to/extra/modules')
            # mypy thinks module could be either a string or ModuleType
            module = import_module(module)
    return module

Expected Behavior

mypy shouldn't have any errors:

$ python -m mypy type_test.py
Success: no issues found in 1 source file

Actual Behavior

mypy finds an error in the except block:

$ python -m mypy type_test.py
type_test.py:14: error: Argument 1 to "import_module" has incompatible type "Union[str, Module]"; expected "str"
Found 1 error in 1 file (checked 1 source file)

Your Environment

• Mypy version used: mypy 0.961 (compiled: yes)
• Mypy command-line flags: N/A
• Mypy configuration options from mypy.ini (and other config files): N/A
• Python version used: 3.10.5
• Operating system and version: Arch Linux (5.18.11-zen1-1-zen)

[NeilGirdhar]

Just a friendly suggestion, but it's probably clearer to use a different variable name when the type is different. Consider renaming the return value to return_module.

[aiudirog]

Of course, that's basically what I did in the actual code, but MyPy should still be able to handle this properly in general.

My issue specifically is that the type narrowing is lost in the except block when it's impossible for it to have changed types.

Here's a much simpler example focused on this:

# type_test_str_int.py
from __future__ import annotations


def ensure_int(num: str | int) -> int:
    if isinstance(num, str):
        try:
            reveal_type(num)  # str
            num = int(num)
        except ValueError:
            reveal_type(num)  # str | int ??
            num = int(num.replace('pfx', ''))
    return num

$ python -m mypy type_test_str_int.py 
type_test_str_int.py:7: note: Revealed type is "builtins.str"
type_test_str_int.py:10: note: Revealed type is "Union[builtins.str, builtins.int]"
type_test_str_int.py:11: error: Item "int" of "Union[str, int]" has no attribute "replace"
Found 1 error in 1 file (checked 1 source file)

As you said, using a new variable avoids this problem, but that doesn't mean it couldn't be handled better. Here's an interesting follow up example that uses a new variable:

# type_test_str_int2.py
from __future__ import annotations


def ensure_int(num: str | int) -> int:
    if isinstance(num, str):
        try:
            actual_num = int(num)
        except ValueError:
            reveal_type(actual_num)
            actual_num = int(num.replace('pfx', ''))
    return actual_num

$python -m mypy type_test_str_int2.py
type_test_str_int2.py:9: note: Revealed type is "builtins.int"
Success: no issues found in 1 source file

This code should really be throwing multiple errors at me, but MyPy isn't really focused on undefined variables, which is fine. However, it probably shouldn't be defined inside the except block since that line couldn't have succeeded. When dealing with the except block(s), ignoring the type changes from the last/only line of the body should avoid this and properly report that it is undefined, as well as solving the original issue so a second variable isn't required.

I've also noticed a similar issue with closure functions, especially decorators with optional arguments that follow this pattern:

# type_test_deco.py 
from __future__ import annotations

from functools import partial
from typing import Callable, Any, TypeVar,  overload

F = TypeVar('F', bound=Callable[..., Any])


@overload
def decorator(func: F, *args: Any, **kwargs) -> F: ...


@overload
def decorator(func: None = None, *args: Any, **kwargs) -> Callable[[F], F]: ...


def decorator(
    func: F | None = None,
    *args: Any,
    **kwargs: Any,
) -> F | Callable[[F], F]:
    if func is None:
        return partial(decorator, *args, **kwargs)

    reveal_type(func)  # `F`

    # Use args and kwargs here

    def _wrapper(*a: Any, **kw: Any) -> Any:
        reveal_type(func)  # `F | None` ??
        return func(*a, **kw)

    return _wrapper

$ python -m mypy type_test_deco.py 
type_test_deco.py:25: note: Revealed type is "F`-1"
type_test_deco.py:30: note: Revealed type is "Union[F`-1, None]"
type_test_deco.py:31: error: "None" not callable
Found 1 error in 1 file (checked 1 source file)

In this example, an explicit typed assignment/cast would solve the problem, but that really shouldn't be necessary due to type narrowing. It's impossible for _wrapper() to even be defined if func is None, so it shouldn't have inherited the original signature but instead the narrowed one.