Support narrowing using `TypeGuard`

[dhruvmanila]

Add support for narrowing on TypeGuard and TypeIs:

• [x] TypeIs https://github.com/astral-sh/ruff/labels/help%20wanted
• [ ] TypeGuard

This will in turn support builtins.callable:

Add support for narrowing on builtins.callable, requires TypeIs support.

builtins.pyi:

def callable(obj: object, /) -> TypeIs[Callable[..., object]]: ...

Otherwise, it causes a few false positives, like:

def _(a: Any | None) -> None:
    # /private/tmp/mypy_primer/projects/bidict/bidict/_iter.py:50:34: Object of type `None` is not callable
    if callable(a):
        a()  # red-knot: Object of type `None` is not callable [lint:call-non-callable]
        reveal_type(a)  # revealed_type: Any | None

[AlexWaygood]

Hmm, won't this fall out naturally from TypeIs support? I'm not sure we'll need to add any special handling for builtins.callable() once we understand TypeIs.

It doesn't look like we have a dedicated issue for TypeGuard and TypeIs support right now, but we should maybe make one. It was one of the items listed in https://github.com/astral-sh/ruff/issues/13694, but that issue is now closed

[dhruvmanila]

Hmm, won't this fall out naturally from TypeIs support? I'm not sure we'll need to add any special handling for builtins.callable() once we understand TypeIs.

Yeah, you're right. There's https://github.com/astral-sh/ruff/pull/16314 which is in draft.

[carljm]

TypeIs at least should not be too difficult to build on top of our existing narrowing support; tagging "help wanted" for that.

TypeGuard may be somewhat more challenging, since it can fully override the previous type (like a cast), meaning we will have to extend our narrowing to support forms of narrowing that don't take the form of intersections. (Probably by changing narrowing constraints from a type to an enum where one variant is "a type to intersect with", another variant is "a typeguard type to override with", etc.)

[InSyncWithFoo]

While working on astral-sh/ruff#16314, I encountered three major problems, all listed at this comment. I'm willing to resume my work if someone could give a few hints on how to handle them, especially the second.

As a note to myself, here's a small edge case example to add to the tests:

def is_int(v: int | str) -> TypeIs[int]: ...

class C:
    a: int | str
    
    def __init__(self, a: int | str):
        self.a = a
        self.a_is_int = is_int(a)

c = C('')

reveal_type(c.a)         # int | str
reveal_type(c.a_is_int)  # TypeIs[int]
                         # Or `TypeIs[a, int]`? `TypeIs[c.a, int]`? `TypeIs[C.a, int]`?

if c.a_is_int:
    reveal_type(c.a)     # ??
else:
    reveal_type(c.a)     # ??

[carljm]

Ah, I wasn't even aware of that PR; I don't get notifications on draft PRs unless pinged. It looks like point 3 there is the same one I mentioned above: TypeGuard will require some significant additions to the current implementation of narrowing in red-knot. The differences in implementation are major enough that I would definitely suggest tackling them in separate PRs. I will try to find some time soon to review that PR and see if I can understand the other two points mentioned.

[carljm]

As a note to myself, here's a small edge case example to add to the tests

FWIW, neither pyright nor mypy support this case, and I don't think we need to either. It isn't clear in the spec, but I don't see any evidence in the spec or in the other typechecker implementations to suggest that type guards should have any effect on expressions outside the scope where the type guard is called.

[InSyncWithFoo]

@carljm How about this, then?

def f(a: int | str):
	class C:
		def __init__(self):
			self.a_is_int = is_int(a)

	reveal_type(C().a_is_int)  # TypeIs[a, int]

	if C().a_is_int:
		reveal_type(a)  # int?

In case it isn't clear, this is just a contrived example to demonstrate the second problem.

Neither Mypy nor Pyright supports this pattern, for presumably a very good reason. Something noteworthy is that the two type checkers differ in how they infer .a_is_int: Mypy considers it bool, while Pyright reveals TypeIs[int] (but the attribute isn't usable as a narrowing condition).

[carljm]

@InSyncWithFoo I don't think that needs to be supported either.

I don't think we should support any pattern that would require carrying along with the TypeIs type the knowledge of what symbol or expression it was applied to (that is, any pattern where the call to a TypeIs function is separated from the actual conditional check). I think our implementation should simply look for calls that return TypeIs in a narrowing constraint, and directly check in the AST of the call (in the narrowing constraint resolution) to see what name (or attribute/subscript expression, once that PR lands) is narrowed by the call.

I realize that pyright does support some limited forms of separating the call from the check, within the same scope, but mypy does not, and there are no examples in the PEP or in the conformance test suite to suggest that this should be supported.

[sharkdp]

@dhruvmanila observed a false positive related to TypeGuard after some changes related to **kwargs. We should see that false positive disappearing when TypeGuard is implemented properly.

https://github.com/mit-ll-responsible-ai/hydra-zen/blob/8f8d30622b85983e0f54004ea945858a72f32469/src/hydra_zen/structured_configs/_implementations.py#L2952. The parse_strict_dataclass_option function return a TypeGuard (https://github.com/mit-ll-responsible-ai/hydra-zen/blob/8f8d30622b85983e0f54004ea945858a72f32469/src/hydra_zen/structured_configs/_utils.py#L380-L386) to narrow the generic mapping into a concrete TypedDict.

[jpgoldberg]

With respect to the M1 milestone (which I assume is after "stable") I would like to point out that TypeIs was introduced in Python 3.13, and 3.12's scheduled end-of-life isn't until late 2028. So like it or not TypeGuard will be a thing for nearly another three years.

[carljm]

M1 is pre-stable

[Avasam]

I would like to point out that TypeIs was introduced in Python 3.13, and 3.12's scheduled end-of-life isn't until late 2028. So like it or not TypeGuard will be a thing for nearly another three years.

You make it sound like TypeGuard is deprecated or going away, but TypeGuard and TypeIs are both actively relevant and do something different (especially when it comes to assignability and the False branch): https://typing.python.org/en/latest/guides/type_narrowing.html#typeis-and-typeguard