python
Introduce `typing.STRICTER_STUBS`
Some functions can return different types depending on passed arguments. For example:
open(name, 'rb')returnsio.BufferedReader, whereasopen(name, 'wb')returnsio.BufferedWriter.
The typeshed accurately models this using @typing.overload and typing.Literal. However there is the case that the argument value deciding the return type cannot be determined statically, for example:
def my_open(name: str, write: bool):
with open(name, 'wb' if write else 'rb') as f:
content = f.read()
In this case typeshed currently just claims that open returns typing.IO[Any], so content ends up having the type Any, resulting in a loss of type safety (e.g. content.startswith('hello') will lead to a runtime error if the file was opened in binary mode, but type checkers won't be able to warn you about this because of Any).
While typeshed could theoretically just change the return type to typing.IO[Union[str, bytes]], that would force all existing code bases that currently rely on Any to type check to update their code, which is of course unacceptable.
When starting a new project I however want the strictest type stubs possible. I explicitly do not want standard library functions to return unsafe values like Any (or the a bit less unsafe AnyOf suggested in #566), when the return types can be modeled by a Union.
I therefore propose the introduction of a new variable typing.STRICTER_STUBS: bool, that's only available during type checking.
Which would allow typeshed to do the following:
if typing.STRICTER_STUBS:
AnyOrUnion = typing.Union
else:
AnyOrUnion = typing.Any
Ambiguous return types could then be annotated as e.g. -> typing.IO[AnyOrUnion[str, bytes]].
This would allow users to opt into stricter type stubs, if they so desire, without forcing changes on existing code bases.
CC: @AlexWaygood, @JelleZijlstra, @srittau, @hauntsaninja, @rchen152, @erictraut
P.S. Since I have seen Union return types being dismissed because "the caller needs to use isinstance()", I want to note that this is not true, if the caller wants to trade type safety for performance, they can always just add an explicit Any annotation to circumvent the runtime overhead of isinstance. Union return types force you to either handle potential type errors or explicitly opt out of type safety, which I find strongly preferable to lack of type safety by default.
Interesting idea. How would users opt into the "stricter" stubs? Do you propose that type checkers add a new config setting?
I'm not a fan of reusing the TYPE_CHECKING constant, by the way -- I'd prefer to see a new constant introduced in typing.py. But that's a minor point.
Yes, type checkers would need to introduce a new config setting. I'd propose --stricter-stubs as a uniform command-line flag.
I'm not a fan of reusing the TYPE_CHECKING constant, by the way
Yes, good point. I updated the proposal to typing.STRICTER_STUBS with a boolean value. It would actually only need to be available during type checking, so no change to typing.py should be necessary (but maybe it's still a good idea?).
Here are two other potential solutions to this problem.
-
Formally introduce an
Unknowntype and use it in place ofAnyin these cases. Pyright internally tracks anUnknownas a form ofAnyexcept that in "strict" mode, it will complain about the use ofUnknown. This idea is borrowed from TypeScript, which formally exposes anunknowntype. -
Introduce a
OneOftype an an unsafe union — one where only one subtype needs to be compatible rather than all subtypes. This has been discussed previously but didn't gain enough support to lead to a PEP.
Of these three options, I think the OneOf provides the most utility because it provides much more information for language servers. For example, it allows good completion suggestions to be presented to users. It can also be flagged as unsafe in the strictest modes.
Why do we need a flag? In the OP's example it seems a pretty good idea to change the typechecker behavior to return io.BufferedReader | io.BufferedWriter. (A better example would actually be open(name, 'rb' if binary else 'r'), which could return io.BufferedReader | io.TextIO.)
@gvanrossum The type checker does not know about the return types of open. These are only encoded in the type stubs. In this case the stdlib stubs of typeshed. There are currently many projects that rely on Any return types of typeshed, changing the return type in typeshed would break the type checking for these projects. Hence the flag.
@erictraut thanks for bringing up these alternatives. I think the appeal of the flag is that you can set it and forget it, whereas the introduction of a new type would require Python programmers to learn about the type ... and for what? I personally wouldn't want to deal with some functions returning Union and some functions returning AnyOf (besides I find the latter very confusing semantically because the definition of a union is that it can be any one of its arguments, having both AnyOf and Union is bound to result in confusion). Yes OneOf would allow richer LSP autocompletion, however I am not sure if providing autocompletion is such a good idea if the suggestions might lead to type errors during runtime.
changing the return type in typeshed would break the type checking for these projects
Are you sure? It may be worth checking whether that's really common in real code. mypy-primer should help.
I like the OneOf idea. Type checkers could add a flag that makes the checker treat the type like a regular Union, which would provide the type safety @not-my-profile asks for.
@gvanrossum The type checker does not know about the return types of
open. These are only encoded in the type stubs. In this case the stdlib stubs of typeshed.
I'm well aware. :-)
My actual proposal was actually more complex. If we have stubs containing e.g.
@overload
def foo(x: int) -> list[int]: ...
@overload
def foo(y: str) -> list[str]: ...
and we have a call like this
def f(flag: bool):
a = foo(0 if flag else "")
reveal_type(a)
then the revealed type could be list[int] | list[str]. But this would require the inferred type of 0 if flag else "" to be int | str, whereas ATM (in mypy at least) the inferred type there is object, being the nearest common (named) type in the MROs of int and str.
There are currently many projects that rely on
Anyreturn types of typeshed, changing the return type in typeshed would break the type checking for these projects. Hence the flag.
But in this example the return type is not Any. The call in my example is rejected; however if we help the type checker by forcing a union it will actually do the right thing:
from typing import overload
@overload
def foo(a: int) -> list[int]: ...
@overload
def foo(a: str) -> list[str]: ...
def foo(a): pass
def f(flag: bool):
a: int | str = 0 if flag else ""
reveal_type(a) # int | str
b = foo(a)
reveal_type(b) # list[int] | list[str]
When I tried the OP's example it seems that because the mode expression's type is inferred as str the fallback overload is used, and that's typing.IO[Any]. if I try to help a bit by adding an explicit type, like this:
from typing import Literal
def my_open(name: str, write: bool):
mode: Literal['rb', 'wb'] = 'wb' if write else 'rb'
with open(name, mode) as f:
reveal_type(f) # typing.IO[Any]
I still get the fallback. This seems to be due to some bug (?) in the typeshed stubs -- if I add buffering=0 it gets the type right:
def my_open(name: str, write: bool):
mode: Literal['rb', 'wb'] = 'wb' if write else 'rb'
with open(name, mode, buffering=0) as f:
reveal_type(f) # io.FileIO
(@JelleZijlstra @srittau Do you think the former result is a bug in the stubs? Those overloads have buffering: int without a default.)
I like the
OneOfidea. Type checkers could add a flag that makes the checker treat the type like a regularUnion, which would provide the type safety @not-my-profile asks for.
Exactly what I was thinking. #566 (AnyOf/OneOf) would work very well with a strictness check and we could get rid of those pesky Any return typesthat no one likes.
Are you sure? It may be worth checking whether that's really common in real code. mypy-primer should help.
I am unsure how representative 100 projects can be for the far larger Python ecosystem. Anyway I tried it with https://github.com/python/typeshed/pull/7416 and it interestingly enough resulted in some INTERNAL ERRORs for mypy.
(AnyOf/OneOf) would work very well with a strictness check
When should a function return Union and when should a function return OneOf?
Exactly what I was thinking. #566 (
AnyOf/OneOf) would work very well with a strictness check and we could get rid of those peskyAnyreturn typesthat no one likes.
But that's not what's going on in the OP's example (see my message).
@gvanrossum It's late here and my brain might be a bit mushy. But aren't those orthogonal issues? AnyOf/OneOf can be useful generally in stubs in many situations. And there seems to be a problem with the way type checkers infer a base type instead of a union type in some situations.
Also if I remember correctly, there are some issues with open() when using mypy, since the latest version still has a plug for it, which overrides the stubs for open().
Edit: I will try to give more coherent thoughts tomorrow.
IIRC AnyOf has been proposed before but always met fierce resistance from mypy core devs.
I just wanted to get to the root of the motivation of the issue as presented by the OP, and came up with some interesting (to me) facts.
The plugin for open has (finally) been deleted from the mypy sources (https://github.com/python/mypy/pull/9275) but that was only three weeks ago, so it's probably not yet in the latest release (0.931). The results for my examples are the same on the master branch and with 0.931 though.
Adding a default value to buffering in the overload you link sounds correct to me — good spot!
There's interest in getting mypy to use a meet instead of a join for ternary, e.g. see https://github.com/python/mypy/issues/12056. I don't think anyone is too opposed philosophically.
i.e. I agree that OP's specific complaint would be fixed if mypy inferred Literal["wb", "rb"] + typeshed adds a default value to buffering in that overload.
OP's general complaint still stands. If you tweak the example to take mode: str, you'd get an Any that I suspect OP would not want. AnyOf + flag to treat AnyOf as Union would work well for this case.
My recollection of past discussions of AnyOf isn't fierce resistance as much as inertia, since the benefits for type checking are fairly marginal (although of course, if substantial numbers of users would use an AnyOf = Union flag, such users would get meaningfully more type safety). The strong argument for AnyOf in my mind has been IDE like use cases, so I'd be curious to hear if @erictraut thinks it worth doing.
I'm a big advocate of the benefits of static type checking, but I also recognize that the vast majority of Python developers (99%?) don't use type checking. Most Python developers do use language server features like completion suggestions, so the type information in typeshed stubs benefits them in other ways. I need to continually remind myself to think about the needs of these developers.
I agree that AnyOf has marginal value to static type checking scenarios. It has some value because it allows for optional stricter type checking, but the real value is for the other Python users who would see improved completion suggestions. So yes, I am supportive of adding something like AnyOf.
On Tue, Mar 1, 2022 at 11:21 PM Eric Traut @.***> wrote:
I'm a big advocate of the benefits of static type checking, but I also recognize that the vast majority of Python developers (99%?) don't use type checking. Most Python developers do use language server features like completion suggestions, so the type information in typeshed stubs benefits them in other ways. I need to continually remind myself to think about the needs of these developers.
I agree that AnyOf has marginal value to static type checking scenarios. It has some value because it allows for optional stricter type checking, but the real value is for the other Python users who would see improved completion suggestions. So yes, I am supportive of adding something like AnyOf.
I also agree that AnyOf has marginal value for static type checking. It also sounds like a complex feature that would take a lot of effort to implement, and it would make type checkers that don't provide completion suggestion functionality harder to maintain for not much benefit.
Completion suggestions are an important use case, however. Assuming Eric's estimate above is not way off the mark, relatively few users of code completion use static type checking. It seems sufficient to only support the proposed functionality in stubs (and possibly PEP 561 compliant packages). Most users that will benefit from this feature wouldn't be using AnyOf in their code, after all (since they aren't using static type checking). I think that we could design the feature in a way that static type checkers don't have to add any major new functionality, but language servers would still be able to generate better suggestions.
The earlier STRICTER_STUBS idea might be enough, possibly with a different name. We'd support specifying a different type for language server use cases and static type checking use cases. Maybe language server users would be fine with a union type return type (with no Any items), even if static type checker users would prefer Any (or a union with Any).
For example, Match.group could be defined like this:
if STRICTER_STUBS: def group(self, __group: str | int) -> AnyStr | None: ... else: def group(self, __group: str | int) -> AnyStr | Any: ...
Now language servers don't need to provide completion suggestions for an Any return, and static type checkers still wouldn't require callers to guard against None. A language server could even infer AnyOf[AnyStr, None] as the return type, by using some simple heuristics to merge the two signatures.
I also agree that AnyOf has marginal value for static type checking.
I still wonder why people think that. I believe that this clearly improves type safety in a lot of cases:
def foo() -> AnyOf[str, bytes]: ...
def bar(x: URL) -> None: ...
f = foo()
bar(f) # will fail
f.schema # will fail
with open(...) as f:
f.write(pow(x, y)) # will fail
#566 now has links to over 20 other issues or PRs from typeshed, most of which state the type checking could be improved by AnyOf.
It also sounds like a complex feature that would take a lot of effort to implement,
Fortunately, as a stopgap, AnyOf could be treated like Any. This provides the same (lack of) type safety that the current situation offers, but would allow type checkers and other tooling to use AnyOf to its full benefit.
I agree with @srittau. I think AnyOf would have great utility for typeshed and be very frequently used.
The disadvantage of STRICTER_STUBS, as it was put forward in @not-my-profile's original profile, is that it's an "all or nothing" approach. Either a project opts into all the stricter stubs, everywhere, or they opt into none at all. That might limit adoption, as lots of projects that previously type-checked might find that they now have many errors.
The two ideas are in some ways complementary, however. If we had an AnyOf special form, type checkers might be able to optionally provide an option to treat these unsafe unions as strict unions, giving additional type safety to users who do want that. (@JelleZijlstra already touched on this idea in https://github.com/python/typing/issues/1096#issuecomment-1055791967.)
Ok, I like the idea of AnyOf but I really dislike the name.
Because AnyOf is essentially a type checking implementation detail. Not every type checker will support it, so having a function of an API return AnyOf feels really weird to me since you don't know which type checker (if any) the calling code is using. And I especially dislike that when you see Union and AnyOf for the first time it's unclear when you should use which.
I feel like we don't need a new type for this. It could just be a type checker setting. The code could be:
def get_animal(name) -> Union[Cat, Dog]: ...
Type checkers (and users) can then decide if they want to treat Union as:
Any- "AnyOf" (improving type safety over Any)
- a strict
Union(improving type safety even further)
This would actually reduce type safety for users that don't use strict mode. Many functions already return strict unions where it's necessary to check the return value at runtime. It's especially common to see X | None. Making all of these non-strict by default would be a step backwards.
X | None could be treated as an exception.
Many functions already return strict unions where it's necessary to check the return value at runtime.
This is really the point I don't get. IMO it is not up to an API to decide the level of type safety for the caller. That is solely up to the API user.
I agree that the name of AnyOf isn't ideal, but I do think we need a new special form for this, for the reasons @srittau sets out. Special-casing X | None would be confusing and counterintuitive.
Well I find having both AnyOf and Union to be confusing and counterintuitive. Both semantically as well as the weird idea of encoding type safety into APIs.
I don't see any problem with special casing None, after all null references are the billion-dollar mistake. So IMO None very much deserves special treatment, and we can take advantage of the existing typing.Optional to communicate that to the user.
def get_text() -> str | None: ...
"Hello " + get_text() # always error: Unsupported operand types for + ("str" and "None")
def get_name() -> str | bytes | None: ...
"Hello " + get_name() # always error: Unsupported operand types for + ("str" and "None")
name = get_name()
assert name is not None # narrowing Optional[Union[str, bytes]] down to Union[str, bytes]
You would now get (depending on your type checker settings):
| expression | unions as Any |
unions as "any of" | strict unions |
|---|---|---|---|
name + 1 |
:heavy_check_mark: | ERROR | ERROR |
"Hello " + name |
:heavy_check_mark: | :heavy_check_mark: | ERROR |
To avoid user confusion, I think when using unions as Any or unions as "any of", type checkers should reveal Union[str, bytes, None] as Optional[Union[str, bytes]] (and thus making it obvious that the Optional has to be unwrapped first.
On Wed, Mar 2, 2022 at 11:33 AM Sebastian Rittau @.***> wrote:
I also agree that AnyOf has marginal value for static type checking.
I still wonder why people think that. I believe that this clearly improves type safety in a lot of cases:
def foo() -> AnyOf[str, bytes]: ...def bar(x: URL) -> None: ... f = foo()bar(f) # will failf.schema # will fail with open(...) as f: f.write(pow(x ,y)) # will fail
#566 https://github.com/python/typing/issues/566 now has links to over 20 other issues or PRs from typeshed, most of which state the type checking could be improved by AnyOf.
Based on quick grepping, typeshed has over 40k functions. If this feature could improve 40 functions, that's still only about 0.1% of the entire typeshed. Clearly not all functions are equally useful, so this is not a very good estimate of the benefit, but I'm not convinced that this would have a big impact. In several of the cases using X | Any already gives pretty good type checking coverage, and AnyOf would only be slightly better. There are also thousands of Any types that could already be given precise types using existing type system features. Improving those seems like a better proposition in terms of cost/benefit.
There are a few common functions which are currently problematic, such as open. For them, I'd prefer to have type-safe alternatives that would give full type checking coverage, instead of improvements which make things only incrementally less unsafe. For example, I've previously suggested open_text and open_binary wrappers for open, and similarly we could have integer-only and float-only pow functions. These are easy improvements and could be provided in a small third-party library, for example.
In contrast, implementing AnyOf in mypy would take several months of full-time effort, and it would require somebody who has a strong understanding of type systems and mypy internals. Finding a volunteer would likely be hard, since the apparent benefits don't seem convincing enough. Clearly there would be some benefit, but I don't think that it is proportional to the required effort.
It also sounds like a complex feature that would take a lot of effort to implement,
Fortunately, as a stopgap, AnyOf could be treated like Any. This provides the same (lack of) type safety that the current situation offers, but would allow type checkers and other tooling to use AnyOf to its full benefit.
I expect that this could result in unproductive discussions about whether to annotate something as "str | Any" or "AnyOf[str, None]", etc. Users of some tools would prefer the prior while users of other tools would prefer the latter. Already annotating functions involving unions is tricky, and this could add another dimension of difficulty that would affect all contributions. I don't like the idea of fragmenting the type system into multiple dialects.
On Wed, Mar 2, 2022 at 12:29 PM Sebastian Rittau @.***> wrote:
This would actually reduce type safety for users that don't use strict mode. Many functions already return strict unions where it's necessary to check the return value at runtime. It's especially common to see X | None. Making all of these non-strict by default would be a step backwards.
This is an important point. Any new features should be in addition to what we currently can express to maintain backward compatibility and existing guarantees. Also having stubs use a different syntax or semantics than normal code for common things sounds quite confusing.
Also having stubs use a different syntax or semantics than normal code for common things sounds quite confusing.
Sorry for the confusion, what I proposed in https://github.com/python/typing/issues/1096#issuecomment-1056891902 is meant to apply to all type checking (so regular code as well).
And I do not think that we would loose any expressiveness, quite the opposite instead. APIs could always just return an expressive Union (instead of being forced to return Any like currently for the sake of backwards compatibility) and type checkers could interpret Union however they see fit (with the sole requirement that None checking must always be enforced).
type checkers could interpret Union however they see fit (with the sole requirement that None checking must always be enforced).
Ah sorry, I misunderstood your proposal. I don't think that we can change the meaning of union types, and we must remain backward compatible. The existing union types are used by (hundreds of?) thousands of projects and supported by many tools. I think that the current definition of union types is perfectly fine, but they don't cover all the use cases well (in particular, legacy APIs not designed for type checking).
Thanks @JukkaL that is a very good observation ... that the root cause of the problem are legacy APIs that weren't designed for type checking!
With that in mind, at least my concern could be addressed by introducing a @typing.warning decorator, which could be used to annotate the overloads that return Any, e.g:
from typing import overload, warning, Literal
@overload
def open(file: str, mode: Literal['rb']) -> io.BufferedReader: ...
@overload
def open(file: str, mode: Literal['wb']) -> io.BufferedWriter: ...
@overload
@warning('cannot statically determine return type because `mode` is unknown')
def open(file: str, mode: str) -> Any: ...
This would also allow deprecated functions and functions that should not be used (but are included in typeshed because their absence would confuse users) to be annotated with warnings, so that type checkers can forward these warnings when the functions (or specific overloads) are used.
For example the standard library docs note:
Note that Loggers should NEVER be instantiated directly, but always through the module-level function logging.getLogger(name).
So typeshed could annotate:
class Logger(Filterer):
@warning('should not be used, use `logging.getLogger` instead')
def __init__(self, name: str, level: _Level = ...) -> None: ...
I think it would even make sense for the __getattr__ functions typeshed uses for incomplete stubs:
@warning('you have reached the end of this type stub')
def __getattr__(name: str) -> Any: ...
Because it is very easy to accidentally call such __getattr__ placeholders without noticing it, resulting in a potentially dangerous loss of type safety.
@JukkaL makes a good point about AnyOf being complex and expensive to implement in its fullest form, and I agree that the value for static type checking probably doesn't justify the work involved.
I think there are three ways AnyOf could be interpreted by a type checker:
- Interpreted as
Any. This would be equivalent to the the behavior today. - Interpreted as a "weak union". As Jukka noted, this would be a lot of work to implement.
- Interpreted as a true
Union. This could be used in the strictest type checking modes.
Interpretations 1 and 3 should be "free" to implement since they are already fully supported by mypy and other type checkers. Interpretation 2 would be expensive, but I think it would be fine for type checkers to ignore this mode.
Here's an idea that should be (relatively) cheap to implement. We could leverage PEP 646 to create a typeshed-specific internal class called _WeakUnion (or similar). It could take a variadic TypeVarTuple but also derive from Any. Here's how this might look:
_Ts = TypeVarTuple("_Ts")
class _WeakUnion(Generic[*_Ts], Any): ...
@overload
def int_or_float(x: Literal[True]) -> int: ...
@overload
def int_or_float(x: Literal[False]) -> float: ...
@overload
def int_or_float(x: bool) -> _WeakUnion[int, float]: ...
Type checkers that want to interpret _WeakUnion as a "true union" could do so in strict modes. Likewise, language servers could use the additional information in the _WeakUnion to provide good completion suggestions. Type checkers that don't know about _WeakUnion would continue to treat it as an Any.
Thoughts?
Couple of notes:
- mypy apparently currently doesn't let you subclass
Any: Class cannot subclass "Any" (has type "Any") - pytype apparently currently doesn't support
TypeVarTuple - so type checkers that wanted to support this typeshed-only type would have to hard-code the full name of the type? what happens for third-party stubs when the stubs are moved outside of typeshed when e.g. a package becomes
py.typed?
mypy does support subclassing from Any, but emits an error for it in strict mode. It should work fine once you type ignore that error or turn off the option for it.
However, mypy doesn't support any of PEP 646 at all.