How to handle typechecker specific bugs

[srittau]

In the past, we've worked around typechecker bugs and missing features in typeshed. But as the amount of typecheckers grows and type stubs are used for more that just type checking, I don't think we can work around any bug or missing feature of every typechecker in typeshed, especially if this means regressing our stubs. Not only is this more maintainer work and it gets hard to remember which typechecker has which bug, it also means that users of bug-free typecheckers and other software will still be affected by those bugs.

I think that we should still accommodate very impactful bugs and missing major features (per our "tracking" bugs) for the "big four" mypy, pytype, pyright, and pyre. We also need to work around some bugs due to our tests. But apart from that, we should work with typecheck authors (and there is some overlap here, of course) to come up with a solution.

Regarding stdlib stubs, I believe one option is that typecheckers ship with a customized version of typeshed for major problems. Apart from that, it's always possible for users to # type: ignore a problem with a specific typechecker.

[Akuli]

I like this approach: it's practical and correct enough. Here are a couple examples of how we have done this in the past:

• If the stubs would have to be be plain wrong to work around the type checker bug, and the type checker bug only happens in an obscure corner case, we don't do anything. I remember one case like this, but can't find it now.
• We can change stubs so that they are still correct even with the bug workaround, just written in a way that seems unnecessary if you aren't aware of the bug. For example, we have lots of stubs with "unnecessary" overloads to help mypy resolve TypeVars better.

[hmc-cs-mdrissi]

How does customize solution work for non stdlib stubs? Main stub I'm thinking of is google-protobuf stub. Current version of that stub/latest release does not work well with pyright and leads to many google packages being treated as Any because of __init__.pyi file being there. Dropping that file is blocked by mypy --namespace-package change as last time it was dropped it caused a lot of mypy errors. At moment pyright produces a lot of errors/Anys in strict mode while the stub remains incorrect. I think current keeping init is preference for status quo, but it does mean experience of using those stubs is worse for pyright.

Personal impact is low as I pin to only release where __init__.pyi was dropped. It is something users occasionally report and my guess is it quietly affects a lot of pyright/pylance users because on basic mode it silently leads to a lot of Any (on strict is where you see errors).

[hauntsaninja]

Cases where a correct typechecker is significantly impacted by a workaround for another typechecker are the most clear cut to me — we should obviously favour the correct typechecker. (Within a reasonable timeline) an incorrect typechecker will just have to deal with it, e.g. like mypy reverting LiteralString changes in its vendored copy.

Regarding protobuf specifically, it appears several users were complaining about being broken by a "micro" release in types-protobuf. That's usually unavoidable due to typeshed's versioning scheme, but as it happens protobuf now has a v4 out: https://github.com/python/typeshed/pull/8360 We could pragmatically take advantage of this timing to do the right thing and drop the __init__.py

But back to the original question. For stubs where we've received specific user complaints and typecheck quality isn't really compromised for other typecheckers, I'm still in favour of merging. This includes stuff like Callable[..., Any] in unittest and the unnecessary overloads Akuli mentioned.

And yes, a huge +1 to working with typecheckers to fix things. Of course, with pyright we have the magical Beetlejuice thing going on where all bugs get instantly fixed, but I think things have been getting better recently with mypy as well.