Preventing code quality regressions

[srittau]

For some time now we've used pyright's stricter configuration to prevent code quality regressions. Unless a stubs package is mentioned in the config file, it's expected to have annotations for all its fields. While this has served us well for a while, it has become unsuitable since the introduction of _typeshed.Incomplete:

• pyright doesn't understand that a (partial) Incomplete annotation is equivalent (for our purposes) to an unannotated item.
• It's tedious to maintain the pyright stricter configuration in addition to the metadata in the stubs directory.
• It's easy to cheat the regression check by using Incomplete.

To improve the situation, I propose to move to a custom solution:

1. Add an incomplete marker to METADATA.toml (defaulting to false).
2. Add a custom script to CI that checks that a stub package marked as complete can't have any incomplete (unannotated or using Incomplete) fields. Alternatively, we could add a disabled-by-default check to flake8-pyi and use that instead.
3. At a later date, we could also add a PR CI script that add a PR comment warning a user if the amount of incomplete increases due to a PR.

(See also https://alexwaygood.github.io/typeshed-stats/.)

[Daverball]

I think it might still be nice to have some support for stricter type checker configs on a per distribution basis on the CI side, but I agree that the current solution for detecting completeness is insufficient.

It has kind of bothered me for a while now, that there's no way to e.g. enforce mypy --strict for a third party stub, even if you use those flags locally for initial development, regressions become inevitable since the stricter checks will not be enforced by the CI.