Improve annotations for various set methods

[AlexWaygood]

• Fixes some regressions introduced by https://github.com/python/typeshed/pull/7161 (see https://github.com/python/typeshed/issues/9004#issuecomment-1295312743).
• Fixes #9004
• Fixes #7121
• Fixes https://github.com/python/mypy/issues/13192 (cc. @finite-state-machine).
  • I wasn't initially sure that set[str | None] - set[None] came up often enough to be worth special-casing, but the activity on that issue has persuaded me otherwise.

It's a bit unfortunate how many type: ignores we have to introduce here, but I can't see a good way round them. I've added some test cases to make sure everything's working correctly.

[github-actions[bot]]

Diff from mypy_primer, showing the effect of this PR on open source code:

pydantic (https://github.com/samuelcolvin/pydantic)
+ pydantic/v1/main.py:901: error: Set comprehension has incompatible type Set[int | str]; expected Set[None]  [misc]
+ pydantic/main.py:1257: error: Argument 1 to "set" has incompatible type "AbstractSet[int] | AbstractSet[str] | Mapping[int, Any] | Mapping[str, Any]"; expected "Iterable[str | None]"  [arg-type]
+ pydantic/deprecated/copy_internals.py:222: error: Set comprehension has incompatible type Set[int | str]; expected Set[str | None]  [misc]

werkzeug (https://github.com/pallets/werkzeug)
+ src/werkzeug/datastructures/structures.pyi:197: error: Argument 1 of "discard" is incompatible with supertype "set"; supertype defines the argument type as "Optional[str]"  [override]
+ src/werkzeug/datastructures/structures.pyi:197: note: This violates the Liskov substitution principle
+ src/werkzeug/datastructures/structures.pyi:197: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
+ src/werkzeug/datastructures/structures.pyi:197: error: Argument 1 of "discard" is incompatible with supertype "MutableSet"; supertype defines the argument type as "Optional[str]"  [override]

jax (https://github.com/google/jax)
+ jax/_src/lax/parallel.py:521: error: Need type annotation for "subs_batch" (hint: "subs_batch: Set[<type>] = ...")  [var-annotated]

ibis (https://github.com/ibis-project/ibis)
- ibis/expr/tests/test_format.py:20: error: Argument 1 to <set> has incompatible type "type[Projection]"; expected "type[Relation] | None"  [arg-type]
+ ibis/expr/tests/test_format.py:20: error: Argument 1 to <set> has incompatible type "type[Projection]"; expected "None"  [arg-type]

[AlexWaygood]

Primer analysis:

• ibis is just an error message changing slightly

• werkzeug would just need to change their annotations on their subclass to match our new annotation for discard()

• The three pydantic hits are all new false positives caused by making set.__sub__ stricter.

  (You could argue that the third one is, strictly speaking, a true positive, in that it's working the way we "want" -- mypy is warning that there's a possibility AbstractSet[int] might be being subtracted from AbstractSet[str], which would be a no-op. But it doesn't seem like a "helpful" true positive. You'd have to do some more work to narrow the type before doing the subtraction, but it wouldn't really make your code any more type-safe.)

  These seem pretty unfortunate to me, and make me think that the better way of fixing https://github.com/python/typeshed/issues/9004 is to make it so that __sub__, __isub__, difference and difference_update all accept AbstractSet[object] or Iterable[Any] rather than AbstractSet[_T | None]. @Akuli, thoughts on that?

  • https://github.com/pydantic/pydantic/blob/832225b90672c68e2d4067bd0ffecf834d62b48b/pydantic/v1/main.py#L901
  • https://github.com/pydantic/pydantic/blob/832225b90672c68e2d4067bd0ffecf834d62b48b/pydantic/main.py#L1257
  • https://github.com/pydantic/pydantic/blob/832225b90672c68e2d4067bd0ffecf834d62b48b/pydantic/deprecated/copy_internals.py#L222

• The new jax hit looks like a false positive, probably due to the fact that set.__sub__ is now an overloaded method. https://github.com/google/jax/blob/9352a00dab3c465a4e2c651cc6a0fa91e50ae7a2/jax/_src/lax/parallel.py#L512

[github-actions[bot]]

Diff from mypy_primer, showing the effect of this PR on open source code:

pydantic (https://github.com/samuelcolvin/pydantic)
+ pydantic/v1/main.py:901: error: Set comprehension has incompatible type Set[int | str]; expected Set[None]  [misc]
+ pydantic/main.py:1257: error: Argument 1 to "set" has incompatible type "AbstractSet[int] | AbstractSet[str] | Mapping[int, Any] | Mapping[str, Any]"; expected "Iterable[str | None]"  [arg-type]
+ pydantic/deprecated/copy_internals.py:222: error: Set comprehension has incompatible type Set[int | str]; expected Set[str | None]  [misc]

werkzeug (https://github.com/pallets/werkzeug)
+ src/werkzeug/datastructures/structures.pyi:197: error: Argument 1 of "discard" is incompatible with supertype "set"; supertype defines the argument type as "Optional[str]"  [override]
+ src/werkzeug/datastructures/structures.pyi:197: note: This violates the Liskov substitution principle
+ src/werkzeug/datastructures/structures.pyi:197: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
+ src/werkzeug/datastructures/structures.pyi:197: error: Argument 1 of "discard" is incompatible with supertype "MutableSet"; supertype defines the argument type as "Optional[str]"  [override]

jax (https://github.com/google/jax)
+ jax/_src/lax/parallel.py:521: error: Need type annotation for "subs_batch" (hint: "subs_batch: Set[<type>] = ...")  [var-annotated]

ibis (https://github.com/ibis-project/ibis)
- ibis/expr/tests/test_format.py:20: error: Argument 1 to <set> has incompatible type "type[Projection]"; expected "type[Relation] | None"  [arg-type]
+ ibis/expr/tests/test_format.py:20: error: Argument 1 to <set> has incompatible type "type[Projection]"; expected "None"  [arg-type]

[srittau]

This has conflicts now.

[AlexWaygood]

This has conflicts now.

I fixed the merge conflicts, but it's still not clear to me whether we want to fix the inconsistency between the different methods by making some annotations stricter (as this PR does), which seems to lead to false positives when running type checkers on user code. I think a better solution might be to make the methods consistent by making some annotations looser instead -- what do you think?

[github-actions[bot]]

Diff from mypy_primer, showing the effect of this PR on open source code:

pydantic (https://github.com/samuelcolvin/pydantic)
+ pydantic/v1/main.py:901: error: Set comprehension has incompatible type Set[int | str]; expected Set[None]  [misc]
+ pydantic/main.py:1268: error: Argument 1 to "set" has incompatible type "AbstractSet[int] | AbstractSet[str] | Mapping[int, Any] | Mapping[str, Any]"; expected "Iterable[str | None]"  [arg-type]
+ pydantic/deprecated/copy_internals.py:222: error: Set comprehension has incompatible type Set[int | str]; expected Set[str | None]  [misc]

werkzeug (https://github.com/pallets/werkzeug)
+ src/werkzeug/datastructures/structures.pyi:197: error: Argument 1 of "discard" is incompatible with supertype "set"; supertype defines the argument type as "Optional[str]"  [override]
+ src/werkzeug/datastructures/structures.pyi:197: note: This violates the Liskov substitution principle
+ src/werkzeug/datastructures/structures.pyi:197: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
+ src/werkzeug/datastructures/structures.pyi:197: error: Argument 1 of "discard" is incompatible with supertype "MutableSet"; supertype defines the argument type as "Optional[str]"  [override]

jax (https://github.com/google/jax)
+ jax/_src/lax/parallel.py:550: error: Need type annotation for "subs_batch" (hint: "subs_batch: Set[<type>] = ...")  [var-annotated]