mypy icon indicating copy to clipboard operation
mypy copied to clipboard
verified publisher icon python

Sketch implementation of "optional-non-truthy" check for `if x` with `x: None | str`

Open huonw opened this issue 1 year ago • 2 comments

Fixed #17892

This is a sketch of an implementation of a check that catches bugs like

def bug(x: None | str):
    if x:
        print(f"the str value is {x}") 
    else:
        print("the value is None") # oops! x could also be `""`

This is heavily inspired by the checks done in check_for_truthy_type for truthy-iterable, truthy-bool etc.

This isn't complete, but is designed to be a proof of concept. If I have agreement on the direction, I'll at least fix these:

  • [ ] an actual error message (currently just FIXME)
  • [ ] maybe put check_for_truthy_type and check_for_optional_non_truthy_type behind a single check_for_type_truthiness call, instead of having two calls sites that call both
  • [ ] Fix the various failures (doc formatting etc)

huonw avatar Oct 07 '24 19:10 huonw

Diff from mypy_primer, showing the effect of this PR on open source code:

check-jsonschema (https://github.com/python-jsonschema/check-jsonschema)
+ src/check_jsonschema/cli/parse_result.py:49: error: Generator has incompatible item type "int"; expected "bool"  [misc]

sphinx (https://github.com/sphinx-doc/sphinx)
+ sphinx/util/http_date.py: note: In function "rfc1123_to_epoch":
+ sphinx/util/http_date.py:47:16: error: Unsupported operand types for - ("float" and "int")  [operator]

koda-validate (https://github.com/keithasaurus/koda-validate)
+ koda_validate/serialization/json_schema.py:95: error: Unsupported operand types for + ("list[Predicate[str]]" and "list[PredicateAsync[str]]")  [operator]
+ koda_validate/serialization/json_schema.py:105: error: Unsupported operand types for + ("list[Predicate[bytes]]" and "list[PredicateAsync[bytes]]")  [operator]
+ koda_validate/serialization/json_schema.py:115: error: Unsupported operand types for + ("list[Predicate[int]]" and "list[PredicateAsync[int]]")  [operator]
+ koda_validate/serialization/json_schema.py:125: error: Unsupported operand types for + ("list[Predicate[Decimal]]" and "list[PredicateAsync[Decimal]]")  [operator]
+ koda_validate/serialization/json_schema.py:134: error: Unsupported operand types for + ("list[Predicate[float]]" and "list[PredicateAsync[float]]")  [operator]
+ koda_validate/serialization/json_schema.py:143: error: Unsupported operand types for + ("list[Predicate[date]]" and "list[PredicateAsync[date]]")  [operator]
+ koda_validate/serialization/json_schema.py:152: error: Unsupported operand types for + ("list[Predicate[datetime]]" and "list[PredicateAsync[datetime]]")  [operator]
+ koda_validate/serialization/json_schema.py:169: error: Unsupported operand types for + ("list[Predicate[bool]]" and "list[PredicateAsync[bool]]")  [operator]
+ koda_validate/serialization/json_schema.py:178: error: Unsupported operand types for + ("list[Predicate[UUID]]" and "list[PredicateAsync[UUID]]")  [operator]
+ koda_validate/serialization/json_schema.py:294: error: Unsupported operand types for + ("list[Predicate[dict[Any, Any]]]" and "list[PredicateAsync[dict[Any, Any]]]")  [operator]

streamlit (https://github.com/streamlit/streamlit)
+ lib/streamlit/runtime/pages_manager.py: note: In member "get_initial_active_script" of class "PagesStrategyV1":
+ lib/streamlit/runtime/pages_manager.py:98:21: error: Argument 1 to "filter" has incompatible type "Callable[[Any], Optional[bool]]"; expected "Callable[[PageInfo], TypeGuard[Optional[PageInfo]]]"  [arg-type]
+ lib/streamlit/runtime/pages_manager.py: note: In member "get_page_script" of class "PagesStrategyV2":
+ lib/streamlit/runtime/pages_manager.py:180:21: error: Argument 1 to "filter" has incompatible type "Callable[[Any], Optional[bool]]"; expected "Callable[[PageInfo], TypeGuard[Optional[PageInfo]]]"  [arg-type]

rich (https://github.com/Textualize/rich)
+ rich/_ratio.py:50: error: Generator has incompatible item type "int"; expected "bool"  [misc]
+ rich/style.py:170: error: Argument 1 to "sum" has incompatible type "tuple[Literal[1, 0], Literal[2, 0], Literal[4, 0], Literal[8, 0], Literal[16, 0], Literal[32, 0], Literal[64, 0], Literal[128, 0], Literal[256, 0], Literal[512, 0], Literal[1024, 0], Literal[2048, 0], Literal[4096, 0]]"; expected "Iterable[bool]"  [arg-type]
+ rich/segment.py:377: error: Generator has incompatible item type "int"; expected "bool"  [misc]

freqtrade (https://github.com/freqtrade/freqtrade)
+ freqtrade/persistence/trade_model.py:775: error: List comprehension has incompatible type List[float]; expected List[bool]  [misc]
+ freqtrade/wallets.py:108: error: Generator has incompatible item type "float"; expected "bool"  [misc]
+ freqtrade/plugins/protections/low_profit_pairs.py:63: error: Generator has incompatible item type "float"; expected "bool"  [misc]

core (https://github.com/home-assistant/core)
+ homeassistant/components/homekit/__init__.py:1099: error: Unsupported operand types for in ("str | None" and "tuple[BinarySensorDeviceClass, SensorDeviceClass]")  [operator]

werkzeug (https://github.com/pallets/werkzeug)
+ src/werkzeug/routing/exceptions.py:94: error: List item 0 has incompatible type "float"; expected "bool"  [list-item]
+ src/werkzeug/routing/exceptions.py:101: error: List item 1 has incompatible type "float"; expected "bool"  [list-item]
+ src/werkzeug/routing/exceptions.py:102: error: Unsupported operand types for * ("float" and "bool")  [operator]

dulwich (https://github.com/dulwich/dulwich)
- dulwich/client.py:2215: error: Unused "type: ignore" comment  [unused-ignore]

github-actions[bot] avatar Oct 07 '24 19:10 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

check-jsonschema (https://github.com/python-jsonschema/check-jsonschema)
+ src/check_jsonschema/cli/parse_result.py:49: error: Generator has incompatible item type "int"; expected "bool"  [misc]

sphinx (https://github.com/sphinx-doc/sphinx)
+ sphinx/util/http_date.py: note: In function "rfc1123_to_epoch":
+ sphinx/util/http_date.py:47:16: error: Unsupported operand types for - ("float" and "int")  [operator]

koda-validate (https://github.com/keithasaurus/koda-validate)
+ koda_validate/serialization/json_schema.py:95: error: Unsupported operand types for + ("list[Predicate[str]]" and "list[PredicateAsync[str]]")  [operator]
+ koda_validate/serialization/json_schema.py:105: error: Unsupported operand types for + ("list[Predicate[bytes]]" and "list[PredicateAsync[bytes]]")  [operator]
+ koda_validate/serialization/json_schema.py:115: error: Unsupported operand types for + ("list[Predicate[int]]" and "list[PredicateAsync[int]]")  [operator]
+ koda_validate/serialization/json_schema.py:125: error: Unsupported operand types for + ("list[Predicate[Decimal]]" and "list[PredicateAsync[Decimal]]")  [operator]
+ koda_validate/serialization/json_schema.py:134: error: Unsupported operand types for + ("list[Predicate[float]]" and "list[PredicateAsync[float]]")  [operator]
+ koda_validate/serialization/json_schema.py:143: error: Unsupported operand types for + ("list[Predicate[date]]" and "list[PredicateAsync[date]]")  [operator]
+ koda_validate/serialization/json_schema.py:152: error: Unsupported operand types for + ("list[Predicate[datetime]]" and "list[PredicateAsync[datetime]]")  [operator]
+ koda_validate/serialization/json_schema.py:169: error: Unsupported operand types for + ("list[Predicate[bool]]" and "list[PredicateAsync[bool]]")  [operator]
+ koda_validate/serialization/json_schema.py:178: error: Unsupported operand types for + ("list[Predicate[UUID]]" and "list[PredicateAsync[UUID]]")  [operator]
+ koda_validate/serialization/json_schema.py:294: error: Unsupported operand types for + ("list[Predicate[dict[Any, Any]]]" and "list[PredicateAsync[dict[Any, Any]]]")  [operator]

streamlit (https://github.com/streamlit/streamlit)
+ lib/streamlit/runtime/pages_manager.py: note: In member "get_initial_active_script" of class "PagesStrategyV1":
+ lib/streamlit/runtime/pages_manager.py:98:21: error: Argument 1 to "filter" has incompatible type "Callable[[Any], Optional[bool]]"; expected "Callable[[PageInfo], TypeGuard[Optional[PageInfo]]]"  [arg-type]
+ lib/streamlit/runtime/pages_manager.py: note: In member "get_page_script" of class "PagesStrategyV2":
+ lib/streamlit/runtime/pages_manager.py:180:21: error: Argument 1 to "filter" has incompatible type "Callable[[Any], Optional[bool]]"; expected "Callable[[PageInfo], TypeGuard[Optional[PageInfo]]]"  [arg-type]

rich (https://github.com/Textualize/rich)
+ rich/_ratio.py:50: error: Generator has incompatible item type "int"; expected "bool"  [misc]
+ rich/style.py:170: error: Argument 1 to "sum" has incompatible type "tuple[Literal[1, 0], Literal[2, 0], Literal[4, 0], Literal[8, 0], Literal[16, 0], Literal[32, 0], Literal[64, 0], Literal[128, 0], Literal[256, 0], Literal[512, 0], Literal[1024, 0], Literal[2048, 0], Literal[4096, 0]]"; expected "Iterable[bool]"  [arg-type]
+ rich/segment.py:377: error: Generator has incompatible item type "int"; expected "bool"  [misc]

freqtrade (https://github.com/freqtrade/freqtrade)
+ freqtrade/persistence/trade_model.py:775: error: List comprehension has incompatible type List[float]; expected List[bool]  [misc]
+ freqtrade/wallets.py:108: error: Generator has incompatible item type "float"; expected "bool"  [misc]
+ freqtrade/plugins/protections/low_profit_pairs.py:63: error: Generator has incompatible item type "float"; expected "bool"  [misc]

core (https://github.com/home-assistant/core)
+ homeassistant/components/homekit/__init__.py:1099: error: Unsupported operand types for in ("str | None" and "tuple[BinarySensorDeviceClass, SensorDeviceClass]")  [operator]

dulwich (https://github.com/dulwich/dulwich)
- dulwich/client.py:2215: error: Unused "type: ignore" comment  [unused-ignore]

werkzeug (https://github.com/pallets/werkzeug)
+ src/werkzeug/routing/exceptions.py:94: error: List item 0 has incompatible type "float"; expected "bool"  [list-item]
+ src/werkzeug/routing/exceptions.py:101: error: List item 1 has incompatible type "float"; expected "bool"  [list-item]
+ src/werkzeug/routing/exceptions.py:102: error: Unsupported operand types for * ("float" and "bool")  [operator]

github-actions[bot] avatar Oct 08 '24 23:10 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

check-jsonschema (https://github.com/python-jsonschema/check-jsonschema)
+ src/check_jsonschema/cli/parse_result.py:49: error: Generator has incompatible item type "int"; expected "bool"  [misc]

sphinx (https://github.com/sphinx-doc/sphinx)
+ sphinx/util/http_date.py: note: In function "rfc1123_to_epoch":
+ sphinx/util/http_date.py:47:16: error: Unsupported operand types for - ("float" and "int")  [operator]

koda-validate (https://github.com/keithasaurus/koda-validate)
+ koda_validate/serialization/json_schema.py:95: error: Unsupported operand types for + ("list[Predicate[str]]" and "list[PredicateAsync[str]]")  [operator]
+ koda_validate/serialization/json_schema.py:105: error: Unsupported operand types for + ("list[Predicate[bytes]]" and "list[PredicateAsync[bytes]]")  [operator]
+ koda_validate/serialization/json_schema.py:115: error: Unsupported operand types for + ("list[Predicate[int]]" and "list[PredicateAsync[int]]")  [operator]
+ koda_validate/serialization/json_schema.py:125: error: Unsupported operand types for + ("list[Predicate[Decimal]]" and "list[PredicateAsync[Decimal]]")  [operator]
+ koda_validate/serialization/json_schema.py:134: error: Unsupported operand types for + ("list[Predicate[float]]" and "list[PredicateAsync[float]]")  [operator]
+ koda_validate/serialization/json_schema.py:143: error: Unsupported operand types for + ("list[Predicate[date]]" and "list[PredicateAsync[date]]")  [operator]
+ koda_validate/serialization/json_schema.py:152: error: Unsupported operand types for + ("list[Predicate[datetime]]" and "list[PredicateAsync[datetime]]")  [operator]
+ koda_validate/serialization/json_schema.py:169: error: Unsupported operand types for + ("list[Predicate[bool]]" and "list[PredicateAsync[bool]]")  [operator]
+ koda_validate/serialization/json_schema.py:178: error: Unsupported operand types for + ("list[Predicate[UUID]]" and "list[PredicateAsync[UUID]]")  [operator]
+ koda_validate/serialization/json_schema.py:294: error: Unsupported operand types for + ("list[Predicate[dict[Any, Any]]]" and "list[PredicateAsync[dict[Any, Any]]]")  [operator]

streamlit (https://github.com/streamlit/streamlit)
+ lib/streamlit/runtime/pages_manager.py: note: In member "get_initial_active_script" of class "PagesStrategyV1":
+ lib/streamlit/runtime/pages_manager.py:98:21: error: Argument 1 to "filter" has incompatible type "Callable[[Any], Optional[bool]]"; expected "Callable[[PageInfo], TypeGuard[Optional[PageInfo]]]"  [arg-type]
+ lib/streamlit/runtime/pages_manager.py: note: In member "get_page_script" of class "PagesStrategyV2":
+ lib/streamlit/runtime/pages_manager.py:180:21: error: Argument 1 to "filter" has incompatible type "Callable[[Any], Optional[bool]]"; expected "Callable[[PageInfo], TypeGuard[Optional[PageInfo]]]"  [arg-type]

rich (https://github.com/Textualize/rich)
+ rich/_ratio.py:50: error: Generator has incompatible item type "int"; expected "bool"  [misc]
+ rich/style.py:170: error: Argument 1 to "sum" has incompatible type "tuple[Literal[1, 0], Literal[2, 0], Literal[4, 0], Literal[8, 0], Literal[16, 0], Literal[32, 0], Literal[64, 0], Literal[128, 0], Literal[256, 0], Literal[512, 0], Literal[1024, 0], Literal[2048, 0], Literal[4096, 0]]"; expected "Iterable[bool]"  [arg-type]
+ rich/segment.py:381: error: Generator has incompatible item type "int"; expected "bool"  [misc]

freqtrade (https://github.com/freqtrade/freqtrade)
+ freqtrade/persistence/trade_model.py:775: error: List comprehension has incompatible type List[float]; expected List[bool]  [misc]
+ freqtrade/wallets.py:108: error: Generator has incompatible item type "float"; expected "bool"  [misc]
+ freqtrade/plugins/protections/low_profit_pairs.py:63: error: Generator has incompatible item type "float"; expected "bool"  [misc]

core (https://github.com/home-assistant/core)
+ homeassistant/components/homekit/__init__.py:1099: error: Unsupported operand types for in ("str | None" and "tuple[BinarySensorDeviceClass, SensorDeviceClass]")  [operator]

werkzeug (https://github.com/pallets/werkzeug)
+ src/werkzeug/routing/exceptions.py:94: error: List item 0 has incompatible type "float"; expected "bool"  [list-item]
+ src/werkzeug/routing/exceptions.py:101: error: List item 1 has incompatible type "float"; expected "bool"  [list-item]
+ src/werkzeug/routing/exceptions.py:102: error: Unsupported operand types for * ("float" and "bool")  [operator]

dulwich (https://github.com/dulwich/dulwich)
- dulwich/client.py:2254: error: Unused "type: ignore" comment  [unused-ignore]

github-actions[bot] avatar Oct 24 '24 06:10 github-actions[bot]

Hm, it seems like this might be causing functions like + and sum to behave strangely. Will have to look later.

huonw avatar Oct 24 '24 07:10 huonw

Hm, looks like they're spurious errors: https://github.com/python/mypy/issues/18026

huonw avatar Oct 24 '24 07:10 huonw

Yep, with further analysis (in #18026) this change is blocked on having a mechanism to avoid those spurious errors.

huonw avatar Oct 31 '24 00:10 huonw