Use of `ParamSpec` introduced in #3396 only works in Python 3.10+

[rsokl]

The pattern used by #3396 to leverage ParamSpec

https://github.com/HypothesisWorks/hypothesis/blob/0738717561e419558a831869ab2b81dd918276aa/hypothesis-python/src/hypothesis/strategies/_internal/core.py#L123-L129

does not actually work with the typing_extensions backport:

# contents of pspec.py

from hypothesis.strategies import composite, DrawFn
from typing_extensions import ParamSpec  # note: typing-extensions is installed

@composite
def comp(draw: DrawFn, x: int) -> int:
    return x

reveal_type(comp)

$ pip freeze | grep typing_extensions
typing_extensions==4.3.0

$ mypy --version
mypy 0.961 (compiled: yes)

$ mypy pspec.py --python-version=3.10
pspec.py:11: note: Revealed type is "def (x: builtins.int) -> hypothesis.strategies._internal.strategies.SearchStrategy[builtins.int]"
Success: no issues found in 1 source file

$ mypy pspec.py --python-version=3.9
pspec.py:11: note: Revealed type is "Any"
Success: no issues found in 1 source file

pyright repro

$ pyright --version
pyright 1.1.257

$ pyright pspec.py --pythonversion=3.10
No configuration file found.
No pyproject.toml file found.
stubPath C:\Users\Ryan Soklaski\hypothesis\scratch\typings is not a valid directory.
Assuming Python platform Windows
Searching for source files
Found 1 source file
pyright 1.1.257
C:\Users\Ryan Soklaski\hypothesis\scratch\pspec.py
  C:\Users\Ryan Soklaski\hypothesis\scratch\pspec.py:11:13 - information: Type of "comp" is "(x: int) -> SearchStrategy[int]"
0 errors, 0 warnings, 1 information

$ pyright pspec.py --pythonversion=3.9
No configuration file found.
No pyproject.toml file found.
stubPath C:\Users\Ryan Soklaski\hypothesis\scratch\typings is not a valid directory.
Assuming Python platform Windows
Searching for source files
Found 1 source file
pyright 1.1.257
C:\Users\Ryan Soklaski\hypothesis\scratch\pspec.py
  C:\Users\Ryan Soklaski\hypothesis\scratch\pspec.py:7:2 - error: Argument of type "(draw: DrawFn, x: int) -> int" cannot be assigned to parameter "f" of type "() -> Ex@composite" in function "composite"
    Type "(draw: DrawFn, x: int) -> int" cannot be assigned to type "() -> Ex@composite"
      Keyword parameter "draw" is missing in destination
      Keyword parameter "x" is missing in destination (reportGeneralTypeIssues)
  C:\Users\Ryan Soklaski\hypothesis\scratch\pspec.py:11:13 - information: Type of "comp" is "() -> SearchStrategy[int]"
1 error, 0 warnings, 1 information

It seems like nested try-excepts are not supported by mypy:

# contents of scratch.py

import typing

try:
    from typing import Concatenate, ParamSpec
except ImportError:
    try:
        from typing_extensions import Concatenate, ParamSpec
    except ImportError:
        ParamSpec = None  # type: ignore


if typing.TYPE_CHECKING or ParamSpec is not None:
    reveal_type(ParamSpec)

$ mypy scratch.py --python-version=3.10
scratch.py:13: note: Revealed type is "def (name: builtins.str, *, bound: Union[Any, None] =, contravariant: builtins.bool =, covariant: builtins.bool =) -> typing.ParamSpec"
Success: no issues found in 1 source file

$ mypy scratch.py --python-version=3.9
scratch.py:4: error: Module "typing" has no attribute "Concatenate"
scratch.py:4: error: Module "typing" has no attribute "ParamSpec"; maybe "_ParamSpec"?
scratch.py:13: note: Revealed type is "Any"
Found 2 errors in 1 file (checked 1 source file)

The following works for mypy for both Python 3.10 and the typing-extensions backport, but only works for pyright for Python 3.10 (potentially relevant comment from Eric Traut):

if sys.version_info >= (3, 10):
    from typing import Concatenate, ParamSpec
elif typing.TYPE_CHECKING:
    try:
        from typing_extensions import Concatenate, ParamSpec
    except ImportError:
        from typing import Any as ParamSpec
else:
    ParamSpec = None

Whereas this works for pyright under all circumstances, but fails in mypy when typing-extensions is not installed:

if sys.version_info >= (3, 10):
    from typing import Concatenate, ParamSpec
elif typing.TYPE_CHECKING:
    from typing_extensions import Concatenate, ParamSpec
else:
    ParamSpec = None

I've gotta go to bed.. @sobolevn I'm wondering if you have any recommendations here.

[Zac-HD]

Thanks for the report! We also have an inadequate-tests problem then...

[rsokl]

We do have a (recent) precedent for parameterizing type-checker tests over python versions 😄

Tests that I can think of:

• [ ] Python 3.10+ mypy/pyright should pass current tests
• [ ] Python 3.10< mypy/pyright & no backport: @composite should mask decorated function's signatures, but express the accurate return type SearchStrategy[Ex]
• [ ] (Add typing_extensions to CI matrix.) With backport: All Python versions should pass current tests.

I was trying to remember: what is the reason why we avoid adding typing_extensions to our dependencies?

[Zac-HD]

I was trying to remember: what is the reason why we avoid adding typing_extensions to our dependencies?

Ensuring that we don't pick up an accidental hard-dependency; it is in our tools and coverage deps, just not the minimal test deps, and I think that's the right balance.

[sobolevn]

In my opinion, depending on typing_extensions is not a big deal. Almost every typed package depend on it.

It will simplify a lot of things for us.

[Zac-HD]

The main argument against is that minimising dependencies is a big deal for PyPy, CPython, and other foundational projects. Maybe it's worth it anyway; I'd need to look into the compatibility story with prerelease and nightly builds.

[Mec-iS]

same error here I think, it broke our schemathesis test pipeline. Error seems to generate in hypothesis core.

Everything works fine with version < 6.49. After upgrading to version >= 6.49, this error appeared:

platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: ..., configfile: pytest.ini
plugins: anyio-3.5.0, schemathesis-3.15.6, hypothesis-6.49.1, subtests-0.5.0, nbmake-1.3.0
collected 0 items / 1 error                                                                                                                                                                       

============================================================================================= ERRORS ==============================================================================================
_________________________________________________________________________ ERROR collecting tests/... _________________________________________________________________________
../.venv/lib/python3.8/site-packages/pluggy/_hooks.py:265: in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
../.venv/lib/python3.8/site-packages/pluggy/_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
../.venv/lib/python3.8/site-packages/schemathesis/extra/pytest_plugin.py:211: in pytest_pycollect_makeitem
    outcome.force_result(create(SchemathesisCase, parent=collector, test_function=obj, name=name))
../.venv/lib/python3.8/site-packages/schemathesis/extra/pytest_plugin.py:38: in create
    return cls.from_parent(*args, **kwargs)  # type: ignore
../.venv/lib/python3.8/site-packages/_pytest/nodes.py:264: in from_parent
    return cls._create(parent=parent, **kw)
../.venv/lib/python3.8/site-packages/_pytest/nodes.py:140: in _create
    return super().__call__(*k, **kw)
../.venv/lib/python3.8/site-packages/schemathesis/extra/pytest_plugin.py:78: in __init__
    failing_test = validate_given_args(test_function, given_args, given_kwargs)
../.venv/lib/python3.8/site-packages/schemathesis/utils.py:446: in validate_given_args
    return is_invalid_test(func, argspec, args, kwargs)  # type: ignore
../.venv/lib/python3.8/site-packages/hypothesis/core.py:278: in is_invalid_test
    params = list(original_sig.parameters.values())
E   AttributeError: 'FullArgSpec' object has no attribute 'parameters'

[rsokl]

@Mec-iS I think you should open a separate issue, even through our issues likely stem from the same change. The bug I reported here pertains specifically to the behavior of static type checkers, whereas you are hitting an actual runtime error which likely warrants a higher priority.

[Zac-HD]

@Mec-iS that's actually using undocumented/private Hypothesis internals, so it'll need to be fixed on the schemathesis side (sorry @Stranger6667!).

[Zac-HD]

Quick untested hack: https://github.com/HypothesisWorks/hypothesis/commit/854e8ab9c3b158abe56793bf64b20baf4146c965

[hauntsaninja]

Your hack should work. Type checkers are special cased to always know what typing_extensions is even if it isn't installed (it's treated like part of the standard library).

[Zac-HD]

Ooh, thanks for dropping by! I'm a little concerned that the hack will fail in some environments, so what about:

if typing.TYPE_CHECKING or sys.version_info[:2] >= (3, 10):
    from typing import Concatenate, ParamSpec
else:
    try:
        from typing_extensions import Concatenate, ParamSpec
    except ImportError:  # pragma: no cover
        ParamSpec = None

[hauntsaninja]

No, type checkers will complain about that when checking targeting Python 3.9. See https://mypy-play.net/?mypy=latest&python=3.9&gist=0495f5e1337cd862c26dbb2de4d05633

I would do:

if typing.TYPE_CHECKING:
    from typing_extensions import Concatenate, ParamSpec
else:
    try:
        from typing import Concatenate, ParamSpec
    except ImportError:
        try:
            from typing_extensions import Concatenate, ParamSpec
        except ImportError:
            Concatenate, ParamSpec = None, None

I guess if you don't fully believe that all type checkers always know about typing_extensions, you could hedge a little more with:

if typing.TYPE_CHECKING:
    if sys.version_info >= (3, 10):
        from typing import Concatenate, ParamSpec
    else:
        from typing_extensions import Concatenate, ParamSpec
else:
    try:
        from typing import Concatenate, ParamSpec
    except ImportError:
        try:
            from typing_extensions import Concatenate, ParamSpec
        except ImportError:
            Concatenate, ParamSpec = None, None

But they really always do. typing_extensions has been in the standard library since Python 2.7, don't you know :-) https://github.com/python/typeshed/blob/2c052651e953109c94ae998f5ccc6d043df060c9/stdlib/VERSIONS#L273

[rsokl]

Option 1 does indeed work with pyright without typing-extensions actually being installed.

Type checkers are special cased to always know what typing_extensions is even if it isn't installed (it's treated like part of the standard library).

In the case of mypy, typing-extensions is installed as a dependency so of course it works in that regard. Out of curiosity I uninstalled typing-extensions after installing mypy and running it on both options 1 and 2 raises ModuleNotFoundError: No module named 'typing_extensions'.

$ mypy scratch/scratch.py --python-version=3.9
Traceback (most recent call last):
  File "C:\Users\rsokl\miniconda3\envs\hydra-zen\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\rsokl\miniconda3\envs\hydra-zen\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\rsokl\miniconda3\envs\hydra-zen\Scripts\mypy.exe\__main__.py", line 4, in <module>
  File "C:\Users\rsokl\miniconda3\envs\hydra-zen\lib\site-packages\mypy\__main__.py", line 6, in <module>
    from mypy.main import main, process_options
  File "mypy\main.py", line 11, in <module>
ModuleNotFoundError: No module named 'typing_extensions'

[hauntsaninja]

@rsokl that traceback is mypy itself crashing. The better proof for this is using the --python-executable flag of mypy to point it at an environment where typing_extensions doesn't exist. Something like:

echo $'import typing_extensions\nreveal_type(typing_extensions.ParamSpec)' > mod.py
python -m venv env
pipx run mypy --python-executable env/bin/python mod.py

Or more thoroughly, see that numpy stops being found, but typing_extensions is still found:

python -m venv env1
python -m venv env2
env1/bin/python -m pip install mypy numpy

echo $'import typing_extensions\nreveal_type(typing_extensions.ParamSpec)\nimport numpy\nreveal_type(numpy.uint32)' > mod.py
env1/bin/python -m mypy --python-executable env1/bin/python mod.py
env1/bin/python -m mypy --python-executable env2/bin/python mod.py

[rsokl]

Ah, gotchya. Thanks for clarifying @hauntsaninja . I should have couched my post with "I am sure that you know what you are talking about, whereas I have little experience with mypy" 😅