cpython icon indicating copy to clipboard operation
cpython copied to clipboard

gh-107001: Add a stdlib decorator that copies/applies the ParameterSpec from one function to another

Open CarliJoy opened this issue 1 year ago • 6 comments

Add a stdlib decorator that copies/applies the ParameterSpec from one function to another.

All information can be found in the related issue

Note for review for JelleZijlstra and AlexWaygood (and maybe the typing council)

  • Issue: gh-107001

📚 Documentation preview 📚: https://cpython-previews--121693.org.readthedocs.build/en/121693/library/typing.html#typing.copy_kwargs

CarliJoy avatar Jul 13 '24 13:07 CarliJoy

All commit authors signed the Contributor License Agreement.
CLA signed

ghost avatar Jul 13 '24 13:07 ghost

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

bedevere-app[bot] avatar Jul 13 '24 13:07 bedevere-app[bot]

Since this decorator would need to be handled specially by type checkers, let's not add it to CPython until there is a corresponding change to the typing spec.

JelleZijlstra avatar Jul 13 '24 14:07 JelleZijlstra

Since this decorator would need to be handled specially by type checkers, let's not add it to CPython until there is a corresponding change to the typing spec.

What are you referring to as "handled specially"? Do you mean that checking that the signature of the decorated and source function shall be checked "by specification" by the type checkers? Otherwise it already works with the existing type system without any changes to the type checkers (see linked SO/MyPy Play in related Issue)

I can create a PR for the Spec as well.

Independent of this I still would be interested if the name is okay and if I should include skip_first (see PR description).

CarliJoy avatar Jul 13 '24 15:07 CarliJoy

Let's convert to draft for now.

hugovk avatar Jul 13 '24 15:07 hugovk

Otherwise it already works with the existing type system without any changes to the type checkers (see linked SO/MyPy Play in related Issue)

You're right, I missed this, sorry. If this signature can be expressed in the existing type system, it doesn't need to be specified explicitly.

JelleZijlstra avatar Jul 13 '24 15:07 JelleZijlstra

If this function doesn't need special handling from type checkers, what is the motivation for adding it to typing?

We have a few existing helpers in typing that are not special forms (e.g., assert_never, AnyStr), but I don't know if this proposed addition is quite as pervasively useful as the existing ones.

As an alternative, have you considered contributing this function to https://github.com/hauntsaninja/useful_types/?

The function doesn't currently require type checkers to add any special support. However, introducing it in the typing module could pave the way for future enhancements, where type checkers or static analysis tools might validate that the parameters of two functions actually match. This would allow Python projects to ensure that signature changes—whether in their own code or due to third-party library updates—are caught early, preventing mismatches that could otherwise lead to subtle bugs.

While contributing this function to a separate project, like useful_types, is an option, it has limitations. Projects that suggest vendorization can complicate maintenance, especially in larger codebases. Moreover, this decorator addresses a common enough use case that it warrants inclusion in the standard library, ensuring both accessibility and support across Python versions.

CarliJoy avatar Oct 25 '24 14:10 CarliJoy

While contributing this function to a separate project, like useful_types, is an option, it has limitations. Projects that suggest vendorization can complicate maintenance, especially in larger codebases. Moreover, this decorator addresses a common enough use case that it warrants inclusion in the standard library, ensuring both accessibility and support across Python versions.

Then I'd suggest you contribute it to useful_types first, and only add it to the standard library after it has actually proven to be widely useful.

Note that anything we add now will only be released with Python 3.14, in October 2025 (though we'd add it to typing-extensions earlier).

JelleZijlstra avatar Oct 25 '24 16:10 JelleZijlstra

Then I'd suggest you contribute it to useful_types first, and only add it to the standard library after it has actually proven to be widely useful.

Actually, I believe this decorator needs the visibility only the standard library can provide.

The purpose of this PR is to make typing in Python both easier and more robust. Extending functions and methods is common across all experience levels, but creating this kind of decorator is challenging—I found it difficult to arrive at the solution myself. The number of threads on discuss.python.org, along with the Stack Overflow issue referenced in #107001, demonstrates that there’s a real demand and that others are also struggling.

Note that anything we add now will only be released with Python 3.14, in October 2025 (though we'd add it to typing-extensions earlier).

Adding it now means it will already be visible in the Python 3.14 documentation, making it easy to find and use as an example.

I’m not in favor of placing this in a third-party library, as it would limit additional ideas I have for this decorator. For example, as noted in the documentation, using it incorrectly can actually reduce type safety. I’m already considering ways to support tools like ruff in checking that decorated functions correctly include *args and **kwargs.

In the longer term, I’d even like to explore extending the Python type system itself to allow checking if a source function’s call signature is compatible with a decorated function.

Adding this to a third-party library would impede this progress and keep the solution obscure.

I understand every addition has maintenance costs, but the actual code is just six lines returning the function. So what’s the reason for being so reluctant to add it?

In the end, my goal is to contribute to a Python ecosystem with fewer untyped *args and **kwargs that neither I nor my IDE or type checker can fully interpret. I hope that including this decorator is a small but meaningful step in that direction.

CarliJoy avatar Oct 28 '24 00:10 CarliJoy

All commit authors signed the Contributor License Agreement.
CLA signed

ghost avatar Oct 28 '24 00:10 ghost

@JelleZijlstra Happy New year. Did you have some time to look at my arguments?

CarliJoy avatar Jan 07 '25 12:01 CarliJoy

Your statements about "additional ideas" and "extending the Python type system" make me even more reluctant to add this function now. Putting something in the standard library effectively means freezing it, and any future additions or tweaks have to overcome a large compatibility barrier.

And if nobody is willing to contribute this feature to a third-party library like useful-types, that again makes me question how commonly useful the feature is. The standard library isn't the place to incubate new features.

JelleZijlstra avatar Jan 10 '25 03:01 JelleZijlstra

that again makes me question how commonly useful the feature is.

Let me just chime in here. I found this PR while looking for a feature such as this. My project, Locust, subclasses requests.Session and overrides a few methods to add logging/tracking to each HTTP request, sometimes with identical signature, sometimes adding a few additional keyword arguments. Requests has a lot methods (get, post, put, etc) that support the same parameters as a more generic request()-method. Being able to copy the method signatures from that class to further wrappers in subclasses is very convenient (and I've been able to successfully use a copy of the code in this PR).

Most importantly, I think there is overwhelming evidence out there that a feature such as this is frequently sought, with no really good answers: https://stackoverflow.com/questions/59717828/copy-type-signature-from-another-function https://stackoverflow.com/questions/65954587/copying-function-signature-in-a-method https://stackoverflow.com/questions/42420810/copy-signature-forward-all-arguments-from-wrapper-function https://stackoverflow.com/questions/35512112/how-can-i-copy-a-python-functions-signature-and-define-a-new-function https://www.reddit.com/r/learnpython/comments/sksy85/copy_type_signature_of_a_parameter_of_another/ https://discuss.python.org/t/copying-signature-of-init-to-a-mixin-class-method/58907/4

(that being said, personally I would have no problem using this feature just because it wasn't in the stdlib, I just want it somewhere :)

cyberw avatar Jan 11 '25 21:01 cyberw

Thanks for chiming in, cyberw. I didn't find all the results you were looking for even so I looked quite intensively in the past.

The sole reason for this MR is to make the already existing way to copy function parameters from one function to another known and easily usable.

This PR achieves both goals.

As documented, there is a possible issue with this decorator: if the copied call signature is incompatible with the decorated function's call signature, it can lead to a Runtime TypeError that type checkers won’t catch 💣. Essentially, the decorated function’s signature is overwritten. [Details]

My "additional ideas" about extending the Python type system aim to assist user to prevent this.

"Putting something in the standard library effectively means freezing it, and any future additions or tweaks have to overcome a large compatibility barrier."

I did not propose changing the stdlib implementation in future here. Instead, I have two ideas to address the issue above,:

  1. Short-term: Implement a Ruff rule to ensure all decorated functions use *args and **kwargs, reducing the risk of runtime type errors (as noted in the documentation). The stdlib "frozen" state is desired here.
  2. Long-term/Maybe: Advocate for a change to the typing specification, enabling type checkers to validate the compatibility of the copied call signature with the decorated function.

Including this in useful-types wouldn’t make it easier to find or use¹. I’d rather keep it on StackOverflow, where users can also learn about the potential pitfalls..

Never the less: Both options make it equally hard to create a Ruff rule to help users avoid the issue.

¹ useful-types does not have a documentation and is rather unknown.

CarliJoy avatar Jan 13 '25 16:01 CarliJoy

I haven't heard about useful-types, but I deeply care about typing my python code, and that's not the first time I'm looking for such function.

I copied it into my app/utils/typing.py, and it works fine.

Would it be too stupid to put this into typeshed as functools.wraps? It is copying signature...

In [1]: from functools import wraps

In [2]: def f1(a: str, b: str):
   ...:     """Hello from f1"""
   ...:

In [3]: @wraps(f1)
   ...: def f2(*args, **kwargs):
   ...:     return f1(*args, **kwargs)
   ...:

In [4]: f2?
Signature: f2(a: str, b: str)
Docstring: Hello from f1
File:      ~/<ipython-input-2-8ca42cdf805a>
Type:      function

last-partizan avatar Aug 22 '25 08:08 last-partizan