Next steps for type introspection and stub generation

[Tpt]

This is an incomplete list of next steps to build proper stubs from PyO3 codebases:

• [ ] introspect all class methods (including adding a test for all magic methods) (partial: #5273)
• [x] introspect classes associated constants (const) #5272
• [x] introspect modules associated constants (const) #5096
• [ ] introspect simple enums built by #[pyclass]
• [ ] introspect complex enums built by #[pyclass]
• [ ] introspect exceptions built by create_exception!
• [x] introspect class inheritance (#[pyclass(extends=)]) #5331
• [x] @typing.final decorator on class that cannot be subclasses (without #[pyclass(subclass)]) #5552
• [x] introspect auto-generated magic methods (#[pyclass(eq, eq_int, ord, hash, str)]) #5338
• [x] introspect fields getter and setter (#[pyo3(get, set)], #[pyclass(get_all, set_all)]) #5370
• [x] implement return type annotation #5208
• [x] fill PYTHON_TYPE constant on all implementations of the relevant traits (FromPyObject and IntoPyObject) #5634 #5637 #5639 #5640
• [ ] fill INPUT_TYPE in #[derive(FromPyObject)] #5339
• [ ] fill OUTPUT_TYPE in #[derive(IntoPyObject)] #5365
• [ ] figure out how to emit #[classattr] in the stubs
• [x] support cross-modules types (class defined in module A and used in module B as an input/output type)
• [ ] add doc strings to the generated stubs
• [ ] proper test coverage
• [ ] proper formatting of stubs (not a blocker)
• [ ] integration into maturin
• [x] proper type stubs for containers (list[T] instead of list). #5639
• [x] allow to set custom type annotations (both inputs and output) #5241
• [ ] allow to set custom stubs for eg. protocols
• [x] add _typeshed.Incomplete to relevant places (modules with a #[pymodule_init] function...) (doc) #5207
• [ ] choose between _typeshed.Incomplete and typing.Any in type annotations
• [ ] make sure to properly gate introspection element with cfg macros
• [ ] provide a way to set @overloads
• [ ] Support WASM files in pyo3-introspection
• [ ] Allow custom type annotations without strings (example: #[pyo3(signature = (arg: list[int]) -> list[int])])
• [ ] Add version information to the generated JSON blobs
• [ ] Figure out if the function used to tag incomplete modules should be generated by pyo3-macros instead
• [ ] Make sure we don't try to import None from builtins, it does not work

Reference on stubs file

[davidhewitt]

For "custom type annotations", I think maybe adding it to #[pyo3(signature)] attribute is the way to go (xref #5138)

// `a` and return type set explicitly, b will be inferred
#[pyo3(pyfunction, signature = (a: int, b) -> int)]
fn foo(a: i32, b: i32) -> i32 {
    a + b
}

... I guess the alternative would be #[pyo3(type = "int")] annotations on each argument, but I think using the Python syntax is much better.

[yogevm15]

Hi, I would like to help here, is there anything I should be aware of before I start working on it?

[Tpt]

@yogevm15 Thank you! It would be amazing!

I am currently working on function type annotations #5089

There are a bunch of topics that are quite independent from it like exposing const (without type annotation), supporting properly enums...

To help you deep dive in the code:

• Introspection data are emitted as const elements by the various PyO3 macros. Code for that is in pyo3-macros-backend/src/introspection.rs. Each element is a JSON string.
• The new pyo3-introspection parses the built dynamic libs objects, extracts the elements and builds the pyi stub files.
• Testing is done by building the pytests crate, run pyo3-introspection on it and check if the output is equal to the stub files in pytests/stubs. To run the test do nox -s test-introspection.

Note that introspection is only working with modules declared using inline Rust modules (mod XXX) and not with modules declared with functions (fn XXX)

[yogevm15]

Thanks, I opened a PR for the modules associated consts #5150

I'm not sure about the classes associated consts. If I understood the code correctly, it exports them as class methods rather than actual class variables. So, how should we generate them in the stub?

[purepani]

Thanks for all this work; I'm quite excited for this! Could you update this issue with the current in progress PRs, and the already merged ones?

[wiktor-k]

I've used experimental-inspect feature of pyo3 0.26 and pyo3_introspection and I must say it already works very, very well! 👏

Two issues found when running mypy on a code which uses generated stubs:

pysequoia.pyi:43: error: "__new__" must return a class instance (got "None")  [misc]

Which is caused by the following fragment of the auto-generated stub:

class Notation:
    def __new__(cls, /, key: str, value: str) -> None: ...

Generated from this Rust-code:

#[pymethods]
impl Notation {
    #[new]
    pub fn new(key: String, value: String) -> Self {
        Self { key, value }
    }
...

It seems the None in the stub case shouldn't be there.

The second issue is more subtle:

pysequoia.pyi:69: error: Argument 1 of "__eq__" is incompatible with supertype "builtins.object"; supertype defines the argument type as "object"  [override]
pysequoia.pyi:69: note: This violates the Liskov substitution principle
pysequoia.pyi:69: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
pysequoia.pyi:69: note: It is recommended for "__eq__" to work with arbitrary objects, for example:
pysequoia.pyi:69: note:     def __eq__(self, other: object) -> bool:
pysequoia.pyi:69: note:         if not isinstance(other, SignatureMode):
pysequoia.pyi:69: note:             return NotImplemented
pysequoia.pyi:69: note:         return <logic to compare two SignatureMode instances>
pysequoia.pyi:70: error: Argument 1 of "__ne__" is incompatible with supertype "builtins.object"; supertype defines the argument type as "object"  [override]
pysequoia.pyi:70: note: This violates the Liskov substitution principle
pysequoia.pyi:70: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides

Caused by this:

class SignatureMode:
    def __eq__(self, /, other: SignatureMode | int) -> bool: ...
    def __ne__(self, /, other: SignatureMode | int) -> bool: ...

Generated from this:

#[pyclass(eq, eq_int)]
#[derive(PartialEq)]
pub enum SignatureMode {
    #[pyo3(name = "INLINE")]
    Inline,
    #[pyo3(name = "DETACHED")]
    Detached,
    #[pyo3(name = "CLEAR")]
    Clear,
}

(Removing eq, eq_int helps). I'm not sure if mypy is overly sensitive here (it wants object instead of SignatureMode | int) as the type describe exactly what can go in there... but maybe it's a separate issue.

(Just in case it helps, here's my code: https://github.com/wiktor-k/pysequoia/pull/51/files#diff-ca800b2c3ea5475c099fdab5613417c051dd4435a00279d4220badc43537bf28R43)

I hope you don't mind me sharing these code fragments here, but if they should go in separate issues, then sorry for the noise. 🙏

Thanks for your work on this!