pdoc Force display of method if defined in stub file

I'm currently writing a (company internal) library with one module behaving somewhat similar to python ctypes but im programming it in rust using pyo3. I have implemented fix sized types like U8 for unsigned integer with 8 bits and the possibility to create arrays by using the __get_item__ method, like:

U8(8) # instance of a non array type
U8[3]([0x01, 0x02, 0x03]) # instance of a array type with size 3

Since this is a method on the class but not the instance, I need to define the method on the metaclass. This is currently not supported in pyo3, so instead I load the rust class, and inherit into a new class like this (__init__.py):

from typing import Protocol, Any
from pdoc_example import _rust

class _ArrayAble(Protocol):
    @classmethod
    def __make_array__[C](cls: type[C], size: int) -> Any: ...

U8Array = _rust.U8Array

class _U8ArrayMeta(type):
    def __getitem__[C: _ArrayAble](cls: type[C], size: int) -> type[U8Array]:
        return cls.__make_array__(size)

class U8(_rust.U8, metaclass=_U8ArrayMeta): ...

Classes that dont need to be changed are just reexported like U8Array. The import is from a local module named _rust.pyd and stup files for the content are provided under a folder _rust. This way both the type checker and the runtime imports are satisfied.

pdoc does currently not resolve the pyi file for imports like vscode does with the pylance plugin. For pdoc to properly annotate the types i need to add a __init__.pyi next to the __nit__.py file so my file structure looks like this:

PDOC_EXAMPLE │ py.typed │ _rust.cp312-win_amd64.pyd │ init.py │ init.pyi └───_rust/__init__.pyi

class U8:
    """U8 in __init__.pyi"""
    def __init__(self, value: int) -> None:
        """U8 init function in __init__.pyi"""
    @property
    def value(self) -> int:
        """U8 property value in __init__.pyi"""
    
class U8Array:
    """U8Array in __init__.pyi"""
    def __init__(self, value: list[int]) -> None:
        """U8Array init function in __init__.pyi"""
    @property
    def value(self) -> list[int]:
        """U8Array property value in __init__.pyi"""

If i run pdoc, the documentation will be correct for U8Array (since it is directly imported) but not for U8 since this is an inherited method and we dont publish those.

I understand the decision to not show inherited members that are not itself documented in the package. I wouldnt even want the information what methods are inherited if i have no possibility to jump to the inherited methods documentation.

However I would like to have a way to document methods that actually are available. It's also not possible to document class attributes if they are not actually available in the variable but i want the user to set them:

# the user should set the size variable if he inherits from U8Array and i want to tell him that in the documentation
class MyArray(U8Array):
    _size_ = 3

On creation U8Array will look for this classvar and raise an exception if it is not available. You are not supposed to use the U8Array directly but inherit like in the above example or use the U8[size] syntax.

Any documentation inside the pyi file will be dismissed because pdoc will do the dynamic analysis and not allow me to add anything other than annotations for methods and variables that were resolved before.

I'd be willing to contribute a pull request but since there are probably more than one way to solve this issue and it might involve some architectural decisions i first wanted to raise the discussion. Overall my goal is to find a way to satisfy both vscode/pyright and pdoc

should pdoc try to enhance import statements from compiled modules with stub files
should there be a possibility to document features that are not available or inherited in the dynamic code resolution

I can also provide a mvp setup to reproduce the issues i described if wanted.

Mar 13 '25 13:03 BenediktMaag

This is quite some metaprogramming you are doing here. I understand the motivation, but at the same time I don't think that this is a use case that pdoc should make special accomodations (i.e. add complexity in pdoc) for.

should pdoc try to enhance import statements from compiled modules with stub files

Is there a PEP that advocates for such an approach? Otherwise I don't see a good justification for doing this.

should there be a possibility to document features that are not available or inherited in the dynamic code resolution

There already is. You can use pdoc as a library (https://pdoc.dev/docs/pdoc.html#using-pdoc-as-a-library) and patch the documentation objects to match your needs. There's an example over at https://github.com/mitmproxy/pdoc/blob/main/examples/library-usage/make.py. These types of use cases will always be super specific, so having the full power of Python available to fix them is handy.

Mar 13 '25 18:03 mhils

I understand your reluctance to add complexity. I'll give the Python api a try and see if it fits my needs.

As for PEP compliance: https://peps.python.org/pep-0484/#stub-files

It states: There are several use cases for stub files: Extension modules (imo what a pyd file is) ... If a stub file is found the type checker should not read the corresponding “real” module.

So this could be interpreted as "pyd never contain stub files so look in the folder instead".

The usecase i displayed is simplified as it does not contain submodules which my real project does.

Mar 13 '25 18:03 BenediktMaag

So I did some investigation and was able to patch the modules with the inherited types. Here is what i did incase anybody wants to repeat it:

copy the calls in pdoc.pdoc since i want the recursive module search as well
after loading the modules in the all_modules dict execute the following:

from pdoc import doc

def treat_inheritance_with_same_name_as_uninherited(module: doc.Module, cls_names: list[str]):
    for cls_name in cls_names:
        cls = module.get(cls_name)
        for method in cls.inherited_members[("builtins", cls_name)]:
            method.is_inherited = False
            if not method.name.startswith("_"):
                cls.own_members.append(method)

treat_inheritance_with_same_name_as_uninherited(
    all_modules["pdoc_example"],
    ["U8", "U16", "U32", "U64", "I8", "I16", "I32", "I64", "String", "ByteVector"],
)

So to sum it up i copy the methods from inherited_members to own_members and set the is_inherited flag of the method to false. This works as expected, only downside is I had to copy paste alot of code.

Mar 14 '25 14:03 BenediktMaag

should pdoc try to enhance import statements from compiled modules with stub files

Is there a PEP that advocates for such an approach? Otherwise I don't see a good justification for doing this.

I also looked into this. There is no reason pdoc would not do this the way since the logic for patching doesnt care if it is a compiled file or a written in python: https://github.com/mitmproxy/pdoc/pull/390/commits/1fc683aaf65a922622d09bd4a6ae10838059fe8a#diff-c3e5f41573442524d95f9463acefe2b34b9f7ab982068a693c8f4e804896ec06

In pdoc.extract.itermodules2 the following code snipped correctly states that pkgutil does not detect pyo3 submodules:

    # 2023-12: PyO3 and pybind11 submodules are not detected by pkgutil
    # This is a hacky workaround to register them.
    members = dir(module) if mod_all is None else mod_all
    for name in members:
        if name in submodules or name == "__main__" or not isinstance(name, str):
            continue
        ...

However I'm not sure how this code snipped aims to accomplish what it trys to. In pdoc_example/__init__.py I define __all__ with my members.

Wether or not I include '_rust' into __all__ the following statement holds true:

>>> sorted(members) == list(submodules.keys())
True

So the first if condition is always hit the the loop continues. If the condition would be skipped is_wild_child_module would be set to true and the module would be added as ispkg=True

Mar 14 '25 14:03 BenediktMaag