pdoc3 references internal symbols instead of public ones

[CarstenLeue]

I am keeping my python implementation details in a package called _internal and export the methods, classes and variables that are supposed to be the public API by re-exporting them in the __init__.py of the parent package, e.g. like so:

from ._internal.interface import MyInterface
from ._internal.sample import MY_VAR, my_function

__all__ = ['MyInterface', 'MY_VAR', 'my_function']

the signature e.g. of my_function is:

def my_function(iface: MyInterface) -> str:
    """My function doc

    Args:
        iface: interface parameter doc

    Returns:
        return value doc
    """    

   ...

Expected Behavior

• all three entities from __all__ to be visible in the documentation
• the documentation of my_function to reference the public type MyInterface
• the documentation of the arguments of my_function to show type information
• ideally a link from the type MyInterface in the signature of my_function to the documentation of MyInterface

Actual Behavior

• the variable MY_VAR is not documented at all
• the signature of my_function shows the internal type name not the public one (it shows def my_function (iface: pdoc_test._internal.interface.MyInterface) ‑> str) instead of def my_function(iface: pdoc_test.MyInterface) ‑> str
• the documentation of the arguments and the return value of my_function does not show any type hint

Steps to Reproduce

I have create a small repo to reproduce at: https://github.com/Carsten-Leue/pdoc-test

Generated documentation showing the issue is here: https://carsten-leue.github.io/pdoc-test/pdoc_test/

Additional info

• pdoc version: 0.8.3

[kernc]

Thanks for these issues!

• the variable MY_VAR is not documented at all

MY_VAR is neither function nor class and it doesn't have a PEP 224 variable docstring, so the branch skips it. I guess we might amend the branch to check for inclusion in __all__ too, albeit sans any docstring. https://github.com/pdoc3/pdoc/blob/04960e41b9bb598b664a50078f4c233817326170/pdoc/init.py#L622-L628

• the signature of my_function shows the internal type name not the public one (it shows def my_function (iface: pdoc_test._internal.interface.MyInterface) ‑> str) instead of def my_function(iface: pdoc_test.MyInterface) ‑> str

Duplicate of https://github.com/pdoc3/pdoc/issues/231. Not quite sure how to tackle this one.

• he documentation of the arguments and the return value of my_function does not show any type hint

Duplicate of https://github.com/pdoc3/pdoc/issues/112. Interpolating literal docstring strings with inferred types will be clumsy at best. If someone wants to try to tackle it, sure, but it likely won't be me. :sweat_smile:

[Carsten-Leue]

MY_VAR is neither function nor class and it doesn't have a PEP 224 variable docstring, so the branch skips it. I guess we might amend the branch to check for inclusion in __all__ too, albeit sans any docstring.

The source of MY_VAR is:

MY_VAR = 10
"""This is my variable"""

Isn't this string a PEP224 docstring for this variable?

[kernc]

It is. But that's in the module _internal, I suppose, not in the module where __all__ is defined and MY_VAR exported ...

[Carsten-Leue]

It is. But that's in the module _internal, I suppose, not in the module where all is defined and MY_VAR exported ...

True, but the same is true for the classes and functions in the sample as well. And those are being documented. Is there a conceptual reason why variables are handled differently?

[Carsten-Leue]

the signature of my_function shows the internal type name not the public one (it shows def my_function (iface: pdoc_test._internal.interface.MyInterface) ‑> str) instead of def my_function(iface: pdoc_test.MyInterface) ‑> str Duplicate of #231. Not quite sure how to tackle this one.

If I change the source to this:

def my_function(iface: MyInterface) -> str:
    """My function doc



    Args:
        iface (MyInterface): interface parameter doc

    Returns:
        return value doc
    """    

    return ''

e.g. include type information into the argument explicitly (iface (MyInterface)) then I find that the type in the function declaration is being documented using the internal type, but the type in the docstring is being documented using the API level type, even including the correct hyperlink to the documentation of MyInterface. Why this difference, wouldn't it be possible to use the "docstring" behaviour both for the type in the declaration and in the docstring?

Example of the generated HTML:

<dl>
<dt id="pdoc_test.my_function"><code class="name flex">
<span>def <span class="ident">my_function</span></span>(<span>iface:&nbsp;pdoc_test._internal.interface.MyInterface) ‑&gt;&nbsp;str</span>
</code></dt>
<dd>
<div class="desc"><p>My function doc</p>
<h2 id="args">Args</h2>
<dl>
<dt><strong><code>iface</code></strong> : <code><a title="pdoc_test.MyInterface" href="#pdoc_test.MyInterface">MyInterface</a></code></dt>
<dd>interface parameter doc</dd>
</dl>
<h2 id="returns">Returns</h2>
<p>return value doc</p></div>
<details class="source">
<summary>
<span>Expand source code</span>
</summary>
<pre><code class="python hljs"><span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">my_function</span><span class="hljs-params">(iface: MyInterface)</span> -&gt; str:</span>
    <span class="hljs-string">"""My function doc



    Args:
        iface (MyInterface): interface parameter doc

    Returns:
        return value doc
    """</span>    

    <span class="hljs-keyword">return</span> <span class="hljs-string">''</span></code></pre>
</details>
</dd>
</dl>

<a title="pdoc_test.MyInterface" href="#pdoc_test.MyInterface">MyInterface</a>

[kernc]

Is there a conceptual reason why variables are handled differently?

Yeah, there's no way to access PEP224 docstrings at runtime (i.e. via __doc__), so we parse source code ASTs instead.

https://github.com/pdoc3/pdoc/blob/04960e41b9bb598b664a50078f4c233817326170/pdoc/init.py#L220-L230

Likewise with the type annotations in functions parameters vs. annotations in docstrings — two different code paths. Type strings in docstrings are not modified, just queried, thus making MyInterface resolve to the available module-local definition.

Not at all saying it's all good and perfect, but it's what we currently have.

[Carsten-Leue]

Type strings in docstrings are not modified, just queried, thus making MyInterface resolve to the available module-local definition.

I do not fully understand this comment. MyInterface defined in pdoc_test\_internal\sample.py and formally referenced as the same string in the docstring and the argument list. There is no "module-local" definition.

[kernc]

Should have rather said the module-local variable, the one present in the module, inferred from __all__, and available in this module's .doc. https://github.com/pdoc3/pdoc/blob/04960e41b9bb598b664a50078f4c233817326170/pdoc/init.py#L591-L592

Function params with linked annotations are constructed in one way: https://github.com/pdoc3/pdoc/blob/04960e41b9bb598b664a50078f4c233817326170/pdoc/init.py#L1312-L1318 and types in docstrings (any backtick-quoted identifier references, really; not only Args section types) in quite another: https://github.com/pdoc3/pdoc/blob/04960e41b9bb598b664a50078f4c233817326170/pdoc/html_helpers.py#L465-L477

Better ideas welcome.