Non-Python Type for pybind11 Module

[Queuecumber]

Expected Behavior

When I try to document my codebase which contains a pybind11 module, pdoc seems to be trying to generate python code with a c++ function signature leading to the error "Name 'at' is not defined". The full error is

Traceback (most recent call last):
  File "/home/max/code/torchjpeg/.venv/bin/pdoc", line 11, in <module>
    load_entry_point('pdoc3==0.8.4', 'console_scripts', 'pdoc')()
  File "/home/max/code/torchjpeg/.venv/lib/python3.7/site-packages/pdoc/cli.py", line 501, in main
    recursive_write_files(module, ext='.html', **template_config)
  File "/home/max/code/torchjpeg/.venv/lib/python3.7/site-packages/pdoc/cli.py", line 340, in recursive_write_files
    recursive_write_files(submodule, ext=ext, **kwargs)
  File "/home/max/code/torchjpeg/.venv/lib/python3.7/site-packages/pdoc/cli.py", line 340, in recursive_write_files
    recursive_write_files(submodule, ext=ext, **kwargs)
  File "/home/max/code/torchjpeg/.venv/lib/python3.7/site-packages/pdoc/cli.py", line 328, in recursive_write_files
    w.write(m.html(**kwargs))
  File "/home/max/code/torchjpeg/.venv/lib/python3.7/site-packages/pdoc/__init__.py", line 789, in html
    html = _render_template('/html.mako', module=self, **kwargs)
  File "/home/max/code/torchjpeg/.venv/lib/python3.7/site-packages/pdoc/__init__.py", line 133, in _render_template
    return t.render(**config).strip()
  File "/home/max/code/torchjpeg/.venv/lib/python3.7/site-packages/mako/template.py", line 476, in render
    return runtime._render(self, self.callable_, args, data)
  File "/home/max/code/torchjpeg/.venv/lib/python3.7/site-packages/mako/runtime.py", line 883, in _render
    **_kwargs_for_callable(callable_, data)
  File "/home/max/code/torchjpeg/.venv/lib/python3.7/site-packages/mako/runtime.py", line 920, in _render_context
    _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
  File "/home/max/code/torchjpeg/.venv/lib/python3.7/site-packages/mako/runtime.py", line 947, in _exec_template
    callable_(context, *args, **kwargs)
  File "_html_mako", line 143, in render_body
  File "_html_mako", line 34, in show_module
  File "_html_mako", line 414, in render_show_module
  File "_html_mako", line 328, in show_func
  File "/home/max/code/torchjpeg/.venv/lib/python3.7/site-packages/pdoc/__init__.py", line 1240, in params
    return self._params(self, annotate=annotate, link=link, module=self.module)
  File "/home/max/code/torchjpeg/.venv/lib/python3.7/site-packages/pdoc/__init__.py", line 1247, in _params
    signature = Function._signature_from_string(doc_obj)
  File "/home/max/code/torchjpeg/.venv/lib/python3.7/site-packages/pdoc/__init__.py", line 1364, in _signature_from_string
    exec('def {}: pass'.format(string), _globals, _locals)
  File "<string>", line 1, in <module>
NameError: name 'at' is not defined

When I print the string variable on that line I can see it is containing:

read_coefficients(path: str) -> Tuple[at::Tensor, at::Tensor, at::Tensor, Optional[at::Tensor]]

but pdoc is trying to exec that as a python function. at:: is a C++ namespace which causes the problem.

Actual Behavior

It works.

Steps to Reproduce

1. Clone https://gitlab.com/Queuecumber/torchjpeg
2. pdoc --html --output-dir . --config latex_math=True --force torchjpeg (arguments are likely irrelevant but that's what I use)
3. Observe error

Additional info

• pdoc version: 0.8.4

[kernc]

pdoc is trying to exec that as a python function. at:: is a C++ namespace which causes the problem.

Yeah, we try to prepopulate custom _globals from the parent module: https://github.com/pdoc3/pdoc/blob/04960e41b9bb598b664a50078f4c233817326170/pdoc/init.py#L1353-L1367 But I guess at is just not in there. :disappointed: (If it were, translating at::Tensor → at.Tensor might work.)

We likely should also except NameError and, in lack of better ideas, return None. Investigation/PR welcome.

[Queuecumber]

if it were, translating at::Tensor → at.Tensor might work

The correct name is actually torch.Tensor for the python type, I have no idea why it's giving that back as the python function signature.

[kernc]

at seems to be the namespace that actually defines the class. Issue https://github.com/pytorch/extension-cpp/issues/31. Pybind11 is just exact.

[kernc]

except NameError and, in lack of better ideas, return None

Here's one better idea:

1. strip type annotations,
2. issue a warning,
3. try exec again.

This way we at least get argument names.

[Queuecumber]

That might be good for avoiding the crash, but as a workaround I'm trying to define my docstring manually, e.g. I disable function signature docstrings from pybind11 and rebuild, then manually define my docstring as

read_coefficients(path: str) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor, Optional[torch.Tensor]]

but now it's saying module torch is not defined. So I think I'm close?

[kernc]

I'm trying to define my docstring manually

This sounds like a pain to maintain. :grimacing: Prefer some patching like:

read_coefficients.__doc__ = read_coefficients.__doc__.replace('at::', 'torch.')

if it can be made to work.

But first see what _globals contain after 1356 (i.e. in the module read_coefficients() is defined in): https://github.com/pdoc3/pdoc/blob/04960e41b9bb598b664a50078f4c233817326170/pdoc/init.py#L1353-L1356 That's what's available to reference in exec.