sphinx
sphinx copied to clipboard
sphinx-doc
Intersphinx does not create references to external project when using import as alias
Describe the bug
I'm creating a code docu using the docstring included in the code. In the conf.py file the intersphinx extension is included with mapping dicts to numpy, scipy, matplotlib, ...
E. g. When I import numpy as np or from typing import List, the external references do not exists in the build html docu as expected. It only works for standard python types.
I could not find anything in the Sphinx docu about this. Is this a bug or how must Sphinx be configured here?
How to Reproduce
$ git clone https://github.com/.../some_project
$ cd some_project
$ pip install -r requirements.txt
$ cd docs
$ make html SPHINXOPTS="-D language=de"
$ # open _build/html/index and see bla bla
Expected behavior
When I use absolute import like import numpy, import typing, the external docu is references in the html docu.
I would expected that the external docu references are created also for the import aliases.
Your project
--
Screenshots
No response
OS
Win
Python version
3.8
Sphinx version
1.44
Sphinx extensions
"sphinx.ext.autodoc", "sphinx.ext.viewcode", "sphinx.ext.todo", "sphinx.addnodes", "sphinx.ext.napoleon", "sphinx_autodoc_typehints", "sphinx.ext.intersphinx", "sphinx.ext.mathjax", "sphinx.ext.viewcode",
Extra tools
No response
Additional context
No response
We might have to implement it ourselves via a PR.
This looks like the code for the extension: https://github.com/sphinx-doc/sphinx/blob/5.x/sphinx/ext/intersphinx.py
I had this problem a while ago, which was why I subscribed to this thread. I found using type hints solves this problem. I don't know if it's worth it to you to switch, but... Also with proper type hints you can use the autodoc-process-signature callback to further modify the presentation of the type hint signatures.
Types in docstring
None of these are "clickable" except float.
def func_2(x, y):
"""
This is the docstring of func_2.
Parameters
----------
x: np.ndarray
The first argument.
y: np.ndarray
The second argument.
Returns
-------
List[float]
The return value.
"""
pass
Types as type hints
All of these are "clickable".
def func(x: np.ndarray, y: np.ndarray) -> List[float]:
"""
This is the docstring of func.
Parameters
----------
x
The first argument.
y
The second argument.
Returns
-------
The return value.
"""
pass
Conf.py
intersphinx_mapping = {
"python": ("https://docs.python.org/3", None),
"numpy": ("https://numpy.org/doc/stable/", None),
}
autodoc_typehints = "both"
# autodoc_typehints_format = "short" # This is handy too
You can optionally add the autodoc-process-signature callback if you'd like.
def autodoc_process_signature(app, what, name, obj, options, signature, return_annotation):
# Do something
return signature, return_annotation
def setup(app):
app.connect("autodoc-process-signature", autodoc_process_signature)
OK I think I see the difference now. When you use a docstring generator as you have it in some python IDEs, it typically generates the docstring with the type included as it is defined in the docstring conventions: Numpy Docstring Format.
def func(self, val: np.ndarray) -> List[float]:
"""docstring of func
Args:
val (np.ndarray): Input argument
Returns:
List[float]: return value
"""
return
--> When you delete all typehints in the docstring, it works.
def func(self, val: np.ndarray) -> List[float]:
"""docstring of func
Args:
val: Input argument
Returns:
return value
"""
return
Does that mean the link to the numpy doc is overwritten?
@photoniker how does this function render?
def func(self, val: np.ndarray) -> List[float]:
"""docstring of func
Args:
val (int): Input argument
Returns:
str: return value
"""
return
Seems to work with python build-in types
That's a bummer. It appears that the docstring types supersede the function's actual annotations. I would've hoped for the opposite.
There's clearly an issue (as you reported) with how Napoleon processes types from the docstring. The workaround I've used is to simply remove types from the docstring and use type hints.
Here's another workaround... You could attach to autodoc-process-docstring and find "np." and replace it with "numpy.". That would then add a link to numpy.ndarray in your docs.
Here's another workaround... You could attach to
autodoc-process-docstringand find"np."and replace it with"numpy.". That would then add a link tonumpy.ndarrayin your docs.
Do you have an example for that? I'd like to ensure that nn.Module (torch.nn.Module) and F.interpolate (torch.nn.functional.interpolate) redirect to the appropriate PyTorch docs, without having to type out the full path.
Below is an example with OPs function. The same trick can be applied to nn.Module to torch.nn.Module.
Here's the full example project. foo2-example.zip
Before
Notice np.ndarray isn't linked.
def func(self, val: np.ndarray) -> List[float]:
"""docstring of func
Args:
val (np.ndarray): Input argument
Returns:
List[float]: return value
"""
pass
After
Notice np.ndarray was converted to numpy.ndarray and is now linked.
#conf.py
def autodoc_process_docstring(app, what, name, obj, options, lines):
for i in range(len(lines)):
lines[i] = lines[i].replace("np.", "numpy.")
lines[i] = lines[i].replace("List[", "~typing.List[")
def setup(app):
app.connect("autodoc-process-docstring", autodoc_process_docstring)
Shorten links
If you add autodoc_typehints = "short" to conf.py and do things like lines[i] = lines[i].replace("np.", "~numpy.") the hyperlinks will be shortened like below.
@mhostetter Thank you!
Is there an easy way to do something similar in the doc string and parameter descriptions as well?
Like making this:
"""
:func:`F.interpolate`
"""
Equivalent to this:
"""
:func:`torch.nn.functional.interpolate`
"""
Or does your solution work on the entire doc string? If it works on the entire doc string, how can I only target the parameters?
Yes, the entire docstring.
def func(self, val: np.ndarray) -> List[float]:
"""docstring of func
Here is a detailed docstring. You know :func:`F.interpolate` is a cool function.
Args:
val (np.ndarray): Input argument
Returns:
List[float]: return value
"""
pass
# conf.py
intersphinx_mapping = {
"python": ("https://docs.python.org/3", None),
"numpy": ("https://numpy.org/doc/stable/", None),
"torch": ("https://pytorch.org/docs/master/", None),
}
def autodoc_process_docstring(app, what, name, obj, options, lines):
print(name)
print("before:", lines)
for i in range(len(lines)):
lines[i] = lines[i].replace("np.", "numpy.")
# lines[i] = lines[i].replace("np.", "~numpy.") # For shorter links
lines[i] = lines[i].replace("F.", "torch.nn.functional.")
lines[i] = lines[i].replace("List[", "~typing.List[")
print("after:", lines)
def setup(app):
app.connect("autodoc-process-docstring", autodoc_process_docstring)
If you print the lines, you see exactly what is getting processed. If you only want to modify the types, you need to look for :type or :rtype at the beginning of a line.
reading sources... [ 50%] foo.func
foo.func
before: ['docstring of func', '', 'Here is a detailed docstring. You know :func:`F.interpolate` is a cool function.', '', ':param val: Input argument', ':type val: np.ndarray', '', ':returns: return value', ':rtype: List[float]', '']
after: ['docstring of func', '', 'Here is a detailed docstring. You know :func:`torch.nn.functional.interpolate` is a cool function.', '', ':param val: Input argument', ':type val: numpy.ndarray', '', ':returns: return value', ':rtype: ~typing.List[float]', '']
And it renders as below, with everything hyperlinked.
@mhostetter Ah, okay. Thanks for the help!
from typing import Callable
import torch
import torch.nn as nn
def test_func(x: torch.Tensor, test_module: nn.Module, test_fn: Callable) -> torch.Tensor:
"""
Description of function.
Args:
x (torch.Tensor): The input tensor.
test_module (nn.Module): The nn.Module instance to run x through.
test_fn (Callable): A callable class or function to pass x through.
Returns:
x (torch.Tensor): The output tensor.
"""
return test_module(test_fn(x))
My code uses doc strings like the one above, so I think that I could do something like this then (including increasing the specificity by targeting values inside / outside brackets, potentially with regex):
def autodoc_process_docstring(app, what, name, obj, options, lines):
parameters_docs = False:
for i in range(len(lines)):
if "Args:" in lines[i]:
parameters_docs = True
if not parameters_docs:
continue
lines[i] = lines[i].replace("nn.Module", "torch.nn.Module")
lines[i] = lines[i].replace("Callable[", "~typing.Callable[")
def setup(app):
app.connect("autodoc-process-docstring", autodoc_process_docstring)
You can't filter on "Args:". Sphinx converts the Google-style docstrings Args: and Returns: sections into rST syntax, which makes it even easier to filter. See my example in my previous post. You just need to look for :type or :rtype at the beginning of a line.
reading sources... [ 50%] foo.func
foo.func
before: ['docstring of func', '', 'Here is a detailed docstring. You know :func:`F.interpolate` is a cool function.', '', ':param val: Input argument', ':type val: np.ndarray', '', ':returns: return value', ':rtype: List[float]', '']
after: ['docstring of func', '', 'Here is a detailed docstring. You know :func:`torch.nn.functional.interpolate` is a cool function.', '', ':param val: Input argument', ':type val: numpy.ndarray', '', ':returns: return value', ':rtype: ~typing.List[float]', '']
You instead would want something like this:
for i in range(len(lines)):
if not (lines[i].startswith(":type") or lines[i].startswith(":rtype")):
continue
# Do stuff
@mhostetter You don't need to use autodoc_typehints = "short", and you can change the link to say whatever you want.
For example:
lines[i] = lines[i].replace("Callable", ":obj:`Callable <typing.Callable>`")
lines[i] = lines[i].replace("F.interpolate", ":obj:`F.interpolate <torch.nn.functional.interpolate>`")
I've discovered a new issue. When manually adding :obj: or :class: to the docstring types, they are no longer formatted as italics. Yet somehow Sphinx is able to implicitly hyperlink some of the types and italicize them when they are not manually specified with :obj: or :class:.
Is there anyway to fix this issue?
Edit:
# conf.py
extensions = [
"sphinx.ext.autodoc",
"sphinx.ext.napoleon",
"sphinx.ext.intersphinx",
]
intersphinx_mapping = {
"python": ("https://docs.python.org/3", None),
"pytorch": ("https://pytorch.org/docs/stable", None),
}
def my_func(show_progress: bool = True) -> bool:
"""
Args:
show_progress (bool, optional): Displays the progress of computation.
"""
return show_progress
This is an example of HTML code generated by Sphinx without explicitly defining the types:
<li><p><strong>show_progress</strong> (<a class="reference external" href="https://docs.python.org/3/library/functions.html#bool" title="(in Python v3.10)"><em>bool</em></a><em>, </em><em>optional</em>) – Displays the progress of computation.
And this is the same HTML, except for the bool variable being explicitly wrapped in :class:bool``:
<li><p><strong>show_progress</strong> (<a class="reference external" href="https://docs.python.org/3/library/functions.html#bool" title="(in Python v3.10)"><code class="xref py py-class docutils literal notranslate"><span class="pre">bool</span></code></a>, optional) – Displays the progress of computation.
The <em> and </em> tags are missing in the second example for some reason.
Would have been nice if my PR had been merged two years ago, then we wouldn’t need workarounds: https://github.com/sphinx-doc/sphinx/pull/9039
Hello,
Not sure if my problem has the same root as yours, so before opening a duplicate issue I wanted to check.
In my project, I am using the traitlets library and basically aliasing their types in my package.
In their code, they do import typing as t and they call t.Any, t.Dict, etc...
then when I compile my docs I see warnings as
WARNING: py:class reference target not found: t.Any
Is this the same problem? Sphinx not able to "intersphinx" aliased modules?
