Support for autosummary :recursive: documentation of entire API

[Yoshanuikabundi]

Hi! Thanks for this package, I love the results it produces!

I've been experimenting with ways to fully automate the generation of API reference documentation from source code and docstrings, a la rustdoc. I know you're aware of this, but there's no easy way to include autodoc_pydantic output in a fully automated API reference.

The following works, of course:

.. autosummary::
   :toctree: _autosummary

   module_with_lots_of_stuff.AutoSummaryModel
   module_with_lots_of_stuff.AutoSummarySettings
   module_with_lots_of_stuff.another_module

But I would like the following:

.. autosummary::
   :toctree: _autosummary
   :recursive:

   module_with_lots_of_stuff

I understand that the reason this doesn't work is that Sphinx doesn't let you expose new variables to the autosummary templates (as you discuss in #11). I've worked around this to some extent in the project I'm working on by adding an autosummary table with all the members not documented elsewhere in the template to the end of the module template, but it's not very good and we probably won't end up using it.

My suggestion would be to workaround the bug in Sphinx by adding a Jinja2 filter along the lines of "keep_pydantic_models" that takes an iterator of names of objects and filters out those that aren't models. So a block like the following could be added to an Autosummary template:

{% set models = members | keep_pydantic_models(module=fullname) | list %}

{% if models %}
Models
---------

{% for item in models %}
.. autopydantic_model:: item
{% endfor %}

I think you should be able to add such a filter to all templates, but I'm not super familiar. Sorry if this isn't helpful.

Thanks!

[StephenHogg]

I'd like to add my voice to this, have just run into this exact problem today and would love it if this would fit into the recursive command along with everything else. For now I'm going to just ignore the pydantic files as I don't quite understand the workaround, unfortunately. If anyone (@Yoshanuikabundi ?) could explain it, I'd be very grateful though.

[mansenfranzen]

@Yoshanuikabundi Thanks for your detailed report and your thoughts on this issue. I know it's a pity that a fully automated API documentation does not yet work properly with autodoc_pydantic.

Unfortunately I haven't had enough time to take a deep dive on this issue, yet. But judging from a high level perspective, I assume effort is best invested in extending sphinx.ext.autosummary instead of finding a workaround in autodoc_pydantic. We might find a workaround but it is going to be hacky I guess (but please prove me wrong otherwise - I'm happy to take a look at any implementation or PR).

I hopefully will come back to this soon. Providing a PR to sphinx with the required functionality also shouldn't be too difficult.

[StephenHogg]

I think I've narrowed it down. Try this with v1.3.2-a.1, you should see an error:

conf.py

import os
import sys

sys.path.insert(0, os.path.abspath('.'))

project = 'Test'
copyright = 'Test'
author = 'Test'

extensions = [
    'sphinx.ext.autodoc',
    'sphinxcontrib.autodoc_pydantic',
    'sphinx.ext.napoleon',
    'sphinx.ext.autosummary'
]

autosummary_generate = True  # Turn on sphinx.ext.autosummary
html_theme = "sphinx_rtd_theme"

index.rst

.. Test documentation master file, created by
   sphinx-quickstart on Sat Aug  7 16:28:18 2021.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

Welcome to Test's documentation!
================================

.. autosummary:: test
    :toctree: _autosummary
    :recursive:


Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

test.py

from pydantic import BaseModel


class TestClass:
    """Test

    Attributes:
        model (TestModel): Model
    """

    def __init__(self):
        self.model = TestModel()


class TestModel(BaseModel):
    pass

Thanks for making this package, by the way!

[StephenHogg]

The error it generates for me is as follows:

shogg@DESKTOP:~/git/test$ make html
Running Sphinx v4.1.2
making output directory... done
[autosummary] generating autosummary for: index.rst
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: [new config] 1 added, 0 changed, 0 removed
reading sources... [100%] index                                                                                                         
/home/shogg/git/test/index.rst.rst:9: WARNING: autosummary: stub file not found 'test'. Check your autosummary_generate setting.
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index                                                                                                          
generating indices... genindex done
writing additional pages... search done
copying static files... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 1 warning.

The HTML pages are in _build/html.

[mansenfranzen]

@StephenHogg I think your index.rst is not correct. Try:

.. autosummary::
   :toctree: _autosummary
   :recursive:

   test

[mansenfranzen]

@Yoshanuikabundi I took a closer look at your proposal.

It is a good idea to extend the standard autosummary template to call the custom function keep_pydantic_models from within the jinja template that filters only pydantic models and then the jinja template loops through all models. The function itself is not a big deal.

However, providing the function to the jinja template namespace is currently not possible because the jinja template environment (which needs to be accessed to register a function) is created here:

https://github.com/sphinx-doc/sphinx/blob/6ac326e019db949c2c8d58f523c2534be36d4e62/sphinx/ext/autosummary/generate.py#L117-L136

There is no proper way to access the jinja environment without monkeypatching this class :-(.

A dirty fix to allow autosummary to pick up pydantic models is to change the following lines:

https://github.com/sphinx-doc/sphinx/blob/6ac326e019db949c2c8d58f523c2534be36d4e62/sphinx/ext/autosummary/generate.py#L325-L326

to:

        ns['classes'], ns['all_classes'] = \
            get_members(obj, {'class', 'pydantic_model', 'pydantic_settings'}, imported=imported_members)

This change will make autosummary respect pydantic models/settings autodocumenters to be added under the classes section in the stub files.

Of course, both solutions are just workarounds. I'm going to provide a PR upstream to sphinx to address https://github.com/sphinx-doc/sphinx/issues/6264 as this should not be too complicated.

The best solution should make the standard templates extensible without overwriting them completely. Instead, it should be possible to add custom sections to them template while also modifying the jinja namespace template as you've proposed. But this is more complicated.

[Yoshanuikabundi]

@mansenfranzen Thanks for the update! I think I completely agree with you. I thought the Jinja2 environment was extensible because autoapi provides a very clean API to modify it, but on closer inspection it looks like they can do this because they use their own templates rather than extending those of Autosummary. Sorry about that!

I definitely think fixing that upstream issue is a much better solution and I'm very grateful that you're taking a shot at it! Let me know if you need another set of eyes on it.

@StephenHogg Sorry for the confusion! My workaround was a proposal, not something I had working.

[utf]

Hi, I'm just wondering if there's any update on this issue?

[mansenfranzen]

Unfortunately not - I haven't gotten any response in the upstream issue https://github.com/sphinx-doc/sphinx/issues/6264. I just added a friendly reminder.

[Yoshanuikabundi]

Hi all! I've come up with a template that works around this issue for the time being. Adding the :toctree: option to the autosummary directives in the autosummary/module.rst template causes functions, classes etc to be documented on their own pages. Then, simply looping over all the members for the current module and excluding anything documented elsewhere (or with a leading underscore) gets you the classes, including pydantic classes. Since the pydantic classes each get their own page, autodoc-pydantic can then take over. Works nicely.

Minimal template (goes in docs/_templates/autosummary/module.rst):

{% extends "!autosummary/module.rst" %}
{% block classes %}

{% set types = [] %}
{% for item in members %}
   {% if not item.startswith('_') and not (item in functions or item in attributes or item in exceptions) %}
      {% set _ = types.append(item) %}
   {% endif %}
{%- endfor %}

{% if types %}
.. rubric:: {{ _('Classes') }}

   .. autosummary::
      :toctree:
      :nosignatures:
   {% for item in types %}
      {{ item }}
   {%- endfor %}

{% endif %}
{% endblock %}

More realistically you might want to also add :toctree: to the functions, exceptions, and module attributes autosummaries as well.

[mansenfranzen]

Thanks for sharing - that's really great! This information could be very useful for others, too. It would be a great candidate for the documentations FAQ section. If you feel motivated and you have enough time, you could a PR to extend the FAQ section with your workaround. If not, no worries - I can also do it.

[Yoshanuikabundi]

I'd love to! But it might take me a while to get to, so if it's a priority for you don't feel like you have to wait for me.

[mansenfranzen]

@Yoshanuikabundi No need to hurry - take your time.

[iwyrkore]

Thanks for @Yoshanuikabundi 's ideas, I implemented a custom autosummary extension. (I was having many issues with other non-pydantic types falling into the "types" list in his template and I wanted more control over the pydantic behavior.)

I simply copied the sphinx.ext.autosummary code to a local folder docs/ext/pydantic_autosummary (my rst/md files are in docs/source)

Edited __init__.py and generate.py to fix imports and system_templates_path to point to my extension templates folder.

Then in generate_autosummary_content() (generate.py around line 341) added:

        ns['pydantic_models'], ns['all_pydantic_models'] = \
            get_members(obj, {'pydantic_model'}, imported=imported_members)
        ns['pydantic_settings'], ns['all_pydantic_settings'] = \
            get_members(obj, {'pydantic_settings'}, imported=imported_members)

In the module.rst template, just before the {% block modules %} (at the same nesting level, NOT nested inside .. automodule:: {{ fullname }} which doesn't work), add the following:

{% block pydantic_models %}
{% if pydantic_models %}
.. rubric:: {{ _('Models') }}

.. autosummary::
{% for item in pydantic_models %}
   {{ item }}
{%- endfor %}
{% endif %}
{% endblock %}

{% block pydantic_settings %}
{% if pydantic_settings %}
.. rubric:: {{ _('Settings') }}

.. autosummary::
{% for item in pydantic_settings %}
   {{ item }}
{%- endfor %}
{% endif %}
{% endblock %}

Then in conf.py replace 'sphinx.ext.autosummary' with 'pydantic_autosummary' (and make sure the pydantic_autosummary folder is in your sys.path)

All the standard autosummary settings will still work with no changes.

I use this with custom templates (optional) to do toctree and custom pydantic settings, so in my module template:

{% block pydantic_models %}
{% if pydantic_models %}
.. rubric:: {{ _('Models') }}

.. autosummary::
   :toctree:
   :template: custom-model-template.rst
   :nosignatures:
{% for item in pydantic_models %}
   {{ item }}
{%- endfor %}
{% endif %}
{% endblock %}

And then custom-model-template.rst file has the following which shows inherited attributes (ie. model superclasses) but not from BaseModel.

.. autopydantic_model:: {{ objname }}
   :members:
   :show-inheritance:
   :inherited-members: BaseModel
   :special-members: __call__, __add__, __mul__

   {% block methods %}
   {% if methods %}
   .. rubric:: {{ _('Methods') }}

   .. autosummary::
      :nosignatures:
   {% for item in methods %}
      {%- if not item.startswith('_') %}
      ~{{ name }}.{{ item }}
      {%- endif -%}
   {%- endfor %}
   {% endif %}
   {% endblock %}

   {% block attributes %}
   {% if attributes %}
   .. rubric:: {{ _('Attributes') }}

   .. autosummary::
   {% for item in attributes %}
      ~{{ name }}.{{ item }}
   {%- endfor %}
   {% endif %}
   {% endblock %}

Hope this helps someone.

[mansenfranzen]

@iwyrkore Thanks for sharing your solution! I'm sure it will be helpful for others having the same issue. I might add a new section to the user's FAQ docs soonish linking to your solution, too (feel free to create PR yourself if you want to ;-)).

[mansenfranzen]

@all-contributors please add @iwyrkore for code

[allcontributors[bot]]

@mansenfranzen

I've put up a pull request to add @iwyrkore! :tada:

[ThomasYAD]

@iwyrkore Specifically how and where did you change the init and generate imports? I can't seem to get this to work, even though I added docs/ext/pydantic_autosummary to my path in conf.py. When making the html, it still defaults to the lib/site-packages/sphinx/ext/autosummary package instead of using my custom defined one :(