python.apigen excluded imports option

[2bndy5]

I just tried this apigen.python ext on some docs I'm migrating to sphinx. While I think its cool to see third party dependencies (& lots of python std libs) get documented in sphinx-immaterial, I don't think its worth the extra 100+ rST files that get written. Is there a way to limit what modules are imported when generating the rST files?

To reproduce:

conf.py

extensions = [
    "sphinx_immaterial",
    "sphinx_immaterial.apidoc.python.apigen",
    "sphinx.ext.intersphinx",
]

intersphinx_mapping = {
    "python": ("https://docs.python.org/3", None),
    "requests": ("https://requests.readthedocs.io/en/latest/", None),
}

# ...


python_apigen_modules = {
    "demo": "api/",
]

demo.py (abridged)

import os
from pathlib import PurePath
from requests import Request

GITHUB_EVENT_PATH = PurePath(os.getenv("GITHUB_EVENT", ""))

class Globals:
    response_buffer: Request = Request()

demo_doc.rst

API Reference
=============

.. python-apigen-group:: Public Members
.. python-apigen-group:: Classes

In addition to docs for demo.py, this results in docs for both the requests module and pathlib modules. Below is a screenshot for the docs I'm migrating:

[jbms]

Agreed --- we definitely don't want to be generating documentation for third-party libraries. I assumed that autodoc already skipped imported stuff in some cases, but I guess not.

[jbms]

This can actually already be controlled in two ways:

• By defining __all__, but note that strangely that suppresses documenting entities without a docstring, even if they are listed in __all__ --- I think that may be a Sphinx bug, since we do specify undoc-members option to Sphinx.
• By adding a listener for the autodoc-skip-member event.

Given those two existing mechanisms, maybe we don't really need an additional config option to control this, and instead could just document those existing mechanisms.

[2bndy5]

I'd opt for the autodoc-skip-member event. Manually maintaining an __all__ attribute makes patching in updates rather cumbersome and is largely discouraged (almost as discouraged as using import *).

[jbms]

To me __all__ seems to have some advantages, though:

• you can specify it right in the module itself, and
• it also controls wildcard imports.

But it is true that maintaining it manually is annoying unless the module has a very small number of exports. Sometimes I define an @export decorator.

[mhostetter]

I've found __all__ is useful for specifying all the public / exportable objects in a private module and then using a wildcard import in __init__.py, as @jbms suggested.

# galois/__init__.py
from ._private_module import *

# galois/_private_module.py
__all__ = ["public_function"]

def set_module(module):
    def decorator(obj):
        if module is not None:
            obj.__module__ = module
        return obj
    return decorator

@set_module("galois")
def public_function(x, y):
    pass

But it is true that maintaining it manually is annoying unless the module has a very small number of exports. Sometimes I define an @export decorator.

@jbms would you mind sharing what you do in the export decorator? Above is what I do (copied from what was done in NumPy), but it only modifies the object module, not marking it for export. It seems you have a more elegant solution. It seems your usage is something like below, correct?

# galois/__init__.py
from ._private_module import *

# galois/_private_module.py
@export
def public_function(x, y):
    pass

[jbms]

See here for an example @export: https://github.com/google/neuroglancer/blob/e7ad27d4cb1061b8b80ab2b008d05bedeeb92a8c/python/neuroglancer/viewer_state.py#L46

[mhostetter]

Thanks @jbms. Inspired from your link, I generalized in this way so the export function can be defined in one place, outside the private module.

For posterity:

# galois/_helper.py
import sys

def export(obj):
    # Determine the private module that defined the object
    module = sys.modules[obj.__module__]

    # Set the object's module to the package name. This way the REPL will display the object
    # as galois.obj and not galois._private_module.obj
    obj.__module__ = "galois"

    # Append this object to the private module's "all" list
    public_members = getattr(module, "__all__", [])
    public_members.append(obj.__name__)
    setattr(module, "__all__", public_members)

    return obj

# galois/_private_module.py
from ._helper import export

@export
def public_function(x, y):
    pass