Problem conditionally defined/generated objects?

[renefritze]

It seems conditionally defined objects are not handled correctly.

Example file

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/sphinx/cmd/build.py", line 276, in build_main
    app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
  File "/usr/local/lib/python3.8/site-packages/sphinx/application.py", line 278, in __init__
    self._init_builder()
  File "/usr/local/lib/python3.8/site-packages/sphinx/application.py", line 337, in _init_builder
    self.events.emit('builder-inited')
  File "/usr/local/lib/python3.8/site-packages/sphinx/events.py", line 110, in emit
    results.append(listener.handler(self.app, *args))
  File "/usr/local/lib/python3.8/site-packages/autoapi/extension.py", line 167, in run_autoapi
    sphinx_mapper_obj.map(options=app.config.autoapi_options)
  File "/usr/local/lib/python3.8/site-packages/autoapi/mappers/python/mapper.py", line 314, in map
    self._resolve_placeholders()
  File "/usr/local/lib/python3.8/site-packages/autoapi/mappers/python/mapper.py", line 311, in _resolve_placeholders
    _resolve_module_placeholders(modules, module_name, visit_path, resolved)
  File "/usr/local/lib/python3.8/site-packages/autoapi/mappers/python/mapper.py", line 123, in _resolve_module_placeholders
    _resolve_module_placeholders(modules, imported_from, visit_path, resolved)
  File "/usr/local/lib/python3.8/site-packages/autoapi/mappers/python/mapper.py", line 123, in _resolve_module_placeholders
    _resolve_module_placeholders(modules, imported_from, visit_path, resolved)
  File "/usr/local/lib/python3.8/site-packages/autoapi/mappers/python/mapper.py", line 123, in _resolve_module_placeholders
    _resolve_module_placeholders(modules, imported_from, visit_path, resolved)
  [Previous line repeated 1 more time]
  File "/usr/local/lib/python3.8/site-packages/autoapi/mappers/python/mapper.py", line 146, in _resolve_module_placeholders
    LOGGER.warning(msg)
  File "/usr/local/lib/python3.8/logging/__init__.py", line 1800, in warning
    self.log(WARNING, msg, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sphinx/util/logging.py", line 126, in log
    super().log(level, msg, *args, **kwargs)
  File "/usr/local/lib/python3.8/logging/__init__.py", line 1832, in log
    self.logger.log(level, msg, *args, **kwargs)
  File "/usr/local/lib/python3.8/logging/__init__.py", line 1500, in log
    self._log(level, msg, args, **kwargs)
  File "/usr/local/lib/python3.8/logging/__init__.py", line 1577, in _log
    self.handle(record)
  File "/usr/local/lib/python3.8/logging/__init__.py", line 1587, in handle
    self.callHandlers(record)
  File "/usr/local/lib/python3.8/logging/__init__.py", line 1649, in callHandlers
    hdlr.handle(record)
  File "/usr/local/lib/python3.8/logging/__init__.py", line 946, in handle
    rv = self.filter(record)
  File "/usr/local/lib/python3.8/logging/__init__.py", line 807, in filter
    result = f.filter(record)
  File "/usr/local/lib/python3.8/site-packages/sphinx/util/logging.py", line 422, in filter
    raise exc
sphinx.errors.SphinxWarning: Cannot resolve import of pymor.core.pickle.dumps in pymor.core.cache

On a hunch, I've tried to avoid this with setting a astroid.MANAGER.register_failed_import_hook to filter, but that didn't do it.

It would be nice if we could avoid this warnings, as we're currently treating warnings as errors.

Full list of warnings also suggests a similar problem for files where we have a cython extension module and a dummy import file:

WARNING: Cannot resolve import of pymor.core.pickle.dumps in pymor.basic
WARNING: Cannot resolve import of pymor.core.pickle.load in pymor.basic
WARNING: Cannot resolve import of pymor.core.pickle.loads in pymor.basic
WARNING: Cannot resolve import of pymor.discretizers.builtin.relations.inverse_relation in pymor.discretizers.builtin.grids.interfaces
WARNING: Cannot resolve import of pymor.discretizers.builtin.relations.inverse_relation in pymor.discretizers.builtin.grids.constructions
WARNING: Cannot resolve import of pymor.discretizers.builtin.inplace.iadd_masked in pymor.discretizers.builtin.fv
WARNING: Cannot resolve import of pymor.discretizers.builtin.inplace.isub_masked in pymor.discretizers.builtin.fv
WARNING: Cannot resolve import of pymor.discretizers.builtin.grids._unstructured.compute_edges in pymor.discretizers.builtin.grids.unstructured
WARNING: Cannot resolve import of pymor.discretizers.builtin.gui.gl.GLPatchWidget in pymor.discretizers.builtin.gui.qt
WARNING: Cannot resolve import of pymor.discretizers.builtin.gui.gl.ColorBarWidget in pymor.discretizers.builtin.gui.qt
WARNING: Cannot resolve import of pymor.discretizers.builtin.gui.matplotlib.MatplotlibPatchWidget in pymor.discretizers.builtin.gui.qt
WARNING: Cannot resolve import of pymor.core.pickle.dumps in pymor.tools.mpi
WARNING: Cannot resolve import of pymor.core.pickle.loads in pymor.tools.mpi

Dummy import file and associated cython source

[AWhetter]

Do you have a branch of pymor that's using autoapi that I can test with? I don't think the conditionals are the problem but I can't tell what the problem is from this traceback unfortunately.

AutoAPI isn't capable of parsing cython modules so you will have to supply stub files for those cython modules so that AutoAPI can parse those instead.

[renefritze]

Do you have a branch of pymor that's using autoapi that I can test with? I don't think the conditionals are the problem but I can't tell what the problem is from this traceback unfortunately.

Sure thing:

git clone --branch=docs_sphinx-autoapi_issue https://github.com/pymor/pymor 
cd pymor

pip install -e .[docs] && make docs
# or
make docker_docs

AutoAPI isn't capable of parsing cython modules so you will have to supply stub files for those cython modules so that AutoAPI can parse those instead.

Alright, adding stub functions to the cython dummy import files is simple enough.

Thank you for looking into this!

[renefritze]

I've tried to understand what the problem might be by adding tests in astroid. I think I replicated the conditional definition setup at a very basic level here. I don't really understand the test setup fundamentals, but the result seems surprising to me. I had expected both cases find the same number of function definitions. That doesn't feel to relate to my problem though, since there's at least one def found either way.

[renefritze]

I've further tracked this down now and to me it looks like there's a problem with astroid and functools.partial in my setup.

This commit basically adds the whole source file from pyMOR to the astroid testsuite. The test fails: https://travis-ci.com/github/renefritze/astroid/jobs/487452325#L353 One of the hits for dump is Uninferable. AFAICT this is why in using autosphinx in the pyMOR branch I get misses for dumps,load,loads but not dump itself. The parser bails after the first Uninferable.

The culprit seems to be the dump = partial(...) assignment. If make those strings instead, the whole test suite passes.

Edit: Further minimized the failure test case: https://github.com/renefritze/astroid/commit/83d04223346dc49834efb993fb83b1749d3348a3 and discovered switching the partial from pickle to something else also fixes the problem: https://github.com/renefritze/astroid/commit/307f3acbfaa8bac3e7e4f1f0d0f38a965e54c53d

[renefritze]

So it looks like the referenced issue in astroid has gone away/was fixed.

I do have a stripped down test case now where if I have multiple classes guarded by an if True, only the first class is documented. If I guard each class individually, all are documented.

Compare the changes

"broken" HTML output "correct" HTML output

So my current guess is that autoapi for some reason stops after adding the first class object on ast traversal down a module node, if that's under a conditional. But I haven't had time to dig into the autoapi internals yet.