Recurse through submodules in extension modules

[robamler]

Fixes https://github.com/pdoc3/pdoc/issues/319

The original method of finding submodules by recursing through the directory structure didn't work with extension modules (aka native "C" extensions) because there is no directory structure to recurse through. This commit adds code to recurse through submodules by inspecting obj.__all__, i.e., using the same mechanism that is already used for finding variables, functions, and classes.

This is a minimalistic first sketch of a solution. It works on a native extension that I tried it on but it introduces some duplication since there are now two strategies to search for submodules. For a native module, the first strategy (searching the directory structure) will fail but the second (new) strategy (using obj.__all__) will work. For regular modules, both strategies will likely work and so the second strategy will just overwrite the results from the first strategy with identical objects. This should technically work but it seems unnecessary. I just don't feel confident enough to remove the first strategy without further guidance.

[robamler]

Thanks for the quick feedback! I've moved the new code into the if self.is_package - else block and added a test.

Studying the existing code for "normal" submodule discovery, I realized I should be more careful to avoid capturing cycles and reexports (not sure if they're possible in native extension modules though). So I made sure the new code only runs if the parent module explicitly exposes public members through the __all__ attribute. This restriction seems to be necessary because the is_from_this_module function doesn't work on submodules (inspect.getmodule(x) returns x rather than x's parent if x is a submodule). This restriction doesn't seem to be a problem for the native extension modules I tried, but I'd be happy to relax it if you have an idea how.

Further, if I:

import .foo as bar

module foo will now be be documented as foo and as bar ... :thinking:

Is that possible with native extension modules? I know too little about Python's module system to tell.

[robamler]

One problem I encountered: the HTML documentation for a native extension module now places all files in the top level directory. The contents of the files shows the correct nested hierarchy of submodules but the paths for the corresponding HTML files doesn't reflect this, so paths may clash if, e.g., a submodule has the same name as its parent module. Any idea how to fix this?