pdoc
pdoc copied to clipboard
Published
20 hours ago •
mitmproxy
mitmproxy
Modify import behavior to support documentation for micropython
Problem Description
I am using micropython in my project and it is fairly easy and straightforward to add docstrings.
However, it is not possible to use pdoc in all cases, as sometimes the source code requires imports that pdoc cannot resolve, since they might be C modules compiled into the micropython binary.
Proposal
To circumvent the stated problem, it would be nice if one could modify the behavior of pdoc to ignore certain imports and unknown modules.
Possibly in a similar way, which linters use via comments:
import unknown_module # pdoc: ignore
Alternatives
If the behavior can't be circumvented, it would be nice if one could "feed" pdoc with the relevant docstrings extracted by micropython. This could simply be in the form of some JSON, YAML or similar file format. Such a file could easily be generated by micropython, in a desired format.
The information read from the file could then be parsed into pdoc to generate the relevant documentation.
Additional context
I was not able to use pdoc on my micropython project based on LVGL, due to the following errors:
Console output
$ pdoc src
pdoc server ready at http://localhost:8080
Warn: Couldn't import src.design_parser:
Traceback (most recent call last):
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 218, in load_module
return importlib.import_module(module)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/src/design_parser.py", line 1, in <module>
from display_driver_utils import driver
ModuleNotFoundError: No module named 'display_driver_utils'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 480, in submodules
module = Module.from_name(mod.name)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 404, in from_name
return cls(extract.load_module(name))
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 220, in load_module
raise RuntimeError(f"Error importing {module}") from e
RuntimeError: Error importing src.design_parser
(/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py:482)
Warn: Couldn't import src.main:
Traceback (most recent call last):
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 218, in load_module
return importlib.import_module(module)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/src/main.py", line 1, in <module>
import cli
ModuleNotFoundError: No module named 'cli'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 480, in submodules
module = Module.from_name(mod.name)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 404, in from_name
return cls(extract.load_module(name))
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 220, in load_module
raise RuntimeError(f"Error importing {module}") from e
RuntimeError: Error importing src.main
(/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py:482)
Warn: Couldn't import src.random_ui:
Traceback (most recent call last):
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 218, in load_module
return importlib.import_module(module)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/src/random_ui.py", line 1, in <module>
from display_driver_utils import driver
ModuleNotFoundError: No module named 'display_driver_utils'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 480, in submodules
module = Module.from_name(mod.name)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 404, in from_name
return cls(extract.load_module(name))
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 220, in load_module
raise RuntimeError(f"Error importing {module}") from e
RuntimeError: Error importing src.random_ui
(/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py:482)
Warn: Couldn't import src.screenshot:
Traceback (most recent call last):
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 218, in load_module
return importlib.import_module(module)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/src/screenshot.py", line 1, in <module>
import lvgl as lv
ModuleNotFoundError: No module named 'lvgl'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 480, in submodules
module = Module.from_name(mod.name)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 404, in from_name
return cls(extract.load_module(name))
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 220, in load_module
raise RuntimeError(f"Error importing {module}") from e
RuntimeError: Error importing src.screenshot
(/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py:482)
Warn: Couldn't import src.screenshot_v2:
Traceback (most recent call last):
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 218, in load_module
return importlib.import_module(module)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/src/screenshot_v2.py", line 1, in <module>
import jpeg
ModuleNotFoundError: No module named 'jpeg'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 480, in submodules
module = Module.from_name(mod.name)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 404, in from_name
return cls(extract.load_module(name))
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 220, in load_module
raise RuntimeError(f"Error importing {module}") from e
RuntimeError: Error importing src.screenshot_v2
(/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py:482)
Warn: Couldn't import src.widget:
Traceback (most recent call last):
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 218, in load_module
return importlib.import_module(module)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/src/widget.py", line 1, in <module>
import lvgl as lv
ModuleNotFoundError: No module named 'lvgl'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 480, in submodules
module = Module.from_name(mod.name)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 404, in from_name
return cls(extract.load_module(name))
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 220, in load_module
raise RuntimeError(f"Error importing {module}") from e
RuntimeError: Error importing src.widget
(/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py:482)
Warn: Couldn't import src.yolo:
Traceback (most recent call last):
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 218, in load_module
return importlib.import_module(module)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/src/yolo.py", line 1, in <module>
from ui import UI
ModuleNotFoundError: No module named 'ui'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 480, in submodules
module = Module.from_name(mod.name)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 404, in from_name
return cls(extract.load_module(name))
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 220, in load_module
raise RuntimeError(f"Error importing {module}") from e
RuntimeError: Error importing src.yolo
(/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py:482)
Just for anyone else stumbling upon this, the import or module errors are actually easily dealt with via the usage of mock
import sys
if sys.implementation.name == "micropython": # Regular import when inside micropython
import lvgl as lv
else: # Mocked import when run through pdoc
import mock # Mock library
from .mock.lvgl import lv # Fake module folder containing source files with necessary mocked defines
ChatGPT gave the final hints
To mock the lvgl module and make your source loadable for pdoc, you can use the unittest.mock library to create a mock version of the lvgl module. Here's a step-by-step guide on how to achieve this:
- Create a mock for lvgl: Create a file, say mock_lvgl.py, where you define a mock version of the lvgl module.
from unittest.mock import MagicMock
# Mock the lvgl module
lv = MagicMock()
# You can define any specific behavior or attributes for the mock here
# For example:
lv.SomeClass = MagicMock()
lv.some_function = MagicMock()
- Modify the import in your code: Adjust your conditional import to use this mock module when running in a regular Python environment.
import sys
if sys.implementation.name == "micropython":
import lvgl as lv
import random
from global_definitions import ascii_letters
else:
import mock_lvgl as lv # Import the mock_lvgl module instead of lvgl
import random
from .global_definitions import ascii_letters
- Ensure the mock is discoverable by pdoc: Make sure pdoc can find and use the mock_lvgl module. This might require adjusting your PYTHONPATH or ensuring that the mock_lvgl.py file is in the same directory as your main script or in a directory that is part of the Python path.
Here's a more complete example to illustrate: Project Structure
your_project/
│
├── your_script.py
├── mock_lvgl.py
└── global_definitions.py
mock_lvgl.py
from unittest.mock import MagicMock
# Mock the lvgl module
lv = MagicMock()
# Define mock attributes and methods
lv.SomeClass = MagicMock()
lv.some_function = MagicMock()
# Optionally, you can add some documentation to the mocks
lv.SomeClass.__doc__ = "Mocked SomeClass"
lv.some_function.__doc__ = "Mocked some_function"
your_script.py
import sys
if sys.implementation.name == "micropython":
import lvgl as lv
import random
from global_definitions import ascii_letters
else:
import mock_lvgl as lv # Import the mock_lvgl module instead of lvgl
import random
from .global_definitions import ascii_letters
# Your existing code here
Running pdoc
When you run pdoc to generate the documentation, ensure that the current directory is in your Python path:
PYTHONPATH=. pdoc your_script.py
This setup ensures that when pdoc processes your code, it will use the mock_lvgl module, which should resolve any import errors and allow pdoc to generate the documentation from the existing docstrings.
Thanks for posting a potential solution! This is out of scope for pdoc itself for now, the workaround you posted seems to be a good approach.
