MaterialX icon indicating copy to clipboard operation
MaterialX copied to clipboard
verified publisher icon AcademySoftwareFoundation

Add pybind11 docs

Open kwokcb opened this issue 3 months ago • 5 comments

Purpose

  • Add in Python docs into the MaterialX package for all modules
  • This allows for standard Python help and the ability to integrate this with things like type stubs (pyi) An example of this integrated into VSCode and PyCharm is shown
pyi_snap
  • Tooling used to create html docs via tools like Sphinx but does not add any reliance on such tools. Here is an example custom help page. image

Changes

  • The existing build flag MATERIALX_BUILD_DOCS is used to:
    • Build docs with XML Doxygen output as extracted from C++ code.
    • Run a utility script to parse the XML to insert into PyMaterialX module files.
  • The flag is enable when building Python wheels.

Workflow

The basic logic for build workflow is below. The regular CI does not build docs so there is no effect. Wheels builds will build docs + install doxygen.

graph TB
    Start([Workflow Start])
    
    Start --> RepoCheck{Repository?}
    RepoCheck -->|AcademySoftwareFoundation/MaterialX| OfficialRepo[Official Repository]
    RepoCheck -->|Other Fork| ForkRepo[Fork Repository]
    
    %% Regular Build Path
    OfficialRepo --> RegularBuild[Regular Build Job]
    ForkRepo --> RegularBuild
    
    RegularBuild --> CMakeBuild[CMake Build]
    
    CMakeBuild --> DoxygenRegular{MATERIALX_BUILD_DOCS?}
    DoxygenRegular -->|ON| GenDocsRegular[MaterialXDocs target<br/>Generate Doxygen XML]
    DoxygenRegular -->|OFF| SkipDocs[No Documentation]
    
    GenDocsRegular --> PyBindDocsRegular[PyBindDocs target<br/>pybind_docs.py extracts XML]
    
    PyBindDocsRegular --> ForceCheckRegular{MATERIALX_PYTHON_FORCE_REPLACE_DOCS?}
    ForceCheckRegular -->|ON| ForceReplace[--force flag<br/>Replace existing docs]
    ForceCheckRegular -->|OFF| NoForce[Update only if newer]
    
    ForceReplace --> PyModulesRegular[Build PyMaterialX modules<br/>with embedded docs]
    NoForce --> PyModulesRegular
    SkipDocs --> PyModulesNoDoc[Build PyMaterialX modules<br/>without docs]
    
    PyModulesRegular --> UploadRegular[Upload Artifact]
    PyModulesNoDoc --> UploadRegular
    
    %% Wheels Build Path
    OfficialRepo --> SDist[SDist Job]
    
    SDist --> WheelsJob[Wheels Job]
    
    WheelsJob --> OSSetup{OS-Specific Setup}
    
    OSSetup -->|Linux| LinuxDoxy[yum install doxygen]
    OSSetup -->|MacOS| MacDoxy[brew install doxygen]
    OSSetup -->|Windows| WinDoxy[choco install doxygen]
    
    LinuxDoxy --> BuildWheel[Build Wheel<br/>CMake runs inside cibuildwheel]
    MacDoxy --> BuildWheel
    WinDoxy --> BuildWheel
    
    BuildWheel --> DoxygenWheel{CMake finds Doxygen?}
    DoxygenWheel -->|Yes| GenDocsWheel[MaterialXDocs target<br/>Generate Doxygen XML]
    DoxygenWheel -->|No| WheelNoDocs[Skip docs]
    
    GenDocsWheel --> PyBindDocsWheel[PyBindDocs target<br/>pybind_docs.py extracts XML]
    
    PyBindDocsWheel --> ForceCheckWheel{FORCE flag?}
    ForceCheckWheel -->|Set| ForceReplaceWheel[--force flag<br/>Replace existing docs]
    ForceCheckWheel -->|Not set| NoForceWheel[Update only if newer]
    
    ForceReplaceWheel --> PyModulesWheel[Build PyMaterialX modules<br/>with embedded docs]
    NoForceWheel --> PyModulesWheel
    WheelNoDocs --> PyModulesWheelNoDoc[Build PyMaterialX modules<br/>without docs]
    
    PyModulesWheel --> WheelPackage[Package into wheel]
    PyModulesWheelNoDoc --> WheelPackage
    WheelPackage --> UploadWheel[Upload Wheel]
    
    %% Styling - only color decision nodes
    style DoxygenRegular fill:#404040,stroke:#666,stroke-width:2px,color:#fff
    style DoxygenWheel fill:#404040,stroke:#666,stroke-width:2px,color:#fff
    style ForceCheckRegular fill:#404040,stroke:#666,stroke-width:2px,color:#fff
    style ForceCheckWheel fill:#404040,stroke:#666,stroke-width:2px,color:#fff
    style RepoCheck fill:#404040,stroke:#666,stroke-width:2px,color:#fff
    style OSSetup fill:#404040,stroke:#666,stroke-width:2px,color:#fff

Example

Run log text from this run

2025-12-10T17:31:55.7979420Z   Extracting documentation from Doxygen XML...
2025-12-10T17:31:55.8036070Z   Extracted 184 classes and 2025 functions
2025-12-10T17:31:55.8096780Z   Built lookup table with 7451 keys
2025-12-10T17:31:55.8097270Z 
2025-12-10T17:31:55.8097470Z   Replacing documentation in pybind11 files...
2025-12-10T17:31:55.8103080Z     - PyMaterialX/PyMaterialXGenMdl/PyModule.cpp
2025-12-10T17:31:55.8103480Z     - PyMaterialX/PyMaterialXGenMdl/PyMdlShaderGenerator.cpp
2025-12-10T17:31:55.8104450Z     - PyMaterialX/PyMaterialXGenOsl/PyOslShaderGenerator.cpp
2025-12-10T17:31:55.8105140Z     - PyMaterialX/PyMaterialXGenOsl/PyModule.cpp
2025-12-10T17:31:55.8105630Z     - PyMaterialX/PyMaterialXGenShader/PyUtil.cpp
2025-12-10T17:31:55.8106030Z     - PyMaterialX/PyMaterialXGenShader/PyUnitSystem.cpp
2025-12-10T17:31:55.8106440Z     - PyMaterialX/PyMaterialXGenShader/PyGenOptions.cpp
2025-12-10T17:31:55.8221810Z     - PyMaterialX/PyMaterialXGenShader/PyModule.cpp
2025-12-10T17:31:55.8300070Z     - PyMaterialX/PyMaterialXGenShader/PyShader.cpp
2025-12-10T17:31:55.8402590Z     - PyMaterialX/PyMaterialXGenShader/PyGenContext.cpp
2025-12-10T17:31:55.8503950Z     - PyMaterialX/PyMaterialXGenShader/PyShaderPort.cpp
2025-12-10T17:31:55.8609650Z     - PyMaterialX/PyMaterialXGenShader/PyColorManagement.cpp
2025-12-10T17:31:55.8711220Z     - PyMaterialX/PyMaterialXGenShader/PyShaderGenerator.cpp
2025-12-10T17:31:55.8827850Z     - PyMaterialX/PyMaterialXGenShader/PyTypeDesc.cpp
2025-12-10T17:31:55.8949710Z     - PyMaterialX/PyMaterialXGenShader/PyShaderTranslator.cpp
2025-12-10T17:31:55.9074300Z     - PyMaterialX/PyMaterialXGenShader/PyShaderStage.cpp
2025-12-10T17:31:55.9182250Z     - PyMaterialX/PyMaterialXGenShader/PyHwShaderGenerator.cpp
2025-12-10T17:31:55.9304470Z     - PyMaterialX/PyMaterialXRender/PyCgltfLoader.cpp
2025-12-10T17:31:55.9406170Z     - PyMaterialX/PyMaterialXRender/PyLightHandler.cpp
2025-12-10T17:31:55.9530860Z     - PyMaterialX/PyMaterialXRender/PyOiioImageLoader.cpp
2025-12-10T17:31:55.9636290Z     - PyMaterialX/PyMaterialXRender/PyImageHandler.cpp
2025-12-10T17:31:55.9737780Z     - PyMaterialX/PyMaterialXRender/PyMesh.cpp
2025-12-10T17:31:55.9849740Z     - PyMaterialX/PyMaterialXRender/PyTinyObjLoader.cpp
2025-12-10T17:31:55.9951480Z     - PyMaterialX/PyMaterialXRender/PyShaderRenderer.cpp
2025-12-10T17:31:56.0078200Z     - PyMaterialX/PyMaterialXRender/PyModule.cpp
2025-12-10T17:31:56.0079330Z     - PyMaterialX/PyMaterialXRender/PyImage.cpp
2025-12-10T17:31:56.0180530Z     - PyMaterialX/PyMaterialXRender/PyStbImageLoader.cpp
2025-12-10T17:31:56.0294220Z     - PyMaterialX/PyMaterialXRender/PyGeometryHandler.cpp
2025-12-10T17:31:56.0400360Z     - PyMaterialX/PyMaterialXRender/PyCamera.cpp
2025-12-10T17:31:56.0510150Z     - PyMaterialX/PyMaterialXRenderOsl/PyModule.cpp
2025-12-10T17:31:56.0614310Z     - PyMaterialX/PyMaterialXRenderOsl/PyOslRenderer.cpp
2025-12-10T17:31:56.0716580Z     - PyMaterialX/PyMaterialXRenderGlsl/PyModule.cpp
2025-12-10T17:31:56.0819350Z     - PyMaterialX/PyMaterialXRenderGlsl/PyGLTextureHandler.cpp
2025-12-10T17:31:56.0932910Z     - PyMaterialX/PyMaterialXRenderGlsl/PyGlslProgram.cpp
2025-12-10T17:31:56.1042400Z     - PyMaterialX/PyMaterialXRenderGlsl/PyGlslRenderer.cpp
2025-12-10T17:31:56.1158430Z     - PyMaterialX/PyMaterialXRenderGlsl/PyTextureBaker.cpp
2025-12-10T17:31:56.1295220Z     - PyMaterialX/PyMaterialXGenGlsl/PyGlslShaderGenerator.cpp
2025-12-10T17:31:56.1420770Z     - PyMaterialX/PyMaterialXGenGlsl/PyModule.cpp
2025-12-10T17:31:56.1421140Z     - PyMaterialX/PyMaterialXGenMsl/PyModule.cpp
2025-12-10T17:31:56.1421590Z     - PyMaterialX/PyMaterialXGenMsl/PyMslShaderGenerator.cpp
2025-12-10T17:31:56.1421990Z     - PyMaterialX/PyMaterialXGenSlang/PySlangShaderGenerator.cpp
2025-12-10T17:31:56.1593770Z     - PyMaterialX/PyMaterialXGenSlang/PyModule.cpp
2025-12-10T17:31:56.1694670Z     - PyMaterialX/PyMaterialXFormat/PyUtil.cpp
2025-12-10T17:31:56.1795230Z     - PyMaterialX/PyMaterialXFormat/PyModule.cpp
2025-12-10T17:31:56.1895820Z     - PyMaterialX/PyMaterialXFormat/PyXmlIo.cpp
2025-12-10T17:31:56.1996750Z     - PyMaterialX/PyMaterialXFormat/PyFile.cpp
2025-12-10T17:31:56.2097550Z     - PyMaterialX/PyMaterialXCore/PyInterface.cpp
2025-12-10T17:31:56.2198320Z     - PyMaterialX/PyMaterialXCore/PyNode.cpp
2025-12-10T17:31:56.2299070Z     - PyMaterialX/PyMaterialXCore/PyValue.cpp
2025-12-10T17:31:56.2400180Z     - PyMaterialX/PyMaterialXCore/PyTraversal.cpp
2025-12-10T17:31:56.2501360Z     - PyMaterialX/PyMaterialXCore/PyDocument.cpp
2025-12-10T17:31:56.2602570Z     - PyMaterialX/PyMaterialXCore/PyMaterial.cpp
2025-12-10T17:31:56.2703730Z     - PyMaterialX/PyMaterialXCore/PyUnitConverter.cpp
2025-12-10T17:31:56.2804960Z     - PyMaterialX/PyMaterialXCore/PyUtil.cpp
2025-12-10T17:31:56.2906220Z     - PyMaterialX/PyMaterialXCore/PyDefinition.cpp
2025-12-10T17:31:56.3007210Z     - PyMaterialX/PyMaterialXCore/PyGeom.cpp
2025-12-10T17:31:56.3108250Z     - PyMaterialX/PyMaterialXCore/PyTypes.cpp
2025-12-10T17:31:56.3208760Z     - PyMaterialX/PyMaterialXCore/PyModule.cpp
2025-12-10T17:31:56.3309510Z     - PyMaterialX/PyMaterialXCore/PyProperty.cpp
2025-12-10T17:31:56.3410130Z     - PyMaterialX/PyMaterialXCore/PyException.cpp
2025-12-10T17:31:56.3511310Z     - PyMaterialX/PyMaterialXCore/PyVariant.cpp
2025-12-10T17:31:56.3611570Z     - PyMaterialX/PyMaterialXCore/PyLook.cpp
2025-12-10T17:31:56.3713100Z     - PyMaterialX/PyMaterialXCore/PyElement.cpp
2025-12-10T17:31:56.3812850Z 
2025-12-10T17:31:56.3913940Z   Processed 63 files, patched 48

kwokcb avatar Nov 06 '25 14:11 kwokcb

This looks promising, @kwokcb, and I'm linking this to our long-standing GitHub Issue for Python API documentation:

https://github.com/AcademySoftwareFoundation/MaterialX/issues/342

For our first Dev Days event, @StefanHabel started a project to implement this feature, and he ended up making some great improvements to MaterialX Python in general, but we were never able to come up with a solution that avoided duplication between the documentation strings for C++ and Python.

In your latest proposal, there's still duplication of doc strings, but it seems slightly clearer how a developer would navigate this when they add a new function to C++ and Python , as the doc strings are now part of the Python bindings themselves.

I'd be very interested in thoughts from the community on this proposal, to judge whether the benefits of Python API documentation would be worth the additional burden we'd be placing on developers when they add or modify API methods.

jstone-lucasfilm avatar Nov 11 '25 19:11 jstone-lucasfilm

There is a variant that could be done but it's really not as developer friendly. Basically we put doc strings in separate resource files and then include that file in C++ and pybind C++ and write macros to insert. This however is a lot of work on the C++ side.

The best I could come up with to build Doxygen + pybind insertion via script. It could be run as needed to update, or add in missing pybind docs before a release.

kwokcb avatar Nov 12 '25 04:11 kwokcb

I've added in an build step so doc building is done if Python is being built and Docs are building built. This changes not for the regular workflow since docs are not built in main.yml.

kwokcb avatar Nov 17 '25 14:11 kwokcb

@jstone-lucasfilm . All checked in Py* files could be reverted as I've made it so that it can build as part of wheel building. This leaves a "single source of truth" for docs. Either way works (build and check-in) or build only on when building packages. I did not add in a non-wheels route but it would not be hard to do.

kwokcb avatar Nov 17 '25 15:11 kwokcb

I've reverted all doc code inside PyMaterialX modules. It's all done when building Python wheels now so there is not duplicate anywhere and devs don't need to do anything locally unless they want to see what the docs look like by running the extraction script locally.

kwokcb avatar Dec 10 '25 18:12 kwokcb

This latest version looks really compelling, thanks @kwokcb, and as a next step I'd like to try this in a local build with each of the different build options.

jstone-lucasfilm avatar Dec 13 '25 00:12 jstone-lucasfilm