Python docs incomplete

[jaxxstorm]

Problem description

Comparing the nodejs docs: https://www.pulumi.com/docs/reference/pkg/nodejs/pulumi/pulumi/#isSecret And the python docs: https://www.pulumi.com/docs/reference/pkg/python/pulumi/

There's lots of missing information in the python docs.

[t0yv0]

I was able to take a quick look here. During triage we were looking at it and though that maybe some small broken HTML code in the pipeline prevented the TOC from rendering on the Python page but it turns out the rabbit hole is rather deep. The pipelines for node docs and Python docs are disjoint. The rendering of the pages is coming from generated sources (gen-src):

   For Pyhon:
   
   page:    https://www.pulumi.com/docs/reference/pkg/python/pulumi/

   gen-src: https://github.com/pulumi/docs/blob/master/content/docs/reference/pkg/python/pulumi/_index.md


   For node:

   page:    https://www.pulumi.com/docs/reference/pkg/nodejs/pulumi/pulumi/#isSecret

   gen-src: https://github.com/pulumi/docs/blob/master/content/docs/reference/pkg/nodejs/pulumi/pulumi/_index.md

The pipeline to do the generation for Python is invoked with ./scripts/generate_python_docs.sh. I had partial success running it on my machine but it seems that it runs regularly via some automation. I understand that it grabs the source material out of Python docstrings from the latest published Python SDK packages.

So:

PyPy: pulumi v3.24.x --> docsgen --> pkg/python/pulumi/_index.md --> hugo --> web

The sources of docsgen have this to say, last touched 10m / 3y ago:

https://github.com/pulumi/docs/blame/master/tools/pydocgen/pydocgen/init.py#L16

 Quote: “This is a little crazy. I will understand if you hate me. ” 

So I think making improvements here to get to parity with Node is a bigger item of work than I originally understood, we need to:

• revise and update docstrings in Python SDK (but note the recent push to make docstrings IDE-friendly that puts other constraints on the format)

• choice 1 - do upgrades to docsgen Python code and templates to generate a richer structure out of the dosctrings, with TOC, etc

• choice 2 - step back and rearchitect how we publish these docstrings; for example Python has widely adopted (if not standard) documentation generators that consume packages with docstrings and produce say readthedocs-themed pages such as https://docs.readthedocs.io/en/stable/api/v3.html - as a Python programmer I'd very much prefer that style of docs formatting as it is immediately familiar and has good support for search and python snippet display. Perhaps we can work out some compromise to add Pulumi branding to this theme without taking away these advantages. We could complement these API docs with narrative docs.

CC @mikhailshilkov