Top level section header ignored in text/reST package description, not in text/markdown

[di]

When using text/reST for the package description in the METADATA package dist-info file, the top level section header is omitted in the rendered page, however it is included when the description is in text/markdown format.

This can be easily verified running python -m readme_renderer README.rst vs python -m readme_renderer README.rst.

Is this intentional? Looking at the docutils configuration used in readme_renderer it looks so. However, why are markdown and reST threated differently?

Originally posted by @dnicolodi in https://github.com/pypi/warehouse/issues/12204#issue-1368602346

[miketheman]

The basic answer is that the RST behavior is intentional, and the markdown behavior needs some work to make it on par with the RST behavior.

This was previously discussed in https://github.com/pypa/readme_renderer/issues/149#issuecomment-1066119816 and confirmed for the reasons discussed there.

The question remains on whether we should change the Markdown behavior to the same, by removing the first H1, or if we should leave the original content unchanged, and look to remove the behavior from RST.

On one hand, the convention of naming the first line of a README.* after the project is widely adopted, but it's not universal, and not required by any means.

On the other hand, removing the top title does help reduce the overall clutter seen in the UI - as previously discussed in https://github.com/pypi/warehouse/issues/3451 (still undecided/resolved)

To modify the markdown render behavior to match RST, we would have to do something like:

• update https://github.com/theacodes/cmarkgfm to have an option to strip out the first header found
• strip out the header before we pass it to markdown.render()
• something else I haven't thought of yet