Support for markdown in standalone pages

[panglesd]

This PR is a revival of @dbuenzli's original PR on odoc-parser. Since the parser has been moved to the odoc repo, I think we should continue the discussion here.

As in the original PR, syntax for ocamldoc references in markdown is made like this: [!Module]; and syntax for heading labels follows pandoc's specification (which is also followed by several other markdown variants). Math and tables uses cmarkit's specification.

However, there are the differences with @dbuenzli's PR:

• In this PR, Markdown can be used only in standalone documentation pages, based on the extension (md vs mld). (@dbuenzli's PR "sniffed" the syntax to determine whether it was ocamldoc or markdown.)

• Math and table support have been added.

• The commit history has been split out in the following commits:

  1. Support for (strict) CommonMark syntax on standalone documentation pages,
  2. Support for ocamldoc references, indexlist and "!modules" syntax.
  3. Support for math,
  4. Support for tables,
  5. Support for heading labels (not part of cmarkit),
  6. Documentation/refactoring commits.

  This hopefully allows to remove features from this PR more easily, depending on how many of them we want as a first step.

(Most of the work has been authored by @dbuenzli. who was added as co-author of relevant commits. Let me know if that is the right way to do it!)

---

Note: there are already several projects that show interest for markdown support in odoc:

• md2mld which converts markdown files into mld files,
• voodoo, the odoc driver responsible for generating content at ocaml.org, translates markdown files directly into an ocamldoc AST
• Not exactly this PR since it's more about comments in mld files, but ocaml-lsp turns docstrings into markdown to make use of this lsp feature. Having them directly as markdown would be nice!

---

Please share your opinion on this new feature!

[panglesd]

I forgot to mention one of the challenge such change will lead to.

cmarkit is only available for ocaml of version at least 4.14. So we have several options here: support markdown only when compiled with a recent release of ocaml, make compatibility patches to cmarkit, drop support for old compilers on new versions of odoc, ...

[dbuenzli]

Please share your opinion on this new feature!

Features are easy to add and hard to remove. Personally I don't think this should be merged, I don't think I need to expand on this, the tradeoffs are listed in the document added to this PR and I think that allowing this only in odoc pages makes it even worse.

[panglesd]

I think that allowing this only in odoc pages makes it even worse

I don't agree. Most projects already have files written using the markdown format: often a README or some documentation. If we stick to strict GFM, this PR is simply adding support to generate docs for files which were already there.

I agree though that it is a step toward markdown support in docstrings, which would bring the question of the deprecation of ocamldoc. However, the adoption of markdown in documentation pages would be a good way to assess whether documentation writer prefer to write in markdown or in ocamldoc, without comitting too much on markdown.

[dbuenzli]

this PR is simply adding support to generate docs for files which were already there.

These files simply use markdown, not ocamlmark which is a different language.

[panglesd]

Your quotation is removing part of my sentence!

If we stick to strict GFM, this PR is simply adding support to generate docs for files which were already there.

My point is that we could remove the commit that adds a new syntax for markdown files from this PR (this one) and see whether the community favors markdown to ocamldoc (despite the lacking features). If yes, it is a good hint that the markdown syntax is preferred. If not, we did not commit too much into markdown, and still have the nice feature that we can render README.md and other CONTRIBUTING.md

[dbuenzli]

If yes, it is a good hint that the markdown syntax is preferred

It's quite evident that the markdown syntax will be preferred, since a lot of people won't bother learning something else, especially newcomers. But of course this glosses over all the problems that using markdown for documentation has.

If not, we did not commit too much into markdown, and still have the nice feature that we can render README.md and other CONTRIBUTING.md

I don't think that's the duty of odoc. This was already discussed elsewhere but a README.md nowadays serves an entirely different purpose, namely code hosting platforms landing pages, where a lot of context you may want to add is purely noise and/or irrelevant (e.g. links into the repo structure) in the context of a landing package page for consolidated api docs (odig, ocaml.org/p/, etc.).

[panglesd]

I see a lot of docs written in markdown here for instance, which would benefit from being rendered by odoc in ocaml.org. I also think that having people actually writing docs is better than having a very good language to write them.

Given the quality of your documentations, it is clear that not having markdown support is not stopping you from writing docs. I often wish I could use markdown syntax, mostly out of habits and better editor support, but I don't think that is a big problem for me either. However, I strongly suspect[^1] it really prevents many people from writing docs. I find it strange, but if that is true I would personally prefer adding markdown support (with all its consequences).

[^1]: I have made no user study!

[dbuenzli]

I see a lot of docs written in markdown here for instance, which would benefit from being rendered by odoc in ocaml.org

There is a difference between rendering markdown and introducing a new language, hacked over a language whose syntactic structure and semantics only really works well to write simple webpages or text blurbs. Most of the time once you add the missing pieces you are no longer in the markdown philosophy and you get to a language with structures that are not far from ocamldoc itself.

I think that if ocaml.org wants to render markdown it can do so without having to add a 7k looc dependency to odoc.

If there is a larger wish to move from ocamldoc to ocamlmark (I would personally advise against). I think we should do so in a much more principled manner which is described in the document you added to this PR. Otherwise we are just adding noise to a system that has so far remained quite clean.

However, I strongly suspect1 it really prevents many people from writing docs.

If we are to speculate then I strongly suspect that it's rather the lack of… documentation. It's all about education. Where is the nice tutorial on ocaml.org on writing doc strings and package documentation with .mld pages ? Where do I learn that this system actually exists ? Unsurprisingly, it's all about writing code there.

[panglesd]

Where is the nice tutorial on ocaml.org on writing doc strings and package documentation with .mld pages ?

I think it is there: https://ocaml.org/docs/generating-documentation. I agree it is a bit minimal, and that is a good idea to improve it.

Maybe it is not made visible enough on ocaml.org, but having it more visible often mean having other important subjects less visible...