Migrate diagrams in system docs to mermaid.js

[Benjamin-Philip]

This PR:

• Adds support to use mermaid diagrams in the documentation
• Updates the relevant documentation
• Migrates all the diagrams in the system docs to mermaid.js

[github-actions[bot]]

CT Test Results

    3 files    143 suites   49m 35s :stopwatch: 1 587 tests 1 538 :white_check_mark: 49 :zzz: 0 :x: 2 326 runs  2 252 :white_check_mark: 74 :zzz: 0 :x:

Results for commit 102ad38f.

:recycle: This comment has been updated with latest results.

To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass.

See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally.

Artifacts

• Complete CT logs (Download Logs)
• HTML Documentation (Download HTML Docs)
• No Windows Installer found

// Erlang/OTP Github Action Bot

[Benjamin-Philip]

@garazdawi, I noticed that there might be many diagrams to migrate:

$ find -iregex '.*html.*\.\(svg\|png\|gif\|fig\)' | grep -v logo | wc -l
73

Do you want to split this into separate PRs for each lib, or would you rather migrate all the diagrams at one go in a single PR? So far I have only migrated 2 from system, and I feel it may take a while to migrate every diagram.

[CLAassistant]

All committers have signed the CLA.

[garazdawi]

Thanks! Do you know if using mermaid means that epub will not contain those diagrams?

Do you want to split this into separate PRs for each lib, or would you rather migrate all the diagrams at one go in a single PR?

It would be better to split them into multiple PRs, one for each lib seems about right.

[Benjamin-Philip]

It would be better to split them into multiple PRs, one for each lib seems about right.

Will do. I will edit the commit message and title soon.

Thanks! Do you know if using mermaid means that epub will not contain those diagrams?

Right now the diagrams are dynamically rendered on the browser. This means you will just get the diagram source as raw text on the EPUB.

[josevalim]

Right now the diagrams are dynamically rendered on the browser. This means you will just get the diagram source as raw text on the EPUB.

I looked into this and it is possible to make the diagrams show up in the EPUB. You need to add mermaid.min.js to the assets folder (instead of a CDN) and use it instead. However, I couldn't get neither Apple Books nor Calibre to render them properly. So if you want the diagrams to work on EPUB, then images are the best format. If you find them optional, then you can probably add some CSS to hide their source from the EPUB.

[Benjamin-Philip]

So if you want the diagrams to work on EPUB, then images are the best format.

This is what I was going suggest. You can render mermaid diagrams to either SVGs or PNGs using the mermaid cli. We can just create a make task to build these and use the output SVGs in the markdown docs. This seems to be the simplest and most reliable way to use mermaid and support EPUBs.

However when we render, we have to pick either the light or dark theme in advance. As far as I know, auto switching is yet to be supported. There are 2 ways I thought of circumventing this:

We could use inline css to manage the auto switching. However this may be difficult to implement, and may not be supported in all browsers (see link). If we manage to get this to work, we could push this upstream and not maintain the CSS ourselves.

We could also pre-render both the light and dark SVGs (defaulting to the light mode for the EPUB) and use some JS to auto switch between them. This is a simpler approach and should work smoothly on all browsers. However this will result in twice the number of assets than the previous approach. I personally prefer this.

[josevalim]

The script we have for Elixir handles color changes: https://github.com/elixir-lang/elixir/blob/main/lib/elixir/scripts/elixir_docs.exs#L191 - I believe it is the same suggested in ExDoc README.

[garazdawi]

If there is a simple way to get mermaid to work in epub, then that would be great. But there is no need to spend too much time on it. I was mostly curious if it would work or not.

[Benjamin-Philip]

The script we have for Elixir handles color changes: https://github.com/elixir-lang/elixir/blob/main/lib/elixir/scripts/elixir_docs.exs#L191 - I believe it is the same suggested in ExDoc README.

Let me test if this works with SVGs.

If there is a simple way to get mermaid to work in epub, then that would be great. But there is no need to spend too much time on it. I was mostly curious if it would work or not.

Like I said, pre-rendering the diagrams into SVGs will make this work both for HTML and EPUB docs. The only concerns are that:

1. We may loose automatic colour switching in the HTML docs
2. We will introduce a new dependency to build the docs

[Benjamin-Philip]

If there is a simple way to get mermaid to work in epub, then that would be great. But there is no need to spend too much time on it. I was mostly curious if it would work or not.

Is this absolutely necessary? If not, we can just do what José suggested:

If you find them optional, then you can probably add some CSS to hide their source from the EPUB.

[garazdawi]

Is this absolutely necessary?

No, it is not. We can hide the diagrams for the time beeing, and if we see that a lot of people use the epub documentation, we can think of a way to do that then. We just need to be vigilant that these types of diagrams don't end up in the function documentation as I'm guessing VS Code/Emacs/VIM also will have a hard time displaying them.

[josevalim]

We just need to be vigilant that these types of diagrams don't end up in the function documentation as I'm guessing VS Code/Emacs/VIM also will have a hard time displaying them.

Good point. I have removed those from ANSI docs in Elixir here: https://github.com/elixir-lang/elixir/commit/0b7e5734eca92fcd01fe42591e87dca7d0a3bc3c

For places that render markdown, such as VS Code, it may be doable with a CSS selector as well.

[garazdawi]

Hello! Just wanted to check if you are done with this PR or if there is more work to be done?

[Benjamin-Philip]

Hi Lukas, there is more work to be done. Here's what's been done so far:

• Support to use mermaid has been added to all the modules
• Documentation has been updated to include this

I've only migrated the mermaid diagrams in the gen_statem design principles page. I still have to migrate the diagrams in following pages in the system docs:

$ find -type f -name '*.md' -exec grep -l assets/ {} \;

./design_principles/design_principles.md
./design_principles/distributed_applications.md
./design_principles/gen_server_concepts.md
./design_principles/included_applications.md
./design_principles/sup_princ.md
./oam/oam.md
./tutorial/c_port.md
./tutorial/c_portdriver.md

Apart from the actual migration, I still need to look into removing the mermaid source from the EPUB docs.

---

Sorry about delaying the PR. I was busy the month as I was speaking at CodeBEAM America, and I went on vacation after that. I will try to have it review ready in 1-2 weeks.

[garazdawi]

Thanks for the update. I just wanted to check if you had abandoned the project. There is no rush to finish, just keep in mind that if you would like this to be part of the Erlang/OTP 27 release, it needs to be done by early May.

[Benjamin-Philip]

@garazdawi, I've finished migrating all the diagrams in the system namespace. I think you will need to rebuild the documentation artifacts to view the diff since the diagrams will need to be rendered.

However, I still need to fix the rendering for EPUB docs.

When you're checking the diagrams, please have a close look at the OAM graphs. I have a feeling that I might have misinterpreted them.

[Benjamin-Philip]

However, I still need to fix the rendering for EPUB docs.

How are EPUB docs built? I couldn't find any info in the internal docs.

[garazdawi]

I pushed a commit fixing the Makefile to work.

How are EPUB docs built? I couldn't find any info in the internal docs.

If you rebase on latest master it will be automatically built

[Benjamin-Philip]

If you rebase on latest master it will be automatically built

I just rebased. I'll look into the EPUB docs now.

[Benjamin-Philip]

@garazdawi, I think we may be ready to merge once you approve the EPUB docs. I have added some suggestions for the legend issue.

[Benjamin-Philip]

Reverted, squashed and force pushed as requested.

[Benjamin-Philip]

I forgot to revert the changes you made to the Makefile on asset copying. For some reason the asset copying wasn't an issue for me, even after cleaning. Hopefully the last commit should pass?

[garazdawi]

I did a force push to your commit to fix one (hopefully?) last remaining issue.

[Benjamin-Philip]

I did a force push to your commit to fix one (hopefully?) last remaining issue.

It looks like we're good to merge. The workflow passed. I wasn't able to replicate the issue on make docs which is why I couldn't catch it (or test my last commit). Very weird.

[garazdawi]

It looks like we're good to merge. The workflow passed.

Yes, indeed. I will have one last look and then merge.

I wasn't able to replicate the issue on make docs which is why I couldn't catch it (or test my last commit). Very weird.

The error resulted in broken links that our link checking script catches. The link checker is not run when you do make docs, instead you need to run ./otp_build check, for more information see https://github.com/erlang/otp/blob/master/HOWTO/DEVELOPMENT.md#validating-documentation

[garazdawi]

Looks good! Thanks for your help!