grass

grass copied to clipboard

Goal

Build user manual from Markdown using mkdocs

How to test

1. Install requirements, see man/mkdocs/requirements.txt
2. Create file /tmp/code.lua:

function CodeBlock (cb)
    return pandoc.RawBlock('markdown', '```\n' .. cb.text .. '\n```\n')
end

3. Convert HTML files to Markdown:

for f in `find . -name *.html`; do echo $f; cat $f | sed 's#<div class="code"><pre>#<pre><code>#g' | sed 's#</pre></div>#</code></pre>#g' | pandoc --from=html --to=markdown --lua-filter /tmp/code.lua > ${f%%.html}.md;done

4. Uncomment parts commented in 6743f1a
5. Recompile GRASS (make distclean; ./configure ...; make)
6. Build documentation from markdown files: (cd man; make build-mkdocs)
7. Open dist.x86_64-pc-linux-gnu/docs/mkdocs/site/index.html in web browser

Tasks

Critical:

• [ ] fix CI
• [x] man : integrate changes from main
• [ ] ~~separate makefile rules (Markdown.make, MarkdownRules.make) ?~~
• [x] make install
• [x] fix gui/script/ compilation
• [x] fix compilation issue in raster
• [x] fix compilation issue in man
• [x] fix segfault in --md-description
• [x] fix code blocks
• [x] fix commit link

Important:

• [x] implement build_md.py and indices
• [ ] solve warnings generated by mkdocs
• [ ] add missing meta page comments to MD files (as YAML metadata block)
• [ ] consider to keep supporting HTML in g.extension for the addons
• [ ] parser_standard_options: choose formatting than table
• [ ] index: choose better formatting than table
• [ ] replace html with md in links
• [x] remove toc-related code from mkmarkdown.py and change mkdocs style to show page-related TOC
• [x] fix "## DESCRIPTION" appearance from keyword list
• [x] logo points to index.html
• [x] site_name to "GRASS GIS 8.5.0dev Reference Manual"
• [x] fix menu items
• [x] implement parser_standard_options
• [x] implement manual_gallery
• [x] use theme footer

Nice to have:

• [ ] implement class_graphical (https://grass.osgeo.org/grass84/manuals/class_graphical.html is empty ???)
• [ ] manual_gallery: different number of images (md vs html), fix layout (run in man dir: GISBASE="/home/martin/src/grass/dist.x86_64-pc-linux-gnu" ARCH="x86_64-pc-linux-gnu" ARCH_DISTDIR="/home/martin/src/grass/dist.x86_64-pc-linux-gnu" VERSION_NUMBER=8.5.0dev VERSION_DATE=2024 python3 ./build_manual_gallery.py) and see dist.x86_64-pc-linux-gnu/docs/markdown/site/manual_gallery.html and https://grass.osgeo.org/grass84/manuals/manual_gallery.html

Tasks before merge

• [ ] rebase main and regenerate MD files from HTML once again

Tasks after merge(?)

• [ ] remove rest-related scripts
• [ ] generate man from markdown
• [ ] remove HTML-based documentation and clean-up scripts

I think we can and should add meta tags to each manual page by adding a YAML metadata block above the markdown.

---
title: "Your Page Title"
description: "A brief description of your page."
author: "Your Name"
date: "2024-08-13"
tags: ["example", "markdown", "opengraph"]

# Open Graph meta tags
og_title: "Your Open Graph Title"
og_description: "A description for Open Graph."
og_type: "website"
og_url: "http://example.com/page-url"
og_image: "http://example.com/image.jpg"
og_site_name: "Your Site Name"
---

# Your Content

Your Markdown content goes here.

I think we can and should add meta tags to each manual page by adding a YAML metadata block above the markdown.

Good point, I have replaced HTML meta comments by YAML metadata block in 723ee558d1.

Hi @landam! After discussion with @wenzeslaus, we would like to propose to remove the auto-converted MD manual pages from this PR to reduce its size.

Once the software part of this PR is working

• we can create a new PR to add the auto-converted MD manual pages
• then compare them to the original HTML pages and update/fix these MD files if necessary
• and finally remove the HTML pages in another PR.

This way it is easier to handle/review and less work in this PR. What do you think?

Hi @landam! After discussion with @wenzeslaus, we would like to propose to remove the auto-converted MD manual pages from this PR to reduce its size. This way it is easier to handle/review and less work in this PR. What do you think?

@neteler Make sense to me, done in 682d266

Thanks for the progress! Existing pages already look very nice.

I have gone through the resulting docs and found some issues:

~~The module pages are not yet converted (expected would be >700 or so):~~

~~All "intro" pages:~~

• ~~not showing module list yet (probably due to missing module pages?)~~

~~Keywords:~~

• ~~yet missing (probably due to missing module pages?)~~

Some HTML conversion issues (probably to be addressed in the LUA filter?) which I found by randomly checking pages converted by pandoc:

raster/r.sun/r.sun.md:

• <br> (white space on the left side) is not yet converted into \
• the table does not survive conversion

lib/init/helptext.md:

• "country_boundaries" in HTML is escaped with \ (no need)
• single line <br> is also not yet converted properly (see end of file)

The module pages are not yet converted (expected would be >700 or so):

cd dist.x86_64-pc-linux-gnu/docs/mkdocs/site 
ls -la | wc -l
116

I am not sure what is a difference, on my PC:

ls -la | wc -l
1099

To clarify: the MD files are generated by pandoc, yet they are not transferred to the target directory.

@landam I think I found it: my local man Makefile system still generates HTML parser descriptions. Guess I am missing something to activate Markdown here (related to https://github.com/OSGeo/grass/commit/6743f1ae78dfbe7bd45f1cdc1ec5afcbb7610a00#diff-c510a888ce7833bc2d9a852174a00c3dd305889b0096bf565e3ae6d709e6c3e0L30 ?). I'd appreciate a pointer or a HOWTO modification above.

(https://github.com/OSGeo/grass/commit/6743f1ae78dfbe7bd45f1cdc1ec5afcbb7610a00#diff-c510a888ce7833bc2d9a852174a00c3dd305889b0096bf565e3ae6d709e6c3e0L30) ?). I'd appreciate a pointer or a HOWTO modification above.

@neteler In HowTo it's point number 4. It's need to uncomment parts which were commented by https://github.com/OSGeo/grass/commit/6743f1ae78dfbe7bd45f1cdc1ec5afcbb7610a00 (in order to fix CI)

It's need to uncomment parts which were commented by 6743f1a (in order to fix CI)

Now I got it :)

:100: and I get the full Markdown based manual pages!

@neteler or @landam Can you please attach the full Lua filter you are using?

Under "How to" above is the LUA code (3 lines) used by Martin and me so far.

I have attempted to expand it with ChatGPT but it seems that it is not chaining the functions as desired (I have no LUA knowledge and didn't find much concerning the application of multiple LUA filters): code.lua.gz

@neteler or @landam Can you please attach the full Lua filter you are using?

The basic Lua filter used by me is mentioned by "How to test" section.

(using prettier on the HTML files generated from MD ( npx prettier --print-width 80 -w *.html) I have found some remove leftover HTML code in the MD part: fixed in ac5e963 and a1fc15e04837abf55df3f526c76242aa0d66594a)

In total, 693 HTML files are generated in dist.x86_64-pc-linux-gnu/docs/mkdocs/site/.

Only 3 files now still contain errors (excellent result!):

i.zc.html
[error] i.zc.html: SyntaxError: Opening tag "meta" not terminated. (9:9)
[error]    7 |       <meta name="viewport" content="width=device-width,initial-scale=1">
[error]    8 |       
[error] >  9 |         <meta name="description" content="Zero-crossing "edge detection" raster function for image processing.">
[error]      |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[error]   10 |       
[error]   11 |       
[error]   12 |       

-> quotes in quotes issue

r.patch.html
[error] r.patch.html: SyntaxError: Opening tag "meta" not terminated. (9:9)
[error]    7 |       <meta name="viewport" content="width=device-width,initial-scale=1">
[error]    8 |       
[error] >  9 |         <meta name="description" content="Creates a composite raster map layer by using known category values from one (or more) map layer(s) to fill in areas of "no data" in another map layer.">
[error]      |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[error]   10 |       
[error]   11 |       
[error]   12 |       

-> also quotes in quotes issue

manual_gallery.html
[error] manual_gallery.html: SyntaxError: Unexpected closing tag "div". It may happen when the tag has already been closed by another tag. For more info see https://www.w3.org/TR/html5/syntax.html#closing-elements-that-have-implied-end-tags (631:4)
[error]   629 | </p>
[error]   630 |
[error] > 631 | <p></div>
[error]       |    ^^^^^^
[error]   632 | <script id="init-glightbox">const lightbox = GLightbox({"touchNavigation": true, "loop": false, "zoomable": true, "draggable": true, "openEffect": "zoom", "closeEffect": "zoom", "slideEffect": "slide"});
[error]   633 | document$.subscribe(() => { lightbox.reload() });
[error]   634 | </script></body>

-> not sure here where this comes from.

Also raster/r.li/r.li.html suffers from a missing </div>, probably the original HTML file contains a bug (I suppose the header stuff therein should be removed, as seen in other "meta page description" pages).

I have taken liberty to change the pandoc code in the HOWTO above to generate Github-Flavored Markdown -t gfm.

Example: the table in r.sun.md is now syntactically accepted and readable (just the width needs to be adjusted somehow):

Background of supported flavors:

• https://pandoc.org/MANUAL.html#extensions
• https://pandoc.org/MANUAL.html#markdown-variants

I have taken liberty to change the pandoc code in the HOWTO above to generate Github-Flavored Markdown -t gfm.

Great! Please feel free to create new PR for HTML to MD conversion. This PR (changing build system to support mkdocs/MD) is ready for review.

Looks great to me! What will be the steps needed in regards to grass-addons after that change? Will they work out of the box?

Looks great to me! What will be the steps needed in regards to grass-addons after that change? Will they work out of the box?

This PR is just an initial step. It adds support for markdown-based documentation in build system. HTML-based documentation is still default.

And yes, we have to convert also Addons documentation to markdown, similarly as done for GRASS core modules, see https://github.com/OSGeo/grass/pull/4620

Steps:

1. Merge this PR
2. Finish & merge #4620
3. Convert Addons documentation to Markdown
4. Remove HTML-related rules from build system and HTML files

It would be also nice to remove RST-related code which was introduced in the past and never finished.

I was about to approve, but did a quick other check. There's two things that stood out.

I get an error with !ENV:

... > make build-mkdocs

MkDocs encountered as error parsing the configuration file: could not determine a constructor for the tag '!ENV'
  in ".../dist.x86_64-pc-linux-gnu/docs/mkdocs/mkdocs.yml", line 2, column 12
make: *** [Makefile:247: build-mkdocs] Error 1

After deleting !ENV from the referenced file:

... > make build-mkdocs
ERROR   -  Config value: 'theme'. Error: Unrecognised theme name: 'material'. The available installed themes are: mkdocs, readthedocs 
ERROR   -  Config value: 'markdown_extensions'. Error: Failed loading extension "pymdownx.details". 
ERROR   -  Config value: 'plugins'. Error: The "glightbox" plugin is not installed 

Aborted with 3 Configuration Errors!
make: *** [Makefile:247: build-mkdocs] Error 1

I get an error with !ENV:

Environmental variables are supported by mkdocs: https://www.mkdocs.org/user-guide/configuration/#environment-variables

@wenzeslaus Which version of mkdocs do you have?

... > make build-mkdocs ERROR - Config value: 'theme'. Error: Unrecognised theme name: 'material'. The available installed themes are: mkdocs, readthedocs ERROR - Config value: 'markdown_extensions'. Error: Failed loading extension "pymdownx.details". ERROR - Config value: 'plugins'. Error: The "glightbox" plugin is not installed

please install requirements: man/mkdocs/requirements.txt

See How To, step 1.

@wenzeslaus Which version of mkdocs do you have?

Probably you have to install https://github.com/waylan/pyyaml-env-tag

@wenzeslaus Which version of mkdocs do you have?

Probably you have to install https://github.com/waylan/pyyaml-env-tag

If so, we have to update requirements file accordingly. Let us now.

I used the requirements file. I have mkdocs 1.1.2. I also added pyyaml-env-tag and reinstalled. I will try to trash everything and start over, but I think I'm still the three mkdocs plugins.

Thanks. Starting from scratch with the pyyaml-env-tag package helped. The result (for core repo) looks great!

Thanks. Starting from scratch with the pyyaml-env-tag package helped. The result (for core repo) looks great!

Ok, requirements file updated in d45c0b68b3

@landam Overall it looks good to me, thanks for your hard work!

One thing I don't understand yet: if we keep both HTML and MD in parallel for a short period, how to deal with the comment parts as seen in 6743f1ae78dfbe7bd45f1cdc1ec5afcbb7610a00? Should this now be activated in the PR?

One thing I don't understand yet: if we keep both HTML and MD in parallel for a short period, how to deal with the comment parts as seen in 6743f1a? Should this now be activated in the PR?

@neteler No, otherwise the GRASS compilation fails (and breaks CI) since there are no MD files in repo. It can be activated in #4620 or in follow-up PR which adds generated MD files to git.