Documentation is out of date, update the documentation.

[blindcant]

Hi,

I was talking with bps and Ake in #helpdesk on the Discord about updating the documentation. They both kindly pointed me to some resources and suggested to raise this issue here. I am interested in writing a Python script to download and parse this JSON and then spit it out as a markdown document that could be used here for a reference.

I was planning on writing a simple Python program to download the JSON, parse it, and output it in markdown. I'd write it so it could be run in a Python virtual environment so I can hand it over to the project and it could just be run manually or as part of a CI/CD pipeline so any new commands added to the JSON can be added to the documentation.

So after thinking about this briefly I wanted to raise this issue to ask:

• Is this something the project would be interested in me doing? If yes:
  • Are you happy with this being written in Python? If not, what would the preferred language be?
    • I am only really comfortable writing in Python, Java, Bash, and Ansible. Although Ansible seems like overkill though. But I have done a little JavaScript. php, and C as well.
  • What kind of output structure do you think would be more suitable / usable for users? I was thinking either between:
    • A single markdown document with all the commands sorted by the groups in the JSON but also with a table of contents at the top so you could easily jump to each section.
    • Having multiple markdown documents with a dedicated document per group in the JSON, perhaps with a single table of contents page linking to them all.
  • Is there any other resources besides the JSON that I should use?
  • Is there anything I've missed or need to clarify?

Happy to receive feedback and suggestions on the language used.

blindcant.

EDIT: feel free to assign this to me if you want me to proceed with the work.

[meag]

Thanks @blindcant - sounds good, if you fancy automating this then go ahead, that would be a big help.

In the past it has just been done periodically, but does tend to get out of date (c# method gist, example update here), which generates individual html files which are then included in a single page per cvar group - I think this is what you're suggesting above. The links are on the manual page

No idea if this is any use to you, feel free to start from scratch. There's not really any space for C# in ezquake build but I'm sure python would be fine, as long as it wasn't required to actually build the executable.

There's another 2 json files now, one for macros and another for command line parameters.

Thanks, meag

[blindcant]

I don't have any experience with CI/CD pipelines so I wouldn't be able to automate it, but I plan to write the script in such a way that it just parses the JSON and spits out Markdown whenever you run it. So if you have a CI/CD pipeline on here or elsewhere, you can just add the script as an extra thing to run after your build.

Thanks for the gist I'll go through it and see what its doing and it will be a good starting point compared to a 'blank piece of paper'. Good to know there are more JSON files as well, I'll make sure the script can support them too. I plan to start this project this week, in regards to that, how would you like me to do that? Should I just clone this repo and make a feature branch and submit a pull request once I am done?

[hemostx]

A bit of an aside with regards to BC's doc output automation, and regarding the structure of the help strings themselves. I've looked at improvements to the formatting etc, and a few vars/cmds have really, really long descriptions. In some cases, some of the legacy stuff might not be needed, but overall, such walls of texts don't really belong in the console output, where they suffer from all of the issues inherent to charset text and which are only partly alleviated by proportional fonts (line spacing + line width + glyph legibility etc).

I'm thinking it might be worth it splitting things up: desc : short description remarks : caveats, quick notes etc with the addition of i.e longdescription : extended info for tricky vars spanning several lines. Would probably be an optimized mix of what is currently in lengthy desc+remarks.

By default, /$cvar would print all the usual fields, but if longdescription is present, also print something along the lines of "more info available with /describe $cvar or in the online manual". You then have the option of suffering through longform text in the console with /describe which gives you the full picture, or refer to the html/md friendly view which would also show all of the data.

[hemostx]

Second take on my suggestion : I've gone through all of help_variables again, and it's only a handful of vars with some really, really long stuff. In nearly every case, it's possible to just split the tangents into remarks and to optimize a bit, without requiring an extra optional doc string. Some of the stuff should arguably go into a manual page covering a topic - half of the babble for r_netgraph, for instance, could make its way into something about identifying + fixing net lag issues.

[blindcant]

Just an FYI guys this is going to be my Christmas holiday project, so I am going to start this around the 27th of December. Should only take a couple of days to pump out (famous last words).

[vikpe]

An update of the documentation is indeed needed!

Some thoughts:

• ezQuake docs is using GitHub Pages (Jekyll).
• Jekyll can read "data" files (JSON, CSV etc) eg. ezQuake/ezquake.github.io/_data/foo.json

Hence output to markdown isn't needed (leave markup generation to Jekyll.) A split into logical (smaller) sections could be useful, however I guess that easily can be done in the creation of the files in the main repo.

My suggestion would be to setup GitHub actions in main repo and docs repo.

1. on commit in main: trigger workflow check-help-files in docs
2. check-help-files will download and compare to existing help files in docs
   • if changed: then commit help files to docs
     • GitHub pages will rebuild and publish

A benefit of storing a copy of the help files in the docs repo is that you can build locally during development.

If you don't want to store copies the help files in docs then a hash of each help file would be enough, just need to be able to tell if they are updated or not.

What kind of output structure do you think would be more suitable / usable for users?

I'd go with option 1 (single markdown document with table of contents) and in addition add Algolia doc search in order to easily find specific stuff in the docs.

// XantoM

[blindcant]

An update of the documentation is indeed needed!

Some thoughts:

* ezQuake docs is using GitHub Pages (Jekyll).

* Jekyll can read ["data" files](https://jekyllrb.com/docs/datafiles/) (JSON, CSV etc) eg. `ezQuake/ezquake.github.io/_data/foo.json`

Hence output to markdown isn't needed (leave markup generation to Jekyll.) A split into logical (smaller) sections could be useful, however I guess that easily can be done in the creation of the files in the main repo.

My suggestion would be to setup GitHub actions in main repo and docs repo.

1. `on commit` in `main`: trigger workflow `check-help-files` in `docs`

2. `check-help-files` will download and compare to existing help files in `docs`
   
   * if changed: then commit help files to `docs`
     
     * GitHub pages will rebuild and publish

A benefit of storing a copy of the help files in the docs repo is that you can build locally during development.

If you don't want to store copies the help files in docs then a hash of each help file would be enough, just need to be able to tell if they are updated or not.

What kind of output structure do you think would be more suitable / usable for users?

I'd go with option 1 (single markdown document with table of contents) and in addition add Algolia doc search in order to easily find specific stuff in the docs.

// XantoM

This is a really good idea. Lucky that I've been procrastinating over my holidays and haven't started yet, was actually going to start today. So it sounds like I could just write a script to do step 2. I assume step 1 would be handled within GitHub GUI somewhere? Is that the same for the rebuild and publish at the end of step 2?

[vikpe]

When using GitHub Pages (which the ezquake.github.io repo has enabled), a workflow is added behind the scenes to the repository. A job with on commit trigger will checkout the repository, build the site using Jekyll and publish it. This is done automatically and no further action is needed (all we need to do is to commit the help files to the site repo).

Workflows consists of jobs, jobs consists of actions, actions are commands (eg git push). Workflows for GitHub actions are defined as .yml files located in the repository (.github/workflows/example.yml). The commands/actions are run on docker containers, e.g. ubuntu-latest, so wget and git are avaiable and thus Python isn't needed at all.

To accomplish my suggestion we need two files:

1. a workflow file in the site repo, containing a job to download, check and commit the help files.
2. a workflow file in the main repo to trigger the workflow in the site repo.

(we also need to generate a "personal access token" to allow main repo to trigger the workflow in the site repo)

I've created a fork of ezquake.com with instructions for local development. I can create a dummy ezquake-source repo for us to play around with until we get things working.

[vikpe]

I got it working! :)

• main (ezquake source) workflow
• docs workflow

[vikpe]

Example variable listing based on JSON file: https://vikpe.org/ezquake.github.io/docs/vars/demos/

[meag]

@vikpe - this looks great, thanks very much. @jite - I think we'd need you to set up the personal access token side of things?

[blindcant]

@vikpe just checking this out now, looks awesome!

[hemostx]

Example variable listing based on JSON file: https://vikpe.org/ezquake.github.io/docs/vars/demos/

Going deeper into all that json and remembered this. It is super neat. I have a couple questions for you @vikpe :)

• The major-group and group properties are used to organize vars in the output - nice. It looks like you're excluding the Obsolete groups, right? I'm interested, because there's probably a discussion to be had with the maintainers about what to do with removed/obsolete stuff that's still in the help strings/manual.
• How tunable is the Algolia search? In this POC I found it too limited to be useful: it only appears to search descriptions, not names, so i.e allow_download doesn't find the var itself (it has no description). Also, it can't take me to a specific result within a page, etc etc. Hopefully this can be adressed if the maintainers want to move on with this promising system.
• Do you think we could use different quoting styles in the json sources to have Jekyll emphasize them in the online manual? For example, use double quotes for actual quote marks, but turn simple quotes into bold to highlight a value or a cvar name. (also, currently, double quotes escaped for json are transcribed as-is (\") by Jekyll)

[vikpe]

The major-group and group properties are used to organize vars in the output - nice. It looks like you're excluding the Obsolete groups, right?

Yes they are excluded.

How tunable is the Algolia search?

You can define whatever "selectors" you like. I didn't put a lot of effort into it.

• Current config: https://github.com/algolia/docsearch-configs/blob/master/configs/ezquake.json
• Docs: https://docsearch.algolia.com/docs/required-configuration

Do you think we could use different quoting styles in the json sources to have Jekyll emphasize them in the online manual?

Yes. We can post-process the json sources into whatever output we like.

[tcsabina]

@vikpe @hemostx @blindcant , Just to up this topic/issue a bit. Any volunteer to make ezQuake documentation great again? :)

[vikpe]

Yes I have some ideas. Thinking about using vitepress, semi automated, from source code and community feedback/pull requests.

[vikpe]

Ported most of the content from my last suggestion to a default vitepress installation: https://ezquake.quake.se

• structuring the doc pages a bit and then add algolia doc search will make it easy to browse
• you can fine tune the theme in any way you like. currently using defaults
• each page has an edit link. I think this is important in order to get the docs in order. no one can do it all by himself
• we can have it automatically update on source code changes and new releases using github actions

[vikpe]

Feedback is welcome. I don't want to continue working on this if there is no response for yet another year.

[tcsabina]

Hi Vikpe,

This is sweet!

So you plan to overhaul to complete ezquake site (http://www.ezquake.com), not just the documentation? I am absolutely fine with it!

FYI, ezquake.com is actually https://github.com/ezQuake/ezquake.github.io. So the website content is in github already. I guess that would be the same for this new site as well, right?

Anyway, you have my blessing on this one!

And as a kind of 'feedback': As this will be a community site, I think the community should define (with some guidance) the content in there. So I don't really want to say that this or that should be in there. Let's put it live asap, replacing ezquake.com, and whenever something is missing, we just add it. (Or ideally, we just approve the PR of the contributor ;)).

cheers!

[vikpe]

So you plan to overhaul to complete ezquake site (http://www.ezquake.com/), not just the documentation?

Yes

So the website content is in github already.I guess that would be the same for this new site as well, right?

Yes. However it will be published using a workflow, not using the standard GitHub Pages (which is besed on Jekyll).

Updates from non-maintainers will be in the form of PRs, which allows us to moderate it.

[vikpe]

• new website: https://ezquake.com
• new repo: https://github.com/QW-Group/ezquake.com

@tcsabina you can close this issue. we'll use the new repo to address invalid docs.