dnscontrol DOCS: TOC builder (for functions and providers)

first attempt to automate TOC creation.

This is a draft so there's probably a few misses in here to sort through after some test doc generations.

Summary Automatically generate the TOC in SUMMARY.md and fill it with the hierarchical doc structure found in the documentation folder. Now we don't need to manually update it or its links. Folder structure depth is mirrored in the TOC.

It's possible to supply file/folder name exceptions at various points.

Checks are simple and can possibly be defeated.

Due to the (renames and) changes for this PR, it's not advisable to 'merge master' into this branch.... yet.

Mar 17 '23 17:03 systemcrash

OMG. So much structure. So many doc. Everything still appears intact on the doc builds. Could do with some polish for filenames.

Mar 17 '23 17:03 systemcrash

I may invest some time to program a scour mechanism that goes through all markdown and updates all the links if a file has moved location or got renamed. But that can come later.

Mar 17 '23 19:03 systemcrash

That would work.

Mar 17 '23 20:03 systemcrash

Thanks @systemcrash! Another step in the right direction! I myself have had similar ideas see https://github.com/StackExchange/dnscontrol/issues/1998. I just don't know if I would make the same choice to generate the full SUMMARY.md myself. With this you lose the ability to show better menu names than in the new situation.

Some examples:

-CI/CD example for GitLab
+Ci-Cd-Gitlab

-Why CNAME/MX/NS targets require a "dot"
+Why Targets Require The Final Dot

I've always thought of two (or more) GitBook menu structures:

I'm curious about your thoughts! @systemcrash @tlimoncelli

Mar 17 '23 20:03 cafferata

OK - I'm satisfied with the results now.

Thanks @systemcrash! Another step in the right direction! I myself have had similar ideas see #1998. I just don't know if I would make the same choice to generate the full SUMMARY.md myself. With this you lose the ability to show better menu names than in the new situation.

Some examples:
-CI/CD example for GitLab
+Ci-Cd-Gitlab
-Why CNAME/MX/NS targets require a "dot"
+Why Targets Require The Final Dot

Just a question of renaming the file (as closely) as possible. Or modifying the code to pick out the first # header from each md and updating every md to have one, if perfection is required. But perfection is often the enemy of good.

I've always thought of two (or more) GitBook menu structures:
1. [The providers](https://github.com/StackExchange/dnscontrol/blob/f912b15/documentation/SUMMARY.md?plain=1#L89-L129)

2. [The language reference](https://github.com/StackExchange/dnscontrol/blob/f912b15/documentation/SUMMARY.md?plain=1#L15-L83)
I'm curious about your thoughts! @systemcrash @tlimoncelli

Do you think they're sufficiently mirrored in the current test results? Take a look and let me know @cafferata

Mar 18 '23 16:03 systemcrash

In tandem with this PR, I recommend that (a few) providers might also be renamed (and their variables too?) to match the docs. Things like Route 53 are findable by search, although this one is actually called "Amazon Route 53".

( PR has overview of which provider md files got renamed )

Mar 18 '23 16:03 systemcrash

@systemcrash a number of feedback points:

The absolutes file paths are not supported by GitBook. This causes the URLs to not work at this time.
By redesigning folders/summary, this pull request not only has a generation process been built, but also a new look and feel. As far as I'm concerned, we take it step by step so that there is clarity about exactly what has changed and for what reason. Perhaps @tlimoncelli thinks differently about this? As far as I'm concerned, this state of pull request is not ready to merge.
~~See also https://github.com/StackExchange/dnscontrol/pull/2197#discussion_r1141080537~~

Before	After

Mar 18 '23 18:03 cafferata

Yes - thanks for the feedback. I made a few minor adjustments to retain the original structure as closely as possible. Seems to have worked nicely. The only part that won't align is the provider names - the provider .md files and the providers need renaming to synchronize or, another solution.

When naming freely the provider md files, it looks like you want: https://docs.dnscontrol.org/~/revisions/284FcG65SMBYAdV2iV3g/providers/providers

If there is some structuring you don't like, one just has to rearrange the files/folders (bearing in mind to adjust any hyperlinks.)

I think this is where we gain by "good code documents itself" - by renaming the providers so everything looks as you want it.

Mar 18 '23 22:03 systemcrash

I know this is still marked as draft but I'm going to be at SRECon until Friday so I thought I'd write to say that this is definitely going in the right direction. I've wanted the TOC to be automatically updated for a long time! I'm excited to see this change!

cafferata did the big migration to the new docs.dnscontrol.org website so I'll defer to him for feedback on this PR. Let me know when you're ready for me to take a look.

Mar 20 '23 18:03 tlimoncelli

I'm satisfied with how it is now. The structure is virtually identical to the current docs site. I just don't know what's going on to put files into place. The weird thing being that using the 'absolute' path, things seemed to work out when I looked at the results (as opposed to shorter, non-absolute paths without the leading (/)documents)

Mar 21 '23 01:03 systemcrash

Looks good! Mark it as "ready for review" (it is a draft currently), rebase to master, and I'll merge it.

Mar 22 '23 05:03 tlimoncelli

Hi @cafferata - are there any other changes you'd like to see for the long term?Or is it now just a question of modifying the processes on the docs server?

Mar 22 '23 23:03 systemcrash

I'm especially curious if @tlimoncelli agrees with the structure change.

The new menu structure shows the technical details/directory names.

-Getting Started
+01 Getting Started

-Language Reference
+02 Language Reference

Determining the order of menu items is no longer possible.

Example Getting Started menu items

-Overview
-Examples
-Migrating zones to DNSControl
-TypeScript autocomplete and type checking
+Examples
+Getting Started
+Migrating Zones To Dnscontrol
+TypeScript Autocomplete And Type Checking

Example Language Reference

-Top Level Functions
-Domain Modifiers
-Record Modifiers
+Domain Modifier Functions
+Record Modifier Functions
+Top Level Functions

The chapters within the menu structure

-Service Providers
-Commands
-Advanced features
-Developer info
-Release
+Advanced Features
+Commands
+Developer Info
+Providers
+Release

External links Randomly adding external links, such as the GitHub releases, is no longer possible.
Provider names that change
```
-Microsoft DNS Server on Microsoft Windows Server
+Msdns
```
- https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/providers/providers/msdns
- https://docs.dnscontrol.org/service-providers/providers/msdns
Absolutes file paths https://github.com/StackExchange/dnscontrol/pull/2197#issuecomment-1474936257 This causes the URLs to link to the GitHub markdown view instead of GitBook documentation URLs. Question, how are the renames ensured that this goes well? How has this been tested?
1. Example Introduction to DNSControl > Quick start tutorial
  - https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-8a992fc8a61d84dc7e49e3b126235f05599212988c01f9b712517995b9a59531R7
  - https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/#:~:text=quick%20start%20tutorial
2. Example Introduction to DNSControl > migrate
  - https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-8a992fc8a61d84dc7e49e3b126235f05599212988c01f9b712517995b9a59531R7
  - https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/#:~:text=migrate
3. Example Introduction to DNSControl > language spec
  - https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-8a992fc8a61d84dc7e49e3b126235f05599212988c01f9b712517995b9a59531R7
  - https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/#:~:text=language%20spec
4. Example Providers > Provider table > Domain modifier deeplink
  - https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-28e66bede643a7156cf9c19950731e1a3a9ab66becf1a063f7705076bfc4dbdaR78
  - https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/providers/providers
5. Example Record Modifier Functions > Service Provider Specific > Amazon Route 53 > R53_ZONE
  - https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-64fa1fd972040542557ac120c90268ff7e8e6fa6407826588f2cf9a82fdbdf7aR15
  - https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/02-language-reference/record-modifier-functions/service-provider-specific/amazon-route-53/r53_zone

Mar 23 '23 16:03 cafferata

I'm especially curious if @tlimoncelli agrees with the structure change.
1. The new menu structure shows the technical details/directory names.
   ```diff
   -Getting Started
   +01 Getting Started
   ```

I don't see the problem with technical details. If anything, it makes it easier for new-comers to find and contribute.

I did not find a convenient way of forcing something to appear first, when everything is sorted alphanumerically, except perhaps naming entries G... instead of g... and sorting alphabetically (A-Z first, then a-z). That has its own problems, eg entries that begin DNS.... This whole project is about DNS...

What is the documentation trying to be? Reference? Howto? User guide (examples)? Reference should be alphabetically sorted (providers, functions). Howtos can land in a different folder and get a different TOC...? But we're trying to avoid all of this.

   ```diff
   -Language Reference
   +02 Language Reference
   ```

2. Determining the order of menu items is no longer possible.
   
   1. **Example Getting Started menu items**
      ```diff
      -Overview
      -Examples
      -Migrating zones to DNSControl
      -TypeScript autocomplete and type checking
      +Examples
      +Getting Started
      +Migrating Zones To Dnscontrol
      +TypeScript Autocomplete And Type Checking
      ```
   2. **Example Language Reference**
      ```diff
      -Top Level Functions
      -Domain Modifiers
      -Record Modifiers
      +Domain Modifier Functions
      +Record Modifier Functions
      +Top Level Functions
      ```
   3. **The chapters within the menu structure**
      ```diff
      -Service Providers
      -Commands
      -Advanced features
      -Developer info
      -Release
      +Advanced Features
      +Commands
      +Developer Info
      +Providers
      +Release
      ```

Do we need to? 'Everything else' in the project is sorted alphabetically, which makes it easier to locate entries (reference).

3. External links
   Randomly adding external links, such as the [GitHub releases](https://github.com/StackExchange/dnscontrol/releases/latest), is no longer possible.

Such entries need to land outside of the tags, true. Other 'decorative' static entries can land somewhere in the middle, but that means more tags and more code complexity. I would rather code be easy to understand.

'Randomly' adding content goes into an existing file or makes a new file somewhere, which gets checked in anyway. Don't see the problem.

4. Provider names that change
   ```diff
   -Microsoft DNS Server on Microsoft Windows Server
   +Msdns
   ```

I made an earlier suggestion that we rename provider entries, to align with the real world. Only drawback is that users might have to (oh no) change a text string in their .js files.

e.g. hedns -> hurricane_electric_dns (underscores replaced to spaces). Frankly I find real world naming much easier. Putting some of the short forms into google to find who they were was difficult for some, and uncertain for others.

   * https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/providers/providers/msdns
   * https://docs.dnscontrol.org/service-providers/providers/msdns

5. Absolutes file paths [DOCS: TOC builder #2197 (comment)](https://github.com/StackExchange/dnscontrol/pull/2197#issuecomment-1474936257)
   This causes the URLs to link to the GitHub markdown view instead of GitBook documentation URLs.

So... how does one fix this, if gitbook is the solution? Relative but with some path depth?

   **Question**, how are the renames ensured that this goes well? How has this been tested?
   
   1. **Example `Introduction to DNSControl` > `Quick start tutorial`**

The renames were made to follow how they are manually named in the TOC. Just rename the source file. 1:1 naming. Everything else was just link decoration. Now it's automatic. File == page == link.

      * https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-8a992fc8a61d84dc7e49e3b126235f05599212988c01f9b712517995b9a59531R7
      * https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/#:~:text=quick%20start%20tutorial
   2. **Example `Introduction to DNSControl` > `migrate`**
      
      * https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-8a992fc8a61d84dc7e49e3b126235f05599212988c01f9b712517995b9a59531R7
      * https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/#:~:text=migrate
   3. **Example `Introduction to DNSControl` > `language spec`**
      
      * https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-8a992fc8a61d84dc7e49e3b126235f05599212988c01f9b712517995b9a59531R7
      * https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/#:~:text=language%20spec
   4. **Example `Providers` > `Provider table` > Domain modifier deeplink**

This one I'm not sure how to resolve - I don't know what happens server side. ¯_(ツ)_/¯ But I'll stick with something that works.

      * https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-28e66bede643a7156cf9c19950731e1a3a9ab66becf1a063f7705076bfc4dbdaR78
      * https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/providers/providers
   5. **Example `Record Modifier Functions` > `Service Provider Specific` > `Amazon Route 53` > `R53_ZONE`**

Same content.

      * https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-64fa1fd972040542557ac120c90268ff7e8e6fa6407826588f2cf9a82fdbdf7aR15
      * https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/02-language-reference/record-modifier-functions/service-provider-specific/amazon-route-53/r53_zone

Documentation search precludes any of this from being a problem. Know what you want to find? ⌘K.

Mar 23 '23 16:03 systemcrash

I'm especially curious if @tlimoncelli agrees with the structure change.
The new menu structure shows the technical details/directory names.
-Getting Started
+01 Getting Started
-Language Reference
+02 Language Reference

My concern here is that if we add a section a lot of links will break. I would prefer a system where the name the user sees is not the directory name. This will also permit us to use characters that can't be in a directory name, such as /.

Each file has a preamble (for example documentation/functions/domain/FRAME.md). Could this be part of the solution?

Determining the order of menu items is no longer possible.

That's also a problem. It is important that articles are listed in an order that improves usability/readability. For example, the more broad/introductory entries should be at the top of the list.

(Fun fact: When you are at a restaurant for the first time and don't know what to order, the first 1-2 items in any section of the menu are the "safe" items that a new and non-adventurous eater should pick. Likewise, Apple's style-guide says that the items in a drop-down menu should be in the order a typical user will learn them. That's why Open comes before Save, and the last few items in the drop-down menu are the obscure features that most people won't use)

External links Randomly adding external links, such as the GitHub releases, is no longer possible.

We only have 1 such link. No biggie. We could delete it or have a doc that tells people to go to the link.

Provider names that change

-Microsoft DNS Server on Microsoft Windows Server
+Msdns

This is fine. Though we probably want to shorten some of the full descriptions.

Mar 23 '23 18:03 tlimoncelli

Hi @systemcrash, your response feels personal. @tlimoncelli has asked me to provide feedback/view of this pull request. So I've listed the changes that aren't immediately obvious from the GitHub pull request description (or the big git diff). In addition, I explicitly asked how @tlimoncelli looks at these feedback points. Now it feels like we have to discuss each other's comments and that's not up to me.

Would I personally have done it differently? Yes, I would have chosen to narrow the scope down to what you call 'reference' (language reference and providers).

Am I grateful that you have taken up this work? Absolute!

Mar 23 '23 18:03 cafferata

Hi @systemcrash, your response feels personal. @tlimoncelli has asked me to provide feedback/view of this pull request. So I've listed the changes that aren't immediately obvious from the GitHub pull request description (or the big git diff). In addition, I explicitly asked how @tlimoncelli looks at these feedback points. Now it feels like we have to discuss each other's comments and that's not up to me.

Would I personally have done it differently? Yes, I would have chosen to narrow the scope down to what you call 'reference' (language reference and providers).

Am I grateful that you have taken up this work? Absolute!

Better we find a long-term solution than just mashing this in there, since I understand that you put in some work server-side to make this work out. And yes, it's personal: it's my opinion based on this work.

How about naming files with a number prefix for some kind of ordering, but the automation trims the number prefix away?

The only other way I can think of is embedding the title you want in the first line(s) of the .md files (in certain folders) and using those. More complex, but it seems to strike a balance. just have to go through all files and ensure the first line always adheres to this standard. ( Similar to the preamble @tlimoncelli mentions, but not for alphabetically sorted reference stuff )

Mar 23 '23 21:03 systemcrash

@systemcrash I appreciate all the effort you're putting into this. Good, readable, documentation is probably the most important way we bring new people into the project.

I like all the ideas so far. Here are my thoughts about them:

Filename prefixes: Use 3-digit numbers with big gaps between them (100_foo, 200_bar, etc) so that new files can fit between them. (Did you program in BASIC on an 8-bit computer?) The downside is that it may be confusing to not see the numbers in the URL.
File preamble: Add a preamble to the individual files: Hugo does this. The file has a preamble where you can set variables. For example add weight = 5 to a file's preamble and it will be listed before a file with weight = 6. Files without preambles are listed last.
A "toc file". That is... if a directory includes a file called toc.yaml, that file could list the filenames in the order they are to appear (files not listed can go at the end).

Thoughts?

Tom

Mar 24 '23 18:03 tlimoncelli

OK, updates:

The TOC auto-creation is reduced in scope now to only ref material, and providers.
Each section has its own  (replacement function more generic now).
I added a link rectifier which replaces links to other .md files with a relative link.
- This operates on files within specified folders - see docuLocalLinkRectifier.go and generate.go

As a result, all links work. TOC is auto-generated. Everybody is happy (I think).

(I may have missed a link or two somewhere).

renames/moves: providers -> service_providers functions (etc) -> language_reference

We can increase scope/complexity of auto-generation later with these ideas if we are satisfied with how this currently works

Mar 24 '23 20:03 systemcrash

Shall we use this automation?

Apr 03 '23 22:04 systemcrash

dnscontrol dnscontrol copied to clipboard

DOCS: TOC builder (for functions and providers)

dnscontrol
dnscontrol copied to clipboard