Feature/directives

[facelessuser]

Completely experimental. The idea is to (eventually) replace Admonitions, Details, and Tabs with a new directive format. We won't be killing off replaced legacy extensions, not for some time.

Pros:

• No indentation is needed.
• Won't mess up your syntax highlighter in your code editor due to said indentation.
• No more inventing new syntax for various new block features.
• We can reuse this syntax for figure captures, embedding videos, etc. Opens the door for cool new features with minimal effort.
• Any existing plugins converted to this format should have outputs that are compatible.
• Would resolve features where people want to add IDs to tabs, Details, etc. We could expose these in the config for the given directive.

Cons:

• New syntax to learn, but we won't be killing off legacy extensions right away.

What are we waiting on?

• [ ] Needs feedback
• [ ] Settle on what we call this extension (currently directives, but we could call them anything)
• [ ] Settle on the final format/syntax.
• [ ] Settle on the feature set/capabilities and API we provide.
• [ ] Provide tests and ensure that there aren't any big corner cases we are missing.
• [ ] Needs documentation

[facelessuser]

I'll duplicate this here as well in order to get feedback:

Just an overall description of how the current prototype is laid out. I think I'm ready to discuss the syntax.

:::{directive-name} arguments
---
option1: value
option2: value
---

content
:::

1. Fencing of blocks requires 3 or more :. If nesting blocks, the outer block should have a greater length of : than the child blocks. Currently, if a closing fence is encountered that is greater than or equal to the starting fence, it is sufficient to close the current fence.

2. directive-name is currently the name of whatever directive you are using. It is case insensitive.

3. arguments can be one or many arguments, and the given directive would specify required delimiters if required.

4. The optional frontmatter containing options is YAML-ish bock that comes immediately after the header, or better put, is part of the header when specified. We currently went with YAML-ish to avoid pulling in PyYAML as a requirement. I'm kind of going with simpler is better right now.

   Due to limitations of the Python Markdown parser, it must be a "tight" config (no empty new lines). This also means there should be no newline after the initial header line.

   The key value pairs are similar to the meta extension (though I haven't yet made them multi-line, was considering this).

   The directive would impose any specific rules that these options require: delimiter split list, etc.

5. content: any valid content can be used, even nested directives. There does not have to be a separation between the header and the content, but can be if desired.

Examples

Note with no content

:::{warning} All we need is a title!
:::

:::{warning} This is important!
Read this notice.
:::

:::{warning}
Arguments are optional with some directives!
:::

:::{figure} some/image.png
---
width: 300
---

I'm a figure caption, and you can specify optional settings for your directives as well!
:::

::::{tip} Here's a secret

:::{note}
You can nest directives as well
:::

::::

[facelessuser]

We could add the ability for these blocks to insert raw text. That is without an HTML wrapper. Why would we want to do this?

Well, IIRC, Python Markdown does not allow HTML except at the root indentation level. So under lists is impossible.

This should be pretty straightforward. If we mark a directive as being a text-only directive, we could simply store the content in the Python Markdown's stash and attach the placeholder to the end of the parent text or its last child's tail.

We could even add an event that allows you to process the text, such as formatting it for HTML, or just HTML escape it.

It's probably the only other case I can think of that people might want to do.

EDIT: I don't think we are going to bother with this right now. We don't want to blindly insert text.

[facelessuser]

The current syntax is what is up for debate. Do we keep the Myst like approach? Do we do something different? Under the hood, I think the functionality is all there.

[facelessuser]

Anyone checking this out should potentially voice their opinion syntax over at https://github.com/Python-Markdown/markdown/issues/1175 as we've already got a good conversation going, but here is fine as well.

[facelessuser]

Chunks are handed to parseChunk, but a chunk could have multiple blocks separated by \n\n. Maybe we should split these before calling on_add? This still won't catch cases where a block could be split automatically later:

Text
!!! note "Legacy admonition"
    content

We allow the same as above for generic blocks, but this would be at least two separate blocks.

Text
/// note | Legacy admonition

content
///

[Andre601]

Regarding the blocks in blocks... I feel like it could become a bit tedious if you for some reason have a lot of nested blocks, to constantly expand the parent by one more :

Wouldn't something similar to the tabbed extension be a solution? By that I mean have a specific character to indicate a new box start.

For the sake of it do I use + here as the indicator.

::::{info} Block

:::{info} Nested block
Hello!
:::

::::

would become

:::{info} Block

:::+{info} Nested block
Hello!
:::

:::

It's a small change, but I feel like it would be a bit better to have this:

:::{info} Extreme nest example

:::+{info} Nested block 1

:::+{info} Nested block 2

:::+{info} Nested block 3

:::+{info} Nested block 4

Hi!

:::

:::

:::

:::

:::

...than this

:::::::{info} Extreme nest example

::::::{info} Nested block 1

:::::{info} Nested block 2

::::{info} Nested block 3

:::{info} Nested block 4

Hi!

:::

::::

:::::

::::::

:::::::

[pawamoy]

I'd say it's easier to find the corresponding start/end of a block when they increase/decrease the number of :.

[2bndy5]

Just looking at your example, I think it is a bit more human readable if the nested blocks have an extra : added to their parents. Mainly because I easily get confused which ::: ends which nested block if it only starts with ::: or :::+. That would be even harder to track when writing lots of content in nested blocks.

[facelessuser]

Currently, I'm of the mind of increasing tokens makes things more readable. Nesting will always borderline on becoming less readable, especially when doing deep nesting, but anything to help ease that I'm all for.

I know that ::: keeps getting referenced, but currently we are using /// just an FYI. Feel free to voice your opinion about ::: vs ///.

The original discussion (https://github.com/Python-Markdown/markdown/issues/1175) led us to use /// instead, but other opinions are welcome as well.

[2bndy5]

I'm good with /// because it means less wear-and-tear on my shift button.

[Andre601]

Personally, I'm more in favour of :::, mainly because it has established itself as a more or less common syntax for admonitions. It was even mentioned in a GitHub community discussion multiple times as the "right way to make admonitions"

Regarding the incrementing colons... Would it be not a bit better/easier to look at if it was in reverse order? Like have childs be increased by 1 compared to parents? Looking at my example, you can see that the most outer start and end are fairly appart and it looks a bit confusing...

The original discussion (https://github.com/Python-Markdown/markdown/issues/1175) led us to use /// instead, but other opinions are welcome as well.

Link is broken. It's /here instead of /1175

[2bndy5]

Would it be not a bit better/easier to look at if it was in reverse order?

I think it would be easier for authors to write this way. And it still maintains readability.

I'm more in favour of :::, mainly because it has established itself as a more or less common syntax for admonitions.

I thought the goal was to break-away from that in favor of a definitive uniformed syntax. If it is modeled after existing syntax, then you risk confusing those who haven't migrated. (see also https://github.com/Python-Markdown/markdown/issues/1175#issuecomment-1222391419)

[Andre601]

I thought the goal was to break-away from that in favor of a definitive uniformed syntax. If it is modeled after existing syntax, then you risk confusing those who haven't migrated. (see also Python-Markdown/markdown#1175 (comment))

From what I read are most people referencing ::: with admonition, which would be the prime use here... Like people did mention Pandoc, but also more popular solutions such as Docosaurus use ::: to indicate admonition blocks here, so if anything would it actually help establishing a more common syntax for markdown (if that isn't already the case as some MD renders do use ::: as official admonition syntax from what I know).

[2bndy5]

It would be cool if it was configurable, but I imagine the choices for such an option would have to be limited to regex friendly chars.

[facelessuser]

I will be releasing a prerelease hopefully sometime soon. The syntax could change for the final release, but I would need a strong majority. I'm personally indifferent to which form gets used /// vs :::. There are some other potentially things that could change as well. I think when I do the prerelease, I will specifically point out some of the things I absolutely want feedback on, but any feedback is of course welcome.

[facelessuser]

One thing that will need a decision is whether block specific options should require yaml fencing or have it optional. Optional means we must require a new line between the block header and the content, but mandatory yaml fence means that a new line is not required.

I thought I'd be fine with optional, but I personally disliked it more than I thought I would. I'm leaning towards mandatory. Currently, at least for the prerelease, this will be optional so people can try it and help decide which is better.

[2bndy5]

Coming from rST, I'm rather used to putting a blank line between directive options and it's content. On the other hand, I do like having explicit delimiters that tell readers/contributors where the actual content begins and where the options end. If the directive options use YAML syntax, then it makes sense to use fences because YAML is rather easily mistakable for natural text.

[facelessuser]

It's not the new line between config and content that's the problem, it's that it isn't easy to tell config from content, so even if you have no config, you have to have a new line.

/// note | title

New line required above
///

With a distinct fence around the config, you are always sure of when a config exists and can avoid the new line requirement:

/// note | title
content
///

But maybe this doesn't bother everyone.

[2bndy5]

If the title is allowed to span multiple lines in the doc src, then I'm also leaning toward mandatory fencing.

[facelessuser]

The "title" does not span multiple lines. That string can be used for anything though, it is dependent on the block:

/// tag | div
Div content
///

But no, the "title" cannot span multiple lines.

[2bndy5]

At the risk of making a pun, I'm on the fence. I'm leaning toward mandatory for readability reasons, but it would probably be best to get my hands dirty first.

I just wrote overrides for Sphinx admonitions in which multi-lined (and optional) titles was a hard feature to support due to docutils' rST parser. We ultimately went with a new :title: option that can concatenate with directive arguments.

[facelessuser]

Yeah, that's why I'm making it configurable via the prerelease just to get feedback. I think it may be hard to tell what you'll like or hate until you try it.

[2bndy5]

Would it be not a bit better/easier to look at if it was in reverse order? Like have childs be increased by 1 compared to parents

On second thought, I'm not sure I like this idea for parsing reasons. If a block can have children, then the furthest descendant is /// which I imagine would signify to the parser that the block shall have no children. If it was reversed, then there's nothing in the syntax to prevent the parser from looking for children.

I have an idea (for a plugin) that I want to try on the eventual RC

/// {render-snippet}

/// {note}
Render this code as literal snippet and again as the resulting admonition.
///

///

The idea is to bypass having to write demonstrations of MD syntax in duplicates. Currently, the Material theme has to do

``` markdown title="Admonition"
!!! note
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla et euismod
    nulla. Curabitur feugiat, tortor non consequat finibus, justo purus auctor
    massa, nec semper lorem quam in massa.
```

<div class="result" markdown>

!!! note

    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla et euismod
    nulla. Curabitur feugiat, tortor non consequat finibus, justo purus auctor
    massa, nec semper lorem quam in massa.

</div>

But with a block that accepts no children, the docs' src can be reduced using a custom directive (named render-snippet in my example).

[facelessuser]

The idea is to bypass having to write demonstrations of MD syntax in duplicates. Currently, the Material theme has to do

It'll probably be possible to some extent:

//// render-snippet

/// note
Render this code as literal snippet and again as the resulting admonition.
///

////

You could store the content in a raw code block, and then using an on_end event extract that text and run it back through the parser for the results. You always run the risk of not catching the content before changes if a preprocessor plugin got to it first as these generic Blocks run as a blockprocessor which runs after preprocessor. We do at lest unwind what SuperFences does if saving content as "raw" text.

I do use a special custom fence in SuperFences that allows me to execute and render Markdown in an isolated sandbox. It allows me to demonstrate Markdown using different options as the content is isolated:

```md-render
---
extensions:
- pymdownx.magiclink
- pymdownx.saneheaders
- markdown.extensions.attr_list

extension_configs:
  pymdownx.magiclink:
    repo_url_shortener: true
    repo_url_shorthand: true
    social_url_shorthand: true
    social_url_shortener: true
    user: facelessuser
    repo: pymdown-extensions
---
- Just paste links directly in the document like this: https://google.com.
- Or even an email address: [email protected]{.magiclink-ignore}.
```

There is some limitations of course. As it is isolated, you have to be careful as you could generate conflicting IDs, and some preprocessing could get a hold of the text before you can process it. I don't generate both the source in a code block and then the results, in this case, I just render the results, but what you describe could be done with custom fences.

[pawamoy]

(I've implemented a similar fence that allows to render both the source and result, see https://pawamoy.github.io/markdown-exec, in particular the literate markdown part https://pawamoy.github.io/markdown-exec/usage/#literate-markdown)

[facelessuser]

Yep, I keep forgetting, but https://pawamoy.github.io/markdown-exec/ does probably already offers what you need in an already-packaged way. There is quite a lot you can do with custom fences. For example, I also created interactive playgrounds that translate code into REPL outputs and preview colors. Then we load Python in the browser (using Pyodide) and allow the viewer to interact and execute code on the fly.

Basically, there is a lot of potential already available via custom fences and SuperFences, and markdown-exec seems to leverage some of that already. Custom fences also capture the content pretty early in a pre-processor. Generic blocks will capture a bit later in the process but allow for translating and wrapping Markdown into more HTML constructs.

[2bndy5]

Good to know. Thanks. markdown-exec does seem to offer exactly what I wanted to do.

[facelessuser]

Both ::: vs /// syntax and mandatory vs optional YAML config fences will be configurable during the alpha/beta stage so that people can try both and help give feedback as to the direction we should take.

[facelessuser]

The first alpha will be focused on syntax feedback. I'm not going to bother documenting the API or anything initially. It will be more to give people a chance to get a feel for the generic blocks. It will implement admonitions, details, tabs, and arbitrary HTML.

[pawamoy]

Thinking about graceful degradation (rendering blocks where they are not supported, for example on GitHub), I think enforcing a new line can be useful. Consider these two examples and how they would render without Blocks support:

1.

/// note
A note about something.
///

Rendered as:

/// note A note about something. ///

2.

/// note

A note about something.
///

Rendered as:

/// note

A note about something. ///

---

I believe the second example is easier to read, i.e. the degradation is more graceful.

Just a thought :smile:

EDIT: GitHub might not be a good example, because I think they always preserve new lines now :thinking: Also, even if a new line is not required, and new lines are not preserved by the Markdown renderer, users can force a more graceful degradation with trailing spaces:

/// note__
A note about something.__
///

(spaces are represented by underscores here)

Rendered as:

/// note A note about something. ///