perf(snippets): simpler syntax for short snippets

[Kristinita]

1. Summary

It would be nice, if would be possible to have simpler syntax for short snippets.

2. Example

For example, I often use in my Markdown articles this native HTML constructions:

• <div class="SashaGingerinasExample" markdown="1">$1</div>
• <div class="SashaVideo"><iframe class="SashaLazy" data-src="$1"></iframe></div>

$1 — is a variable. I change only variable. Example:

<div class="SashaGingerinasExample" markdown="1">

Kristina

<div class="SashaVideo"><iframe class="SashaLazy" data-src="https://pe.ertu.be/videos/embed/4faa7a29-5937-40ce-8191-7b536e6296ec?start=4m24s">

</iframe></div>

Angel 💗

</div>

It a big construction, but I need to use it every time.

I create and use the 4 snippets. Updated part of the articles:

--8<-- "Example.md"

Kristina

--8<-- "Video.md"

https://pe.ertu.be/videos/embed/4faa7a29-5937-40ce-8191-7b536e6296ec?start=4m24s

--8<-- "VideoEnd.md"

Angel 💗

--8<-- "ExampleEnd.md"

But it still turns out to be long. It would be nice to have simpler method.

3. Example of expected behavior

In options:

'pymdownx.snippets': {
    'base_path': 'content/Snippets'
    'custom_snippets': {
        '=ex=': '<div class="SashaGingerinasExample" markdown="1">$1</div>',
        '=vi=': '<div class="SashaVideo"><iframe class="SashaLazy" data-src="$1"></iframe></div>'
    }
},

I can use syntax below and get result as in previous examples.

=ex=

Kristina

=vi=https://pe.ertu.be/videos/embed/4faa7a29-5937-40ce-8191-7b536e6296ec?start=4m24s=vi=

Angel 💗

=ex=

Only 16 symbols (2 =ex= and =vi=) for construction.

Thanks.

[facelessuser]

I'll admit that I definitely regret a little being too clever with the syntax of snippets. This is partly because coming up with a syntax I hadn't already used was difficult.

1. The idea of using a variable to represent the snippet isn't a bad idea. I'd prefer to settle on a very much known syntax instead of allowing anything though. Maybe define a variable ex and insert in the file as {=ex=} or something. I need to be able to reliably find it, so a very specific syntax is helpful for that.

2. The second idea, allowing snippets to be strings instead of a file, take it a little out of what snippets is, an external file importer. This would allow importing of strings or files. But maybe, it would be okay to allow this with a clear way to specify the provided string is a string and not a filename in the config.

3. The last idea, allowing variable substitution within the snippet, while I can see the usefulness, may be a bit out of scope for snippets. Snippets, at its heart, is simply a file import system. A way to have reusable snippets that can be imported (recursively even). This would move this to be something more like a template engine, which I am hesitant to open the doors too. This feels like a job for maybe some kind of Jinja2 extension, or something like that, which would use maybe meta data provided by the meta extension or something.

   I always just wanted a simple way to import external snippets, and while the current syntax to be too whimsical, I'm not sure I am up to supporting a template engine.

So, there are some interesting ideas here to unpack here, some of which I am really considering. I'm open to re-imagining what snippets is to some degree. If it deviates too far from the old version, maybe it could be a separate thing and over time we can deprecate the old version (as I do regret the current cutesy syntax), but I'm not sure if I want to go full template engine.

Thanks, for the suggestions, you've given me something to think about!

[akrisrn]

@facelessuser Hi! I have finished the work to pass the parameters to snippets before I notice this issue. I consulted Wikipedia Template syntax to implement it. Snippet content just looks like this:

{{1}} {{2}} {{3|c}} {{4|d}}

And the syntax:

--8<-- "snippet.md|a|b|c|d"
--8<-- "snippet.md|1=a|2=b|3=c|4=d"
--8<-- "snippet.md|a|b"
--8<-- "snippet.md|a|b|4=d"

All result is 'a b c d'. See more detail at this commit. I will be pleased to create a pull request if you agree.

[facelessuser]

@akrisrn, very nice. I think if, I end up allowing templates, I will release it under a new extension and rework the snippet syntax so that I can deprecate the old version. If we go that route, we may very well utilize something similar to the syntax you've shown, but we will drop the --8<-- syntax. I'm not sure what it would look like yet. Maybe something like {!filename.md|a|b|c|d!} (I want to avoid {{ }} in case someone has a mustache or Jinja2 extension conflict).

We could allow it work as inline as well, but would probably need to a way to specify this behavior. Probably a way to to turn off/on the syntax as well. Also a way to escape content that matches the syntax.

I'm warming up to this idea of templates, but if we do it, we probably won't allow for much more complexity than what is currently demonstrated in your work and described here.

We'll see. I need to time to look into this first.

[akrisrn]

@facelessuser Looking forward to new exciting template extension. Wikipedia Template may provide some help. I think markdown and wiki syntax could merge in some way. Thanks all of your work. I use almost all extension in pymdownx. They are really useful! By the way, --8<-- is pretty cute, I like this.

[facelessuser]

Wikipedia Template may provide some help. I think markdown and wiki syntax could merge in some way.

Yeah, I imagine it would be more a syntax inspired by Wikipedia Template than a direct port of it.

Thanks all of your work. I use almost all extension in pymdownx. They are really useful! By the way, --8<-- is pretty cute, I like this.

Thanks! I thought it looked nice when I created it. In retrospect, it is not the easy syntax to type though. I wish it was.

[Kristinita]

Still it would be nice.

Thanks.

[facelessuser]

It's on my list to look into, but I'm often pulled between the various projects I support and time with my family. Features like this also sometimes take longer for me to implement if I don't have a strong, personal need for them. Currently, I don't have a need for something like this in my documentation needs, so it isn't as high in the list as maybe something I actively need in this or other projects.

[Mattwmaster58]

Random observation markdown-include defines the following syntax for snippets: {!<file>!}

[facelessuser]

@Mattwmaster58 yes they do. We had gone a little more...cutesy if you will. I go back and forth on it. There's a part of me that likes --8<--, there's also a part of me that goes that thinks I need something more utilitarian. I guess we'll see which side wins if/when I get around to re-visiting this 🙂 .

[Mattwmaster58]

Would you be open to supporting both for a time? I would imagine it couldn't be all that hard to cook up a PR for the simpler syntax.

[facelessuser]

You do not like the whimsical? :slightly_smiling_face:

I haven't decided what I'm doing to be honest. Some people do like the current syntax. With that said, there are alternatives, as you've pointed out, for those who do not like the current syntax.

[michaelkrupp]

@facelessuser in case you go and take another look at this extension, you might be interest in this implementation of mine, based on your snippets extension:

https://gist.github.com/michaelkrupp/564e077eef44a9239b4e7ba7693fef0e

(Be aware though, that this was just quickly hacked together in one night and most likely has a ton of bugs still)

I have dropped the one-line syntax and added parameters to the blocks instead. Also added shell-style globbing and support for post-processing of snippets.

The latter one includes the return value of markdown.convert(raw_snippet) instead of the raw snippet itself (postprocess="true"). This is very useful for showing markdown source and converted result side-by-side, and was actually the initial reason for me to create this alternative implementation. I've also added support for pretty-printing of the post-processed result for good measure (prettify="true").

Also, I've made recursive evaluation of snippets optional (recursive="false") and added support for controlling the check_paths option on a per-snippet basis (check_paths="false").

--8<-- check_paths="true"
src/snippets/**/*.md
src/snippets/snippet-[0-9].txt
src/snippets/snippet-??.html
--8<--

This will render src/snippet.md as-is, without replacing any snippets in snippets.md. This makes the white-space based escaping syntax of the original extension obsolete:

--8<-- recursive="false"
src/snippet.md
--8<--

This will render the resulting HTML inside a fenced code-block alongside its markdown source and rendered result. I use this in combination with the Admonition and Tabs extensions to show the Markdown source next to the resulting output and generated (pretty-printed) HTML for our internal MkDocs documentation:

source:
~~~md
--8<--
src/example.md
--8<--
~~~

HTML:
~~~html
--8<-- postprocess="true" prettify="true"
src/example.md
--8<--
~~~

Result:
--8<-- postprocess="true"
src/example.md
--8<--

Maybe some of those changes might be of interest to you. Maybe one could even make use of the attr_list extension instead of a custom parameter implementation and by that also re-gain the single-line syntax:

--8<-- "src/snippet.md" {: postprocess="true" } 

I've also put in some basic (optional) path-checking, so one cannot include arbitrary files outside the base documentation directory (e.g. /etc/shadow).

On a final note: I have to say I really like the --8<-- syntax, as it is much easier to read than all the other include syntax options out there.

[facelessuser]

@michaelkrupp Thanks for the write up. This one has been lower on priority for me, but not forgotten. Part of the issue is mainly due to the fact it is not something that I have an immediate need for, but this info is nice and does present some interesting ideas.

I cannot promise the direction I will take, but seeing working mockups is always nice and helps.

On a final note: I have to say I really like the --8<-- syntax, as it is much easier to read than all the other include syntax options out there.

I'm glad you like it. I am a little partial to it myself. I've tried to talk myself out of it, but darn it, I still like it 🙂 .

[ak185158]

2. The second idea, allowing snippets to be strings instead of a file, take it a little out of what snippets is, an external file importer. This would allow importing of strings or files. But maybe, it would be okay to allow this with a clear way to specify the provided string is a string and not a filename in the config.

@facelessuser - On this idea of yours, would it be possible to make snippet block sources by name within a source file? For instance, I'm using a combination of snippets and the abbreviations extension to use tooltips throughout my site. Unfortunately, there's a lot of limitations doing it this way. Being able to designate a number of snippet strings (ideally supporting other markdown formatting within the "strings") by name would allow users to create a snippets document with multiple snippet sources within that could be included.

==snippet_source.md start==

-->8-- SnippetSample1
### **Snippet Sample 1**

*Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.*

- [Link](http://mytestlink.com/this_is_a_test.html)
-->8--

-->8-- SnippetSample2
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

- test info 
- more info
-->8--

==snippet_source.md end==

==snippet use==

Test document info ....

--8<--
snippet_source.md[SnippetSample1]
--8<--

Test document info continued...

--8<--
snippet_source.md[SnippetSample2]
--8<--

==snippet use end==

This would allow users to put smaller snippets into less files, which would make it easier to keep up with when using lots of different snippets used throughout documentation. This will make snippets much more useful for other scenarios, especially if other markdown formatting is still allowed within the snippet section. Even better, the ability to use a snippet within a snippet would make for even more dynamic content.

[facelessuser]

The thing about Snippets is that it is meant to be very simplistic in design. It's very dumb in its intelligence and is mainly a very simple include extension. This was the intention from the beginning.

There have been some requests for more template like behavior, and I've been sitting on the fence for a while on this. There are Jinja2 extensions out there, and if template behavior is desired, that may be a better option for people.

As for including a syntax to allow for parts of a file to be selected and included, I'm not so sure I would be going that route. I imagine there are some users who want to use Snippets in a far heavier manner than I do, and that is okay, but I'm not sure if I am willing to add the additional complexity needed to allow for containerization of text within a snippet so that they can be specifically selected out of a file. Most likely, we will continue to require each snippet to be a separate file. These can be organized in subfolders if that makes management easier.

While we've added some recent new improvements related to include source directories and may consider adding URL sources in the future, that may be as far as we go with snippets.

[facelessuser]

I'm going to go ahead and close this. We've recently been expanding Snippets to some degree with being able to extract line numbers from a snippet, marking and extracting specific sections, pulling snippets from URLs, and even adding an option to dedent sections automatically. Yes, we've stuck with the cutesy scissor approach. It's been like this for years, and unless I end up going creating some new templating approach, we'll probably just stick with the scissors, for better or worse :).

The original request in this issue was to rewrite using a different syntax. I just don't have the interest or the time to do so. We've been incrementally improving the current system, we'll see how far I'm willing to take it.