Add templating to publishing functionality

[wojciechczerniak]

Hello,

This time a feature request.

Problem

I've seen that there are few requests/issues with customization of the Markdown output:

• [x] #166
• [x] #295
• [x] #469
• [ ] #513
• [x] #314

And I would have few others requirements:

• Start from 2nd lvl header
• Format links as list
• Add custom class to up/down links
• Insert table of contents
• Various smaller formatting adjustments

But IMO a change from string building to one of the templating languages may solve all of the above in a one go

Solution idea

I think Jinja2 is the number one for Python libraries? With a template and an ability to use our own fixing all of the above issues would be just a matter of customization with --template my-markdown.jinja2

Remove hard-coded string building in HTML publisher: https://github.com/doorstop-dev/doorstop/blob/a770dc4566a43de87da0e5be61dfedf87a7ac57d/doorstop/core/publisher.py#L165-L167 and Markdown markdown publisher: https://github.com/doorstop-dev/doorstop/blob/a770dc4566a43de87da0e5be61dfedf87a7ac57d/doorstop/core/publisher.py#L399-L409

With Jinja2 templates defaulting to whatever is in those methods:

# {{ document.header }}

{% for item in document.items %}
  ## {{ item.header }} <small>{{ item.id }}</small> { #{{ item.id }} }
  
  {{ item.text }}
  
  {% if item.PUBLISH_CHILD_LINKS %}  
  Links:
    {% for link in links %}
        [{{ link.header }}]({{ link.file }}#{{ link.id }})
    {% endfor %}
  {% endif %}
  
  {% if item.document.publish %}  
        | Attribute | Value |
        | --------- | ----- |
    {% for attr in item.document.publish %}
        | {{ attr }} | {{ item.attribute(attr) }} |
    {% endfor %}
  {% endif %}
{% endfor %}

AND allow the --template arg for all output formats.

doorstop publish -m all docs/requirements/ --template my-template.jinja2

Use cases

You want Frontmatter/YAML headers in Markdown #295, just add them to the template:

---
title: {{ page.header | title() }}
metadata: {{ page.description | truncate(240) }}
---
{% block content %}
.... content

Footer maybe as in #166? Add it at the end:

.... content
{% endblock  %}

Copyright Year, Restrictions

[eterX]

Hi Wojciech

I realize that it would be a very useful feature. I've been using something like that, for communication documents (but without Jinja, your approach looks far more sound)

As soon as a I get #507 done, count on some help on this :)

Leandro

[eterX]

Hi!

As promissed, started exploring this feature along with @lucasd92, now that #507 is finally waiting to be merged, (maybe @jacebrowning ?) :) @lucasd92 managed to replace Bottle's default template engine with Jinja2. Please, see this fork (branch feature/jinja) that can give us a glipse of the feature's potential, by:

• issuing the command doorstop publish all /mydir -H --template test_jinja2
• tweaking views/test_jinja2_index.tpl
• modifying the test at test_all.py

Looks promissing!

First pitfall we came across: Jinja2's output can be consumed as an iterator, and Doorstop expects that. Thusly allowing for iterator chains and coroutines. But it looks like iterators composition is lost in Bottle.Jinja2Template. In spite of the description saying "Get a rendered template as a string iterator" it calls render() instead of generate() with no altarnetive. Workarounded here, somewhat piercing Bottle's abraction. IMHO, it might not be an important difference in terms for small/big projects, but for future API endpoints. Maybe more if the project moves to FastAPI and async methods?

Best regards,

[monch1962]

I'm also looking into options to use customised reporting templates with Doorstop.

Given the challenges @eterX outlines above, would it make more sense to build a separate, template-based Doorstop reporter and leave the current Doorstop reporting as is? Doorstop's YAML data format is elegant and extensible, and I think the extensibility is where the magic is.

For example, I'm currently using Doorstop with custom requirements fields that I'll want to use every time (e.g. classifying each requirement using FURPS+), but also with fields that will change from project to project. These could include e.g.

• requirement-source-url field to link to external requirements documents,
• requirement-source-category field to describe whether individual requirements are regulatory, project-specific, enterprise-standard, industry best practice, etc.
• a jira-url array that link to any Jira stories that describe how the requirement is implemented
• a terraform-url array that link to any Terraforms that can be checked to validate that specific requirements are met

Given how well Doorstop's extensibility supports this approach, it makes sense to have reporting templates be similarly customisable to specific use cases. There's also Doorstop APIs that give access to all the custom fields, which make it easier still to implement.

I'll knock something simple together and post it here as an example

[monch1962]

Jinja is a pretty handy templating engine for Python. This approach lets you use Jinja templates with doorstop:

#!/usr/bin/env python
import argparse
import doorstop
import sys
from jinja2 import Template, Environment, nodes
from jinja_markdown import MarkdownExtension

with open("./templates/report.html.jinja2", "r") as f:
    # This would work if you didn't have to load a Jinja extension i.e. MarkdownExtension
    #t = Template(f.read())

    # This is the approach to consume a template that includes {% markdown %}..{% endmarkdown %} tags
    s = f.read()
    t = Environment(extensions=[MarkdownExtension]).from_string(s)
output = t.render(dependency_tree=repr(tree))

with open("./dist/rendered.html", "w") as f:
    f.write(output)

Inside your Jinja template, you can then use something like

<h2>Dependency tree: {{dependency_tree|e}}</h2>

{% markdown %}
---
Start of markdown content

# H1 heading
## H2 heading

- list item
- another list item

End of markdown content
---

{% endmarkdown %}

to present your doorstop data. Note that you might need to escape your doorstop data if it contains certain characters, which is why I've got {{ dependency_tree | e }}. Note also that adding the Markdown extension to Jinja lets Jinja work with Markdown content, which you might be using inside your doorstop YAMLs.

Hopefully I haven't cut down this code too much to make it understandable.

Now I just need to work out how to walk the tree of linked documents using the doorstop scripting interface...

[yoyodyn]

Would it be possible to specify the publishing template within the .doorstop.yml file for each document/type?

[neerdoc]

@wojciechczerniak Could you have a look at version v.3.0b12 and see if you think this version solves this issue?