Preserving whitespace prefix in multiline strings

[ttsiodras]

trafficstars

In the StringTemplate engine - which I've used in some projects to emit C code - whitespace prefixes are automatically added in the output lines:

PrintCFunction(linesGlobal, linesLocal) ::= <<
void foo() {
    if (someRuntimeFlag) {
        <linesGlobal>
        if (anotherRuntimeFlag) {
            <linesLocal>
        }
    }
}
>>

When this template is rendered in StringTemplate, the whitespace prefixing the multilined linesGlobal and linesLocal strings, is copied for all the lines emitted. The generated C code is e.g.:

void foo() {
    if (someRuntimeFlag) {
        int i;
        i=1;   // <=== whitespace prefix copied in 2nd
        i++;   // <=== and 3rd line
        if (anotherRuntimeFlag) {
            int j=i;
            j++; //  <=== ditto
        }
    }
}

I am new to Jinja2 - and tried to replicate this:

#!/usr/bin/env python
from jinja2 import Template

linesGlobal='\n'.join(['int i;', 'i=1;'])
linesLocal='\n'.join(['int j=i;', 'j++;'])

tmpl = Template(u'''\
void foo() {
    if (someRuntimeFlag) {
        {{linesGlobal}}
        if (anotherRuntimeFlag) {
            {{linesLocal}}
        }
    }
}
''')

print tmpl.render(
    linesGlobal=linesGlobal,
    linesLocal=linesLocal)

...but saw it produce this:

void foo() {
    if (someRuntimeFlag) {
        int i;
i=1;
        if (anotherRuntimeFlag) {
            int j=i;
j++;
        }
    }
}

...which is not what I want. I managed to make the output emit proper whitespace prefixes with this:

...
if (someRuntimeFlag) {
    {{linesGlobal|indent(8)}}
    if (anotherRuntimeFlag) {
        {{linesLocal|indent(12)}}
    }
}

...but this is arguably bad, since I need to manually count whitespace for every string I emit...

Is there a better way that I am missing?

[jgehrcke]

I am interested in basically the same thing. The issue has also come up at stackoverflow: http://stackoverflow.com/questions/10821539/jinja-keep-indentation-on-include-or-macro

+1

[maxime-esa]

Also true when using the form:

    {{linesGlobal|join('\n')}}

This form does not behave as one would expect - since jinja2 is the one emmitting the newlines, it should make sure that they remain aligned with the last indentation level.

[fmarczin]

This would be very nice to have! It would lead to much nicer templates and rendered output at the same time.

Why not create another whitespace option similar to {%+ and {%- that prepends the current indentation on whatever it evaluates to? Could be {%= or {%|

+1

[mpaolini]

+1

needed here for templating API blueprint documentation:

{% macro entity_one() -%}
{
    "one": 1,
    "two": 2
}
{% endmacro -%}

+ Response 200 (application/json):

        {
            "entity": [
                {{ entity_one() }}
            ]
        }

is rendered now :

+ Response 200 (application/json):

        {
            "entity": [
                {
    "one": 1,
    "two": 2

}

            ]
        }

[inducer]

When emitting YAML or Python, this becomes pretty crucial.

[DataGreed]

Ran into the same problem.

Is there a workaround for now other than defining a macro for every included tempkate and manually entering the indentation?

[kaikuchn]

Sorry for reviving this old issue, I just came across the same problem and googling brought me here. After some more looking around I found that by now there is a nice way to achieve this through the indent filter

[Cigizmoond-Vyhuholev]

@kaikuchn Thank you, dude! It works.

[ttsiodras]

@kaikuchn , @Cigizmoond-Vyhuholev Guys, I am not sure I follow... as you can see in my original report at the top, I do mention a workaround with the indent filter - but also clearly state that it doesn't address the issue in the simple and powerful way that e.g. StringTemplate does, because it forces you to count indentation spaces every time you need to emit a block of lines... If you have to do that and you are generating code of any form (C, Python, whatever) you'll very quickly just abandon the process altogether...

Then again, I may have misunderstood what you meant... Can you share exactly how you'd implement my original requirement shown at the top? i.e. generate the same kind of output with Jinja2 syntax? This is what I don't like...

if (someRuntimeFlag) {
    {{linesGlobal|indent(8)}}
    if (anotherRuntimeFlag) {
        {{linesLocal|indent(12)}}
    }
}

...because I need to count the "8" and "12" in each and every template where I emit code. In comparison, in StringTemplate...

PrintCFunction(linesGlobal, linesLocal) ::= <<
void foo() {
    if (someRuntimeFlag) {
        <linesGlobal>
        if (anotherRuntimeFlag) {
            <linesLocal>
        }
    }
}
>>

[septatrix]

I noticed that #919 was closed due to a code change which would apparently require some major refactoring of the PR. However I would really like to see this feature implemented in my favourite templating engine.

If this is still something the core devs would like to see implemented (the community sure wants it as it is the PR and open issue with the most thumbs up) I would be eager to help and maybe even try to implement this on my own.

[cheshirekow]

Would also love to see this feature. I'll note that the indent workaround can't be used when the indentation level isn't known. e.g.:

{{field.type}} {{field.name}}; {{field.comment|indent(??)}}

Where the indentation level to be preserved depends on the length of the first two value.

[yajo]

I have one hypothesis. What about left-trimming 1st level of indentation in block declaration?

Example:

{%- macro some_yaml_block(sub_block) ~%}
  label indented with how many spaces: 2
  amount of spaces that will be trimmed due to that "~" char above: 2
  {%- if sub_block ~%}
    "yay! we can indent ifs!": true
    the minimal indent in this if block is 2, so:
      this extra-indented value is still indented properly
  {%- endif %}
{%- endmacro -%}

now we invoke that here:
  {{ some_yaml_block(true) }}

The rendering would be:

now we invoke that here:
  label indented with how many spaces: 2
  amount of spaces that will be trimmed due to that "~" char above: 2
  "yay! we can indent ifs!": true
  the minimal indent in this if block is 2, so:
    this extra-indented value is still indented properly

Basically, when finishing a block with ~%}, Jinja would:

1. Remove the 1st characters if it is a EOL sequence, as indicated in the newline_sequence Environment parameter.
2. Render the content internally.
3. Count how many common whitespace characters prefix those rendered lines.
4. Strip them from the beginning of each line.

If you, later, need to include this block with some specific indentation, all you need is to call it like some_yaml_block|indent. Since the indent was normalized at block declaration, you can later specify it without problem. And the block would behave consistently across calls.

[stereobutter]

This feature would be super useful for folks using jinja to template config files which nowadays almost always involves yaml somewhere. Why not use yaml-specific tooling you ask? A lot of tools aren't using just yaml but a mix of different config formats. For instance butane the config format for Fedora CoreOS uses yaml itself but needs to play with other tools like systemd that use other file formats for their configuration. Using jinja to template this out and assemble the file using include would be a nice workflow were it not for the indentation issues with multiline include blocks.

[stereobutter]

For anyone interested I've written an extension that works around this long standing issue so that I may use jinja2 for templating yaml config files. It works by hooking intopreprocess to transform something like {% include "template.j2" indent content %} into a regular include statement with an appropriate {% filter indent(...) %} block that indents the included content accordingly. Somewhat of a gross hack? Maybe?! Does it work? Yup.

[milahu]

What about left-trimming 1st level of indentation in block declaration?

+1

should be optional, similar to the trim_blocks and lstrip_blocks options the new option could be called dedent_blocks or unindent_blocks

probably this should throw error on mixed indent (tabs vs spaces), similar to python

related issue on stackoverflow: Jinja2 correctly indent included block

You can apply filters to whole text blocks: Template Designer Documentation / Filters

{% filter indent(width=4) %}
{% include "./sub-template.yml.j2" %}
{% endfilter %}

[rightaway]

@davidism Do you have any comment on https://github.com/pallets/jinja/pull/1456 which helps with this? This issue is 10 years old and https://github.com/pallets/jinja/pull/919 was closed. Can https://github.com/pallets/jinja/issues/178#issuecomment-1066647157 be included as the fix?

[stereobutter]

@rightaway https://github.com/pallets/jinja/issues/178#issuecomment-1066647157 would only work for include blocks though. Getting this to work for extends would also be useful and I‘ve tinkered a bit with this already but it isn’t pretty. This is to say that fixing the underlying problem of jinja2 not dealing with indentation correctly would be infinitely better.

[GergelyKalmar]

I think that the solution that I've started working on some time ago in https://github.com/pallets/jinja/pull/1456 would be an elegant way of solving this issue, essentially only requiring adding the indent_blocks environment parameter to get automatic indenting/dedenting while rendering. I managed to make an implementation that worked for the majority of cases, however, there were still some more complex nested cases where it didn't work yet.

I only had a weekend to work on it though and I'm not very familiar with the inner workings of everything, nonetheless, if someone could finish that work or provide some input on the PR properly that would be awesome.

[sunhaitao]

I believe I found an elegant way to work around this: by calling template.generate creatively.

The details can be found in this demo:

import jinja2

def j2_gen(template, context, key_for_extras="__EXTRAS"):
    """Call `template.generate` with extras."""
    extras = context.get(key_for_extras, {})
    extras.update({
        "current_column": 0
    })
    newline_sequence = template.environment.newline_sequence

    for chunk in template.generate(
        {**context, key_for_extras: extras}
    ):
        yield chunk

        index = chunk.rfind(newline_sequence)
        if index == -1:
            extras["current_column"] += len(chunk)
        else:
            extras["current_column"] = (
                len(chunk) - len(newline_sequence) - index
            )

t = jinja2.Template(
"""\
{% macro render_lines() -%}
ABC
  XYZ
{%- endmacro -%}
{{" " * indent}}{{render_lines() | indent(__EXTRAS.current_column)}}
"""
)

for i in range(3):
    print("---+" * 4)
    print("".join(list(j2_gen(t, {"indent": 4 * i}))))

It prints the following text:

---+---+---+---+
ABC
  XYZ
---+---+---+---+
    ABC
      XYZ
---+---+---+---+
        ABC
          XYZ

I think whether to introduce a similar mechanism to Jinja is open to discuss, but adding some tests to ensure this always works seems to be a good idea.

[milahu]

btw, this works in nickel

Multiline strings are "indentation-aware". This means that one could use an indented string interpolation and the indentation would behave as expected:

let log = m%"
if log:
  print("log:", s)
"%m in m%"
def concat(str_array, log=false):
  res = []
  for s in str_array:
    %{log}
    res.append(s)
  return res
"%m

def concat(str_array, log=false):
  res = []
  for s in str_array:
    if log:
      print("log:", s)
    res.append(s)
  return res