Custom Jinja2 filters (feature request)

[emptywee]

Allow st2 users extend the power of Stackstorm by letting them keep custom jinja2 filters in some directory that st2 would look into. That would add a bit of flexibility extending result formatting capabilities and so forth.

[lakshmi-kannan]

@emptywee Please look at https://docs.stackstorm.com/latest/reference/jinja.html#applying-filters-with-jinja and let me know if this is what you want. It looks like you want users to define custom filters in a pack. This could lead to collisions unless we namespace by pack. Then the syntax would look awful. Plus, I really think if useful jinja filters are bubbled up to platform level then we can control the filter behavior.

[emptywee]

Yeah, that's something that could be really useful. Just please make sure that the files with custom filters placed in the st2/st2common/st2common/jinja/filters/ directory are not removed during upgrade of Stackstorm installation.

[LindsayHill]

@s2ugimot Does this look like something similar to what you are after?

[shusugmt]

@LindsayHill Yes this is exactly what I wanted

[shusugmt]

What I wanted to do originally is to add a small custom filter which does something like | trim | d('my_value', true) to cleanup my action yaml definition.

What I have currently in my yaml looks like this:

(snip)
parameters:
  hosts:
    type: string
    immutable: true
    default: "{{ system.starrod.config.host | trim | d('ixops-mock', true) }}"
  username:
    type: string
    immutable: true
    default: "{{ system.starrod.config.user | trim | d('naptan', true) }}"
(snip)

Looks bit verbose and ugly. My intention here is to give a default value to the parameter if there is nothing set in st2 key, but the value returned by accessing none-existent key like system.some.none-existent.key seems to be something other than nil, so I need to pass trim first.

[roberterdin]

Is there an update on this? That would quite useful. Having the files in st2/lib/python2.7/site-packages/st2common/jinja/filters/ does not work because they are loaded here: https://github.com/StackStorm/st2/blob/master/st2common/st2common/util/jinja.py#L46

[peschmae]

Being able to add custom Jinja filters in a pack would be great improvement, to templating for ChatOps.

We're using Ansible for lots of our ChatOps commands, since we want to be able to reuse those commands/tasks outside of Stackstorm.

Displaying the Output of a Ansible command using Action-Aliases results in a lot of duplicate Code, since we basically always need to get a Task from the JSON, and display stdout/stdout_lines.

It would also allow for easier error handling based on the Stats Ansible return.

I think namespacing the Jinja filter, would still result in more readable templates, than doing the same stuff in every action-alias, while still allowing to use the same names for filters in multiple packs.

It might make sense, to add the filters to the pack.yaml, instead of loading just all files in a filters directory within the packs.

I imagine something like this in the custom_chatops/pack.yaml

filter:  
  trim: text.trim  
  stdout: ansible.stdout  
  stdout_lines: ansible.stdout_lines  

Resulting in filters like

custom_chatops.trim()
custom_chatops.stdout()
custom_chatops.stdout_lines()

As far as I remeber '.' is a valid character for filter names. Otherwise an underline would work as well for a separator.

[estee-tew]

https://stackstorm.slack.com/archives/community/p1491750938004615

[nmaludy]

+1

[mickmcgrath13]

+1

[dtvuong]

+1

[cognifloyd]

So, there are a variety of different contexts for jinja2 filters.

• in metadata files (Jinja2-only)
  • Action parameter defaults: https://github.com/StackStorm/st2/issues/2956#issuecomment-254687509
  • Action notifications
  • Action alias results: https://github.com/StackStorm/st2/issues/2956#issue-183282874 & https://github.com/StackStorm/st2/issues/2956#issuecomment-277088712
  • Action alias immutable params
  • Action alias ack & extra
  • Rule criteria
  • Pack Config
• in ActionChain workflows (Jinja2-only)
• in Orquesta workflows (Both Jinja2 & YAQL)

In Orquesta, this is already extensible because orquesta uses a stevedore extension point to load expression functions: https://github.com/StackStorm/st2/blob/c172459b61ddb5fa6f1749bffad4373983bd7f7d/contrib/runners/orquesta_runner/setup.py#L54-L55

So, to add functions to Jinja2 & YAQL expressions in Orquesta:

1. build a python package that adds additional orquesta.expressions.functions entry points, and
2. install that package in the main st2 virtualenv /opt/stackstorm/st2.

To allow the same for Metadata (and ActionChain), we could modify st2common.utils.jinja to load expression functions from a stevedore extension point like st2common.expressions.functions.

But, I don't see a good way for packs to take advantage of the stevedore extension points as they can't install packages in the st2 virtualenv (for good reason!). That applies to both the current orquesta extension point, and my proposed st2common one. So, it's not quite a complete solution.

[cognifloyd]

Regarding loading python code from packs: Ansible has a similar problem with loading plugins and modules from various "collections". They used a PEP 451 loader to dynamically add virtual modules under the ansible.collections namespace for python code in the collections. They also have a plugin whitelist so that certain kinds of plugins can only be loaded if enabled in the config.

So, what if packs could define their entry points in packs.yam. They would be disabled by default, and therefore the python code would not be discoverable by stevedore. Once enabled, the filters would be loaded into st2packs.<pack_name>.expressions.functions with a new PEP 451 loader.

This could also allow packs to distribute other stackstorm components, such as runners, auth or rbac backends, etc. The 451 loader would decide what could and could not be added via packs, and then load them into standardized module names

content	importable python module (once loaded from pack)	stevedore extension entry point
Orquesta Expression Functions	st2packs.<pack_name>.orquesta.expressions.functions Should orquesta funcs be separate from metadata funcs?	orquesta.expressions.functions
Metadata & ActionChain Expression Functions	st2packs.<pack_name>.expressions.functions Should ActionChain funcs be the same as Metadata funcs?	st2common.expressions.functions NEW
Runners	st2packs.<pack_name>.runner	st2common.runners.runner
Auth Backends	st2packs.<pack_name>.auth.backend	st2auth.backends.backend
RBAC Backends	st2packs.<pack_name>.rbac.backend	st2common.rbac.backend
pack lib dir	st2packs.<pack_name>.lib A common spot for pack-specific code that can be reused in sensors, python actions, and the extension points listed above. Loaded automatically only for actions/sensors in the same pack, and in st2 virtualenv only when other content (runner, backend, etc) is enabled.	none

[cognifloyd]

I really think if useful jinja filters are bubbled up to platform level then we can control the filter behavior.

To paraphrase @lakshmi-kannan: "filters should end up in core st2". But, I don't think that always makes sense. Here's my use case where a pack-provided filter/expression-function would be ideal, and including it with core st2 would not make sense.

use case for pack-provided filters / expression-functions

I would like to add a vault function in the vault pack that would work like the st2kv function, but pull data from hashicorp vault instead of the datastore.

With a vault function, the credentials would not be visible by default when looking at a workflow's output through the cli or the webui. As is, I have to use an explicit vault.read action (which is analogous to st2.kv.get with decrypt=true), so there's not a good way to mask the returned credentials.

[chris3081]

I know its been a while since this issue has been touched but I've got a similar requirement on the vault access, so I've been looking at how to implement it so I can put in a PR. I've been trying to figure out the best place to add the importlib, my thinking is it would be in the runner init, would this be logical (https://github.com/StackStorm/st2/blob/b9aec99fc49507d0b5b199caa2b1c27f7f5e13a6/contrib/runners/orquesta_runner/orquesta_runner/orquesta_runner.py#L46)?

The question I have is how to do this when packs have their own virtualenvs?