Add custom functions to the Jinja context

[edisongustavo]

Problem

Programming in Jinja is not much fun, so if we need to do more complicated functions, the code can be quite complicated (or impossible).

Take this example (which doesn't work and just frustrates the user):

{% set versionformat = lambda ver: '.'.join(ver) %}
version = {{ versionformat(31) }}

Proposal

It would be great if I could have files (*.conda-devenv.py) with:

def versionformat():
    """Convert string `ver` to a semantic version formatted string if not already"""
    # Case 1: if ver is '372' (string without dots), return '3.7.2'
    # Case 2: if ver is '3.7.2' (string with dots), return ver (in this case, '3.7.2')
    # Case 3: if ver is None, return None
    return '.'.join(ver) if ver and '.' not in ver else ver

Then in my environment.devenv.yml:

version = {{ versionformat(31) }}

Implementation details

• We should export all symbols found in __all__.
• I guess we should be using importlib.import_module to load the file(s).

Questions

1. Do we want 1 single file (conda-devenv.py) or all files in the current directory (*.conda-devenv.py) ?
2. Should all files be loaded automatically (implicit) or should them be loaded explicitly through an exposed function (load_functions("custom-functions-conda-devenv.py")) ?
3. What about dependencies between these files? Do we care about that? Can one file import another?

Why???

This feature request is based on the discussion held in https://github.com/ESSS/conda-devenv/pull/83. If conda-devenv had this capability, then the user would never feel the need to open that PR, he would just write his custom functions.

[marcelotrevisani]

Uunnn... That looks interesting I think we could do it as a plugin system, you can add new functionalities as a plugin and use it in the conda-devenv

[nicoddemus]

Can we get more use cases? The one mentioned as an example is not even necessary anymore.

[tadeu]

I have mixed feeling about this. It's understandable the motivation to allow users to add their own functions, but, on the other hand, environment.devenv.yml files are supposed to be as descriptive as possible, i.e., just plain data, with jinja2 acting as a "facilitator" to select some things given environment variables, and to remove some duplication of data.

Let's continue the discussion, but keeping in mind that opening the door to custom user code that can execute anything and import anything would open the possibility of creating extra-complicated workflows.

[nicoddemus]

Agree with @tadeu, we should ponder first what are the use cases before adding features that we might not even actually use, or use very little. Also the fact that it is just a small utility is attractive, in my view.

[allanleal]

I don't have a strong use case anymore (I guess the one I had before was not strong either :smile: ) , but I like @edisongustavo 's initiative because this could be helpful eventually (if this was already permitted, I would not have wasted all your time yesterday with those discussions!). I understand these are not strong arguments though for this additional feature.

My main concern is that of @tadeu , in which we could open the doors to hell for users to write complicated environment.devenv.yml files that we see references to strange functions that we'll only be able to figure out if we go into the .conda-devenv.py file (where is this file going to be stored? I guess in the same directory of environment.devenv.yml, right?).

What if we at least supported inline lambda functions in comments with the following tentative syntax:

## env = lambda name: os.environ.get(name)
## versionformat = lambda ver: '.'.join(ver)

{% set conda_py = env('CONDA_PY') or '35' %}

name: web-ui-py{{ conda_py }}

dependencies:
  - python={{ versionformat(conda_py) }}
  - boost
  - cmake
  - gcc        # [linux]
  - ccache     # [not win]
  - clcache    # [win]

[marcelotrevisani]

@tadeu , That is why I suggested implementing that as plugin system, any plugin addition will be the programmer's risk.

[nicoddemus]

What if we at least supported inline lambda functions in comments

I'm afraid inventing our own mini-language, or parsing the file ourselves and generating Python code, is not a good direction. We now have to deal with scopes or supporting single-line functions, which limits its usability. We would be supporting "jinja + our own extension language". We would need to parse the file first, execute Python code to generate the lambda functions, and inject it later on the Jinja2 namespace. I'm afraid I'm a 👎 on this idea.

If we want to extend what's available in the Jinja2 namespace, I think using plain Python modules is the way to go, with an environment.devenv.py file in the same directory.

But before we go down that route, I would like to see good use cases first, because we have been using conda-devenv for a good 2 or 3 years now, and have not come across the need for complicated functions. The recent addition of the prepreocessor selectors was something we could have been using for sure, but it is a simple preprocessing line with very reduced scope.

[nicoddemus]

That is why I suggested implementing that as plugin system, any plugin addition will be the programmer's risk.

It is a good idea for a general framework, but again I would like to see more use cases before we even phantom introducing a plugin system.

[prusse-martin]

By "plugin system" you mean the entry points (similar to how pytest do)? I don't belive we should require that be are directly importable. Also, If we go with an external file(s) I think they should not be importable with import stataments (file name with dot) and in out code we do use the imp module like show here.

The inline option is interesting since all pieces are found in the same file. I was going to suggest to not limit it to just lambdas (or single line statements) but once I hand crafted a sample o how a .devenv.yml file will be I saw how ugly it can get.

[prusse-martin]

We could relly on jinja extensions (jinja docs)... but boy that looks complicated.

[tadeu]

Just a side note:

(if this was already permitted, I would not have wasted all your time yesterday with those discussions!).

I don't consider that as wasted time at all :) It was a very productive discussion, in fact, I think it's better in the long term to consider requests like these to be incorporated to conda-devenv itself, so that it can benefit all users.

[allanleal]

I don't consider that as wasted time at all :)

Thanks, @tadeu ! Good to know.

I'm afraid I'm a :-1: on this idea.

I was already expecting a denial from you, @nicoddemus ! :wink: I'm OK with what ever you guys decide.

[edisongustavo]

I agree with the points presented and I vote -1 on the idea as well.

If we ever show a very valid use case, then we can reopen this.

[majidaldo]

I've formed an opinion here. A templating language is good for when you can describe, in this case, an environment in what looks like more or less yaml. It's when you want to be more programmatic or just don't want to use yaml or jinja in yaml for what ever reason that this becomes a problem.

Here's a (practical) example where jinja/yaml fails at expression.

# ugly in jinja but simple in python!
{% set included_workdirs = [] %}   # cant use selectors  if i just do a list
{% if dev_req %}
{{ included_workdirs.append('dir') or "" }} # "" is just a hack to not output None)
{{ included_workdirs.append('dir2') or "" }} # [maybe a selector]
{% endif %}
{% if included_workdirs %}
includes:
{% for included in included_workdirs %}
  - '{{root}}/../{{included}}/environment.devenv.yml'
{% endfor %}
{% endif %}

...


environment:
  PATH:
    - '{{root}}'
   # i also want to use includes here!
    {% for workdir in included_workdirs %}
    - {{ os.path.abspath(os.path.join(root, '..', workdir, "bin")) }}
    {% endfor %}

A low-interfacing way here is to just have the user optionally provide a environment.devenv.py program instead that outputs the yaml that conda devenv expects (almost completely bypassing conda devenv). I could manually do this by somehow triggeringenvironment.devenv.pys before conda devenv but it would be nice to have this coordinated by conda devenv.

This keeps conda devenv 'simple'. While the user is responsible (completely) for 'complicated' setups. I feel like working within jinja to extend makes it more complicated. Just go all the way and use a general purpose programming language .