Fails to render requirements when using Jinja "extends"

[ccharlesgb]

Actual Behavior

I am experimenting with using a base template for my conda packages and believe I have found an issue with how conda renders the Jinja2 templates when using the {% extends "" %} syntax. I have 2 basic test packages which I have been using to isolate the issue. The first has the meta.yaml file:

meta.yaml

package:
  name: cc_pack
  version: "0.1"

source:
  path: ../

requirements:
    build:
        - python
    run:
        - python

build:
  script: python setup.py install
  entry_points:
    - cc_test = cc_pack.do_stuff:main

test:
  imports:
    - cc_pack

Which renders using conda render to:

M:\Python>conda render cc-pack
package:
    name: cc_pack
    version: '0.1'
source:
    path: M:\Python\cc-pack
build:
    entry_points:
        - cc_test = cc_pack.do_stuff:main
    script: python setup.py install
requirements:
    build:
        - wheel 0.30.0 py27ha643586_1
        - setuptools 38.4.0 py27_0
        - pip 9.0.1 py27hdaa76b4_4
        - certifi 2018.1.18 py27_0
        - vs2008_runtime 9.00.30729.1 hfaea7d5_1
        - wincertstore 0.2 py27hf04cefb_0
        - vc 9 h7299396_1
        - python 2.7.14 h8c3f1cb_23
    run:
        - python >=2.7,<2.8.0a0
test:
    imports:
        - cc_pack
extra:
    copy_test_source_files: true
    final: true

This render builds as expected and works fine. However if I use the following Jinja base template:

base_meta.yaml

package:
  name: {% block package_name %}{% endblock %}
  version: 0.1

source:
  path: ../

{% block requirements %}

{% endblock %}

build:
  script: python setup.py install
  entry_points:
    - cc_test = cc_pack.do_stuff:main

test:
  imports:
    - cc_pack

With the corresponding meta.yaml:

{% extends "base_meta.yml" %}

{% block package_name %}cc-pack{% endblock %}

{% block requirements %}

requirements:
  build:
    - python
  run:
    - python
    
{% endblock %}

The render output is:

M:\Python>conda render cc-pack
package:
    name: cc-pack
    version: '0.1'
source:
    path: M:\Python\cc-pack
build:
    entry_points:
        - cc_test = cc_pack.do_stuff:main
    script: python setup.py install
test:
    imports:
        - cc_pack
extra:
    copy_test_source_files: true
    final: true

The requirements section is missing! This then causes the build to fail and the import test failed to import the module. I can't find anything in the documentation that says this behavior is not supported.

I would expect the requirement section to render fine however it appears that this is ignored by the render stage.

Output of conda info

M:\Python>conda info

     active environment : None
       user config file : C:\Users\ccharles\.condarc
 populated config files : C:\Users\ccharles\.condarc
          conda version : 4.4.10
    conda-build version : 3.4.2
         python version : 2.7.13.final.0
       base environment : C:\dev\bin\Anaconda  (writable)
           channel URLs : https://conda.anaconda.org/r/win-64
                          https://conda.anaconda.org/r/noarch
                          https://repo.continuum.io/pkgs/main/win-64
                          https://repo.continuum.io/pkgs/main/noarch
                          https://repo.continuum.io/pkgs/free/win-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/win-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/win-64
                          https://repo.continuum.io/pkgs/pro/noarch
                          https://repo.continuum.io/pkgs/msys2/win-64
                          https://repo.continuum.io/pkgs/msys2/noarch
                          http://conda/external/win-64
                          http://conda/external/noarch
                          http://conda/ci/win-64
                          http://conda/ci/noarch
                          file://fsrsvr06pr/departments/Pricing%20%26%20Forecast
ing/Department/Python/conda/ge/win-64
                          file://fsrsvr06pr/departments/Pricing%20%26%20Forecast
ing/Department/Python/conda/ge/noarch
          package cache : C:\dev\bin\Anaconda\pkgs
                          C:\Users\ccharles\AppData\Local\conda\conda\pkgs
       envs directories : C:\dev\bin\Anaconda\envs
                          C:\Users\ccharles\AppData\Local\conda\conda\envs
                          C:\Users\ccharles\.conda\envs
               platform : win-64
             user-agent : conda/4.4.10 requests/2.14.2 CPython/2.7.13 Windows/7
Windows/6.1.7601
          administrator : False
             netrc file : None
           offline mode : False

The test repo for reproducing this can be found: https://github.com/ccharlesgb/cc-pack

[msarahan]

This is not a currently supported feature, but it would be nice to get it to work. I'll add it to our backlog. If you'd like to see it sooner, please submit a PR.

[mbargull]

Then this is a regression. This always worked with conda-build 2.1.x.

[mbargull]

$ conda-build --version
conda-build 2.1.18
$  conda render .
package:
    name: cc-pack
    version: '0.1'
source:
    path: ../
build:
    entry_points:
        - cc_test = cc_pack.do_stuff:main
    script: python setup.py install
requirements:
    build:
        - python
    run:
        - python
test:
    imports:
        - cc_pack
extra:
    final: true

[msarahan]

A regression of something that wasn't officially supported isn't really a regression. I would like to see this fixed, but it's still in the backlog. If it matters to you, please submit a PR with a test, and/or a fix. The test is the key part here - if it's not tested, it's not supported.

[ccharlesgb]

Thanks for the response. I did a bit of debugging in to why this is happening and the only thing I was able to find out is that the code seems to execute in the same way until line 358 in conda-build/render.py :

        # extract the topmost section where variables are defined, and put it on top of the
        #     requirements for a particular output
        # Re-parse the output from the original recipe, so that we re-consider any jinja2 stuff
        output = m.get_rendered_output(m.name())
        rendered_metadata = m.get_output_metadata(output)

After this line the "output" is different for the un-extended meta it returns:

{'requirements': {u'run': [u'python'], u'build': [u'python']}, 'name': u'cc_pack', 'build': {u'entry_points': [u'cc_test = cc_pack.do_stuff:main'], u'script': u'python setup.py install'}}

And for the extended it just returns:

{'name': u'cc-pack'}

I'm not really sure why this is happening, I guess there is a function call to MetaData.get_recipe_text() that does not apply the full Jinja templating? Would be nice to see this working though.

[msarahan]

Thanks for debugging. That saves me time, and I appreciate it.

a function call to MetaData.get_recipe_text() that does not apply the full Jinja templating?

That's a reasonable guess. I don't know how it would be happening - the whole point of get_recipe_text is to apply the full jinja templating - but it could certainly still be happening somehow. The purpose of all of these "get_rendered_*" methods are so that we bridge the gap between the raw templates and rendered recipe content. That gap is what allows us to detect which variables get used in templating, which in turn determines how variables get treated for looping at the top-level vs the output level. I'm pretty sure conda-build assumes that there's only ever one template. That's fine in effect, but we need jinja2 to assemble any "extends" stuff before conda-build thinks it has its template.

[msarahan]

Unfortunately, this is a lot more tricky than I had hoped. It is as I described - we are loading raw text here: https://github.com/conda/conda-build/blob/master/conda_build/metadata.py#L713

that's called from https://github.com/conda/conda-build/blob/master/conda_build/metadata.py#L1497

In order to really support this use case while also maintaining the "used variable" detection stuff, we'd need to have some kind of "partial" jinja2 rendering. It would need to evaluate ONLY the extends blocks, but leave the template tags everywhere else. I've searched around some, and didn't see any good way to do that. If anyone wants to get (deep) into jinja2 and find a way to do this, I'll be grateful. The hacky, horrible way I thought of was to start in meta.yaml, and then regex for any use of "extends" tags, and follow that trail. That would be nasty, but it might work. Alternatively, if anyone has better ideas of how to detect "used" variables, I'm wide open to them, as well.

Conda-build 2 didn't support build matrices, and didn't need to understand "used" variables. It is unfortunate that these features are in competition, but the ability to have build matrices and know which variables are used far outweigh this template composition.

[ccharlesgb]

In order to do partial jinja rendering I think you need to start using Jinja extensions. Where you can hook into the rendering process and decide to parse/render selectively. From a quick glance the Jinja extensions api method filter_stream might be needed here. But I am not really familiar enough with the internals of Jinja go much further.

[github-actions[bot]]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include: - What OS and version you reproduced the issue on - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!