Cant copy image file stored in template

[CaptnMorgan833]

Hello,

I tried the nice template feature with my own template ("conan new myModuie/1.0.0 -m <myTemplate>"). It works fine so far, except for the images I stored in the template. No matter which file type (png, jpg, bmp) the copied version cannot be displayed. Is this a Jinja issue or a conan issue?

Thanks!

[memsharded]

Hi @CaptnMorgan833

You mean that you have asset files in the template? That is interesting, why would you be including the same image files in the conan package recipe files, so often to make it a template?

I don't find anything specific why an image will not be copied, let me have a look and try to reproduce with a unittest.

[CaptnMorgan833]

Hey @memsharded

Yes, exactly. I have a logo image, which I want to place inside my /doc folder to have the logo in the doxygen generated documentation.

[memsharded]

I can't reproduce, it seems to be working fine, this is the test unit I wrote:

    def test_template_image_files(self):
        client = TestClient()
        template_dir = "templates/command/new/t_dir"
        save(os.path.join(client.cache_folder, template_dir + "/myimage.png"), "$&()")

        client.run("new hello/0.1 --template=t_dir")
        myimage = client.load("myimage.png")
        assert myimage == "$&()"

Could you please double check it? Any other hint to reproduce the issue?

[CaptnMorgan833]

Hmm that is strange...

This is the log I get:

C:\FAS\conan_test\gesdgdsgf>conan new myTestmodule/0.0.1 -m basis_library Traceback (most recent call last): File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\conans\client\command.py", line 2115, in run method(args[0][1:]) File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\conans\client\command.py", line 206, in new self._conan.new(args.name, header=args.header, pure_c=args.pure_c, test=args.test, File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\conans\client\conan_api.py", line 94, in wrapper return f(api, *args, **kwargs) File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\conans\client\conan_api.py", line 257, in new files = cmd_new(name, header=header, pure_c=pure_c, test=test, File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\conans\client\cmd\new.py", line 402, in cmd_new files = _get_files_from_template_dir(template_dir=template, File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\conans\client\cmd\new.py", line 314, in _get_files_from_template_dir rendered_file = _render_template(load(f_path), name=name, version=version, File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\conans\client\cmd\new.py", line 294, in _render_template t = Template(text, keep_trailing_newline=True) File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\jinja2\environment.py", line 1031, in new return env.from_string(source, template_class=cls) File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\jinja2\environment.py", line 941, in from_string return cls.from_code(self, self.compile(source), globals, None) File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\jinja2\environment.py", line 638, in compile self.handle_exception(source=source_hint) File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\jinja2\environment.py", line 832, in handle_exception reraise(*rewrite_traceback_stack(source=source)) File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\jinja2_compat.py", line 28, in reraise raise value.with_traceback(tb) File "", line 1436, in template jinja2.exceptions.TemplateSyntaxError: unexpected '}'

ERROR: unexpected '}'

After removing the .png file it works fine.

[CaptnMorgan833]

Im using conan 1.33.0 btw. May this be a problem?

[memsharded]

Oh, that could be it, indeed, the template feature improved recentely, the directory case was not handled before. I'd suggest trying to upgrade.

[CaptnMorgan833]

Ok I tried it with version 1.38.0 , but I ran into the same problem unfortunately :(

[CaptnMorgan833]

I will let one of my colleagues try to reproduce the problem. Let's see if they got the same issue.

[memsharded]

Oh, I have an idea. Maybe the image files are being parsed and tried to apply the template! I will update my test for this

[memsharded]

No, no success, I cannot make the test fail, even using characters like {}. Can you please share your jinja2 version? Doing a pip list or similar (I am testing with Jinja2 2.11.3)

[CaptnMorgan833]

Got the same version: Jinja2 2.11.3

[CaptnMorgan833]

That is a really weird issue. I will try it together with another colleague on Monday.

I will give feedback :)

Thanks for your quick response, have a nice weekend!

[CaptnMorgan833]

Hey James,

My colleague tried it and could reproduce the error with a different png Image.

It seems like it depends on the specific image whether the copying is successful or not.

Jinja2 parses all these images and it seems to result in one of these three outcomes (depending on the parsed image data):

1. Image can be copied
2. Image can be copied, but as a broken file. Can not open the picture with an image editor.
3. Syntax error while parsing the files, right after the "conan new foo/1.0.0 -m

Does it make sense to each try it with the opposing iamges every one of us used? :)

[memsharded]

Ok, managed to reproduce it, trying different binary patterns.

I am working to try a fix, but it happens that is not that evident, to differentiate which files should be tried to apply the template, and which files shouldn't. We could start using file extension and guessing, maybe exclude some known binary formats, but eventually there will be someone with a file without extension that they don't want to be templatized.

We could use a file inside the template to list the files to be templatized or the files to exclude.

In any case, I was wondering, because maybe adding that logo .png to all the package sources might not be the best approach, as you will be reproducing, storing and transferring the same file for every package, as part of the source of the package. It seems that you want to have that logo as part of the final binary package (the docs), so why not getting the logos (and any other files used to generate those company branded docs) as part of the build process? The idea is that the Conan recipes, the smallest in size, the better, as they are transferred many more times than the binaries.

[memsharded]

https://github.com/conan-io/conan/pull/9323 would be reproducing the issue, with a dirty poc of a fix based on try-except to avoid jinja2 errors

[CaptnMorgan833]

Hey thanks. I will try that commit on my machine.

Yeah I thought about something like a ".gitignore" kind of file to exclude filetypes I do not want to be parsed.

And yes I agree having the same binary file in the sources is not the best approach. But for now it seems like the simplest solution to make it work. But in the future it is prefered to make this a part of the build process, just as you said.

Anyways, thank you for your support so far!

[CaptnMorgan833]

Hey James, I tried your commit with my logo.png file, but I get the following error, when using the template:

[...]\new.py", line 310, in _get_files_from_template_dir content = content.decode("utf-8") UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

ERROR: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

Without the image it works fine as usual.

[memsharded]

Yep, definitely the "gitignore" kind solution would be the more robust, no magic or autodetection, just declare which files shouldn't be applied the template. That would be a fairly easy feature.

[memsharded]

Yes, I forgot about the potential Unicode error. I have added a capture for that exception too, please try again. Thanks!

[CaptnMorgan833]

I'll try.

[CaptnMorgan833]

Now it works! Nice one!

[memsharded]

Functionality of https://github.com/conan-io/conan/pull/9323 has been moved to 2.0 to https://github.com/conan-io/conan/pull/10255, because this never moved forward in 1.X, as it didn't feel right. In that other PR, the exclusion is explicit, with a file that defines which patterns to be excluded.

I still thing that the way to go is to have those binary assets somewhere in the build, but I hope this helps, even if it needs to wait until 2.0 (now we are in alpha).