tera icon indicating copy to clipboard operation
tera copied to clipboard

Include content without parsing it

Open ranfdev opened this issue 6 years ago • 24 comments

Something like this https://tera.netlify.com/docs/templates/#include but instead of parsing the content of the file as if it was a template, just include the raw content of the file. Maybe a function like: include_raw. Why? It's needed to be able to import html files from external tools such as ansi2html or https://git.sr.ht/~sircmpwn/openring.

ranfdev avatar Jul 03 '19 20:07 ranfdev

Is there some precedent in Jinja2/Django? As long as there isn't {{, {% etc in the template it should already work fine

Keats avatar Jul 04 '19 19:07 Keats

The problem is, i may work with user generated html files. A user may write {{ in the file and read other files on my system. That's a security issue. Manually escaping user input is really hard/impossible, so i hope this can be solved at a deeper level, inside tera

ranfdev avatar Jul 04 '19 20:07 ranfdev

Where would the files be located? In the templates directory or would the tag/function have to load them from the filesystem first?

Keats avatar Jul 05 '19 13:07 Keats

Just to be clear about what i want to build:

  • I have a website (made with zola, which is awesome btw).
  • I want to create a section "read next" in the footer of my website, like this: image using https://git.sr.ht/~sircmpwn/openring.
  • the program openring works by fetching a list of rss feeds and putting a preview of the latest articles inside an output html file (let's say openring.html, inside the templates folder. If you want a better explanation of how openring works, checkout the website).
  • I now need to include the openring.html file inside my footer.

Problem: someone may put {{ on his blog, so in the file openring.html i have {{. I need to include openring.html without parsing it.

ranfdev avatar Jul 05 '19 21:07 ranfdev

I agree that this is a useful feature, and also think I should be able to tackle the implementation if a PR is welcome. I would expect include_raw to use the same search path as include (which isn't clear from documentation; are paths in include just always relative to the directory the including file is in?)

jplatte avatar Nov 06 '19 23:11 jplatte

include paths are based on their own paths: if you are using globs like templates/**/*.html and have templates/something/else.html and templates/something/different.html, you would include "something/else.html in different.html.

Keats avatar Nov 07 '19 00:11 Keats

@Keats So do you think this is something that's worthwhile to support in tera?

jplatte avatar Nov 11 '19 22:11 jplatte

I think so, is there a precedent in other template engine we can use for the naming?

Keats avatar Nov 12 '19 19:11 Keats

I can't find any. pug has this feature, but decides based on the file extension whether to treat a file as a template or a static file. I could imagine going with include_raw.

jplatte avatar Nov 12 '19 19:11 jplatte

include_raw sounds good to me

Keats avatar Nov 13 '19 17:11 Keats

Okay! I'm working on this now :)

jplatte avatar Nov 15 '19 11:11 jplatte

I see now that the general approach is for a user to specify a template directory, from which all templates are loaded first, and includes are then later resolved to the already-parsed templates. I'm not sure how to proceed. We could also pre-register non-template / raw include files like template files (whether or not they're eagerly loaded), but I'm not sure if that's the most reasonable way forward. Also, users might want to save these files alongside the templates :thinking:

What about this: Allow users to specify template and static "include paths" (or single files) like it's possible for templates with load_from_glob now, and additionally have a convenience method to add a directory to both. Static files will be loaded lazily by default, and we could implement this for templates later as well. (only parsing them on first use, rather than eagerly when registered)

jplatte avatar Nov 15 '19 11:11 jplatte

Hmm it does sounds like it would be easier to just ignore them in the glob or name them something that won't get parsed like blogroll.html.ignored or whatever. I'm a bit wary about adding the include directory concept. Zola already has a Tera function to load files and I don't think it's a super common usecase in Tera itself so it might be better to focus on loading files from the template directory only, and ensure something like {% include_raw "~/.ssh/id_rsa" %} errors.

Honestly I got even more confused after writing that x)

Keats avatar Nov 15 '19 19:11 Keats

The "template directory" from which templates are currently eagerly loaded is exactly what I mean by "template include path". Just that for raw includes I think it makes even less sense to load them eagerly from the user-specified directory, rather than just remembering the path and only allowing access to files within that path.

jplatte avatar Nov 15 '19 21:11 jplatte

Could you write some pseudocode on how it would look like from a user pov?

Keats avatar Nov 17 '19 18:11 Keats

I was thinking of {% include_raw "file.html" %}

file.html is located at include_raw/file.html

ranfdev avatar Dec 01 '19 20:12 ranfdev

so the files can only be loaded from a hardcoded directory name? That doesn't sound great :/

Keats avatar Dec 07 '19 16:12 Keats

@ranfdev I think for openring Zola's load_data filter could be used actually.

jplatte avatar Dec 30 '19 19:12 jplatte

@Keats I'm not sure whether you're talking about Rust code or Tera code. I'll try to explain again:

Like tera currently has add_template_file and add_template_files, it could have something like add_static_file / add_static_files. The files added like that could then be included via include_raw in tera code.

Personally I think it makes sense to only actually read one of those files from disk when a template using it is instantiated, rather than on the call to add_static_file / add_static_files, especially because static file directories might contain lots of files not included from templates via include_raw, but that's more or less an implementation detail.

jplatte avatar Dec 30 '19 19:12 jplatte

The main issue with that is that it cannot load static files dynamically, you have to call Tera::add_static_files which isn't great for many usecases.

I'm tempted to have a Tera function (or keyword like include_raw) like include_file that loads a file from the root of the template directory of the glob.

Keats avatar Dec 30 '19 19:12 Keats

Oh, I didn't look at the API well enough. I was actually thinking about something where you'd specify a directory. I thought add_template_files would add everything from a directory, but it's just a list of files.

Either way, I think instead of loading from the root of the template directory it should load from a directory the user specifies with another method of struct Tera (or maybe, for more flexibility, one of possibly multiple directories). So there'd be

impl Tera {
    pub fn add_asset_directory(&mut self, path: impl AsRef<Path>) -> Result<()> { ... }
}

that adds a directory to the include_raw "search path". When {% include_raw x/y.txt %} is then used, it looks for x/y.txt in all directories added that way in the order they were specified until it finds that file, then it includes it.

jplatte avatar Dec 30 '19 20:12 jplatte

@ranfdev I think for openring Zola's load_data filter could be used actually.

That's it! It does what i need, thank you!

ranfdev avatar Dec 31 '19 20:12 ranfdev

This is also something that I'd like. In Jekyll, there are a few includes for that (see Jekyll - Docs - Includes) that are very handy.

heitorPB avatar Apr 18 '23 19:04 heitorPB