Stream operation for pystache

[gonvaled]

This is what I usually do to render a file with pystache:

txt = open(filename, 'r').read() print pystache.render(txt, context).encode('utf-8') ("context" is my mapping)

But now I have to render very big templates, which causes memory problems. It would be great if pystache could operate on streams.

[cjerdonek]

Thanks for the report. For future reference, using memory mapped files might be another possible approach to dealing with large files, but I'm not sure.

[gonvaled]

That is a good idea. For the input data this would probably work, but (and I am talking here without knowing the details of the pystache implementation) I guess this would not completely solve the issue, since the output data would still be in memory

[gonvaled]

And it seems not even input mmap is supported:

import mmap
import pystache
tmp_file = '/tmp/tmp_file'
template_text = 'Hello {{name}}'
d = {'name' : 'Steve'}
f = open(tmp_file, 'w')
f.write(template_text)
f.flush()
f = open(tmp_file, 'r+b')
mymmap = mmap.mmap(f.fileno(), 0)
print pystache.render(mymmap, d)
mymmap.close()

» python test_pystache_mmap.py 
Traceback (most recent call last):
  File "test_pystache_mmap.py", line 11, in 
    print pystache.render(mymmap, d)
  File "/home/gonvaled/.virtualenvs/python2.7.2-wavilon1/lib/python2.7/site-packages/pystache/__init__.py", line 8, in render
    return Template(template, context).render()
  File "/home/gonvaled/.virtualenvs/python2.7.2-wavilon1/lib/python2.7/site-packages/pystache/template.py", line 176, in render
    result = self._render_tags(template)
  File "/home/gonvaled/.virtualenvs/python2.7.2-wavilon1/lib/python2.7/site-packages/pystache/template.py", line 119, in _render_tags
    template = template.replace(tag, replacement)
AttributeError: 'mmap.mmap' object has no attribute 'replace'

[cjerdonek]

Just to clarify, I didn't mean to suggest that mmap would work without making additional changes. :) Work would need to be done there, too (but probably not as much). And yes, you are right. If support for mmap were added to the API, we would not want those methods to return the output as a string, etc.

[cjerdonek]

Could you describe what your large files "look" like? For example, are they deeply nested with sections, is it long strings of plain text in between a few scattered tags, or is it equally dense with both plain text and tags? Also, are there section tags that contain huge strings, etc?

[markfink]

1++ I would love to see the streaming, too. I create large output files similar to sql format with multiple entities. Basically the rendering processes one list with 500k entries. Currently it fits all in memory but the rendering takes ~5 Minutes. I could split the file up to multiple files but this makes the handling much more complex.