mako
mako copied to clipboard
sqlalchemy
Lingua 4.x breaks the Lingua extractor
Migrated issue, originally created by Laurent DAVERIO (@ldaverio)
When upgrading from Lingua 3.12 to Lingua 4.x, an attempt to extract strings from a Mako template fails with the following message:
File "/Users/xxxxxxx/pydev/lib/python3.5/tokenize.py", line 560, in _tokenize if line[pos] in '#\r\n': # skip comments or blank lines
This message also appeared in issue #72 of the wichert/lingua repository on GitHub:
https://github.com/wichert/lingua/issues/72
The problem seems to be that "tokenize requires unicode input". And in effect, it seems that I can get rid of the problem by changing line 35 of ext/linguaplugin.py from :
code = io.ByteIO(source)
to:
code = io.StringIO(str(source, encoding='utf-8'))
I'm not sure of all the implications yet, so I don't want to issue a pull request, but this fix seems to work for me.
Wichert Akkerman (@wichert) wrote:
This is my fault: I had not realised that a recent refactoring impacted the API used by plugins. Since this is likely to impact more plugins I think this should be addressed in lingua.
Laurent DAVERIO (@ldaverio) wrote:
FWIW, I've been using the patched version of Mako (with the change in ext/linguaplugin.py) since I reported the problem, everything works fine for me :)
Michael Bayer (@zzzeek) wrote:
that's not fixing the failure I'm getting, which is:
#!
========================================================================= FAILURES =========================================================================
_____________________________________________________________ ExtractMakoTestCase.test_extract _____________________________________________________________
Traceback (most recent call last):
File "/home/classic/dev/mako/test/ext/test_linguaplugin.py", line 30, in test_extract
plugin(os.path.join(template_base, 'gettext.mako'), MockOptions()))
File "/home/classic/dev/mako/mako/ext/extract.py", line 13, in process_file
for extracted in self.extract_nodes(template_node.get_children()):
File "/home/classic/dev/mako/mako/ext/extract.py", line 91, in extract_nodes
code, node.lineno - 1, translator_strings):
File "/home/classic/dev/mako/mako/ext/linguaplugin.py", line 36, in process_python
self.filename, self.options, code, code_lineno -1):
File "/home/classic/dev/mako/.tox/py27/lib/python2.7/site-packages/lingua/extractors/python.py", line 356, in __call__
return parser(token_stream, options, filename, lineno)
File "/home/classic/dev/mako/.tox/py27/lib/python2.7/site-packages/lingua/extractors/python.py", line 153, in __call__
if options.comment_tag is True:
AttributeError: MockOptions instance has no attribute 'comment_tag'
===================================================== 1 failed, 460 passed, 2 skipped in 11.15 seconds =====================================================
ERROR: InvocationError: '/home/classic/dev/mako/.tox/py27/bin/py.test'
_________________________________________________________________________ summary __________________________________________________________________________
ERROR: py27: commands failed
this might be just local to the test but I don't have resources to debug this plugin myself.
Wichert Akkerman (@wichert) wrote:
I've pushed a change to lingua which allows passing bytes instances to the Python extractor again. @ldaverio can you please check if that fixes your problem?
Laurent DAVERIO (@ldaverio) wrote:
Hello Wichert, sorry for the late reply.
I've just tested the 4 following combinations :
- 2 Mako versions (1.0.4, and my git)
- 2 Lingua versions (4.8, and 4.9)
With Lingua 4.8, my patched Mako works for me, the other fails with:
Traceback (most recent call last):
File "/usr/home/daverio/pydev/bin/pot-create", line 11, in <module>
sys.exit(main())
File "/usr/home/daverio/pydev/lib/python3.5/site-packages/lingua/extract.py", line 280, in main
for message in extractor(real_filename, options):
File "/usr/home/daverio/pydev/mako/mako/ext/extract.py", line 13, in process_file
for extracted in self.extract_nodes(template_node.get_children()):
File "/usr/home/daverio/pydev/mako/mako/ext/extract.py", line 91, in extract_nodes
code, node.lineno - 1, translator_strings):
File "/usr/home/daverio/pydev/mako/mako/ext/linguaplugin.py", line 36, in process_python
self.filename, self.options, code, code_lineno -1):
File "/usr/home/daverio/pydev/lib/python3.5/site-packages/lingua/extractors/python.py", line 356, in __call__
return parser(token_stream, options, filename, lineno)
File "/usr/home/daverio/pydev/lib/python3.5/site-packages/lingua/extractors/python.py", line 166, in __call__
for (token_type, token, location, _) in token_stream:
File "/usr/home/daverio/pydev/lib/python3.5/site-packages/lingua/extractors/python.py", line 140, in next
token = self._transform(next(self.queue))
File "/usr/home/daverio/pydev/lib/python3.5/tokenize.py", line 560, in _tokenize
if line[pos] in '#\r\n': # skip comments or blank lines
TypeError: 'in <string>' requires string as left operand, not int
With Lingua 4.9, both fail with a second error message (but the UnicodeWarning message happens only with regular Mako 1.0.4):
/usr/home/daverio/pydev/lib/python3.5/site-packages/mako/ext/linguaplugin.py:36: UnicodeWarning: Python extractor called with bytes input. Please update your plugin to submit unicode instead.
self.filename, self.options, code, code_lineno -1):
Traceback (most recent call last):
File "/usr/home/daverio/pydev/bin/pot-create", line 11, in <module>
sys.exit(main())
File "/usr/home/daverio/pydev/lib/python3.5/site-packages/lingua/extract.py", line 356, in main
save_catalog(catalog, options.output)
File "/usr/home/daverio/pydev/lib/python3.5/site-packages/lingua/extract.py", line 230, in save_catalog
if old_catalog is not None and identical(catalog, old_catalog):
File "/usr/home/daverio/pydev/lib/python3.5/site-packages/lingua/extract.py", line 220, in identical
return a == b
File "/usr/home/daverio/pydev/lib/python3.5/site-packages/lingua/extract.py", line 64, in __eq__
r = super(POEntry, self).__eq__(other)
File "/usr/home/daverio/pydev/lib/python3.5/site-packages/polib-1.0.7-py3.5.egg/polib.py", line 1083, in __eq__
File "/usr/home/daverio/pydev/lib/python3.5/site-packages/polib-1.0.7-py3.5.egg/polib.py", line 1059, in __cmp__
TypeError: string indices must be integers
I can't investigate the problem right now, but I'll try to do it asap, possibly this weekend.
Laurent DAVERIO (@ldaverio) wrote:
Hello again, in addition to the Lingua 4.9 problem, which forced me to stay at 4.8.1, it now appears that setuptools >= 20.6.6 breaks "po-create" :( :( :(
So, I have to pin setuptools==20.4...
Traceback (most recent call last):
File "/Users/daverio/pydev/bin/pot-create", line 11, in <module>
sys.exit(main())
File "/Users/daverio/pydev/lib/python3.5/site-packages/lingua/extract.py", line 252, in main
register_babel_plugins()
File "/Users/daverio/pydev/lib/python3.5/site-packages/lingua/extractors/babel.py", line 51, in register_babel_plugins
extractor = entry_point.load(require=True)
File "/Users/daverio/pydev/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2206, in load
self.require(*args, **kwargs)
File "/Users/daverio/pydev/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2223, in require
items = working_set.resolve(reqs, env, installer)
File "/Users/daverio/pydev/lib/python3.5/site-packages/pkg_resources/__init__.py", line 814, in resolve
if req.marker and not req.marker.evaluate():
File "/Users/daverio/pydev/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/markers.py", line 278, in evaluate
return _evaluate_markers(self._markers, current_environment)
File "/Users/daverio/pydev/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/markers.py", line 203, in _evaluate_markers
lhs_value = _get_env(environment, lhs.value)
File "/Users/daverio/pydev/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/markers.py", line 185, in _get_env
"{0!r} does not exist in evaluation environment.".format(name)
pkg_resources.extern.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.