Support unicode in journal

[clehner]

When I load pages I get a 500 Internal Server Error. I added some logging and see stack traces like the following:

Exception on /balance [GET]
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/srv/http/ledgible/ledgible.py", line 214, in balance
    return render_template('balance.html', rows=rows, encoding='utf-8')
  File "/usr/lib/python2.7/dist-packages/flask/templating.py", line 128, in render_template
    context, ctx.app)
  File "/usr/lib/python2.7/dist-packages/flask/templating.py", line 110, in _render
    rv = template.render(context)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 989, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "templates/balance.html", line 1, in top-level template code
    {% extends "base.html" %}
  File "templates/base.html", line 23, in top-level template code
    {% block content %}{% endblock %}
  File "templates/balance.html", line 18, in block "content"
    {{ (total.strip_annotations()|string).splitlines() | join('<br>'|safe) }}
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 263: ordinal not in range(128)

The error went away when I used a journal that didn't have unicode characters in it (e.g., currency symbols)

[lipidity]

Thank you for the report.

The error does appear to be related to the currency. The "string" filter in jinja2 is supposed to ensure unicode, but according to that error somehow there is an ascii encoding being used. I will test with unicode currency symbols.

[lipidity]

I have tested the use of unicode characters for the currency string; there were no issues.

Could you please provide a sample file to demonstrate the problem?

[clehner]

The following journal triggers the issue for me:

2016/05/02 Foo
    Assets:Cash                              €100.00
    Income

result:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)

[lipidity]

@clehner , it works for me. (I added that transaction verbatim, login as demo/demo here to see)

What versions of python and Flask are you using?

[clehner]

On Mon, 02 May 2016 17:03:45 -0700 Ankur Kothari [email protected] wrote:

What versions of python and Flask are you using?

python 2.7.11 flask 0.10.1

[lipidity]

It's apparently not a ledgible issue, in any case...

I'm not sure why your python2 is trying to use ascii. Are you using a unicode locale?

[lipidity]

Unicode in definitely supported in the journal, so I'll close this issue.

I'll try various configurations to reproduce your error, and feel free to contact me for help in debugging.

[lipidity]

I was able to reproduce this issue on one of my servers. Will report back soon.