Unicode strings with encoding declaration are not supported.

[tonioo]

From @tonioo on December 4, 2013 15:48

Originally created by Antoine Nguyen on 2013-06-04T11:16:36Z

Some emails cannot be parsed by lxml due to the following error:

Unicode strings with encoding declaration are not supported.

Modoboa uses unicode strings but it appears it should stay with std strings...

Copied from original issue: tonioo/modoboa#410

[tonioo]

Posted by Simon Kern on 2013-07-30T23:31:07Z

Is this related to the following error? In general I found force_text (from django.utils.encoding) to be pretty helpful. If so I fixed it like that:

from django.utils.encoding import force_text
....
....
....
class EmailAddress(object):
    def __init__(self, address):
#        self.fulladdress = u2u_decode.u2u_decode(address).strip("\r\t\n")
#test fix
        self.fulladdress = force_text(address).strip("\r\t\n")
        (self.name, self.address) = parseaddr(self.fulladdress)
        if self.name == "":
            self.fulladdress = self.address
        
    def __str__(self):
        return self.fulladdress

Traceback (most recent call last):

  File "/foo/venv/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 115, in get_response
    response = callback(request, _callback_args, *_callback_kwargs)

  File "/foo/venv/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 25, in _wrapped_view
    return view_func(request, _args, *_kwargs)

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/decorators.py", line 19, in wrapped_f
    return f(request, _args, *_kwargs)

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/extensions/webmail/views.py", line 497, in index
    response = globals()[action](request)

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/extensions/webmail/views.py", line 337, in listmailbox
    return lst.render(request, request.session["pageid"])

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/email_listing.py", line 139, in render
    listing = self.fetch(request, page.id_start, page.id_stop)

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/email_listing.py", line 123, in fetch
    nbelems=self.elems_per_page))

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/tables.py", line 165, in **init**
    self.populate(rows)

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/tables.py", line 183, in populate
    c.transform(row, newcol, self)

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/tables.py", line 40, in transform
    col["value"] = table.parse(self.name, row[self.name])

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/extensions/webmail/lib.py", line 45, in parse
    value = getattr(IMAPheader, "parse_%s" % header)(value)

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/extensions/webmail/lib.py", line 90, in parse_from
    return IMAPheader.parse_address(value, **kwargs)

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/extensions/webmail/lib.py", line 72, in parse_address
    addr = EmailAddress(value)

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/emailutils.py", line 17, in **init**
    self.fulladdress = u2u_decode.u2u_decode(address).strip("\r\t\n")

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/u2u_decode.py", line 31, in u2u_decode
    u = mre.sub(decode_mime, s)

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 3: ordinal not in range(128)

[tonioo]

Posted by Antoine Nguyen on 2013-08-01T19:43:41Z

Unfortunately, your proposal would not solve this issue. The problem I encounter is described here:

http://lxml.de/parsing.html#python-unicode-strings

I need to find a proper way to avoid sending unicode strings to lxml...