UnicodeDecodeError: RateScaleField with non-ascii values and without Thank-you-page

[jensens]

Wenn we have a RateScaleField with german umlauts (äöü) in the choices and do not provide a thank you page the following traceback occurs:

Traceback (innermost last): Module ZPublisher.Publish, line 138, in publish Module ZPublisher.mapply, line 77, in mapply Module ZPublisher.Publish, line 48, in call_object Module Products.CMFFormController.FSControllerPageTemplate, line 91, in call Module Products.CMFFormController.BaseControllerPageTemplate, line 28, in _call Module Products.CMFFormController.ControllerBase, line 231, in getNext Module Products.CMFFormController.Actions.TraverseTo, line 38, in call Module ZPublisher.mapply, line 77, in mapply Module ZPublisher.Publish, line 48, in call_object Module Shared.DC.Scripts.Bindings, line 322, in call Module Shared.DC.Scripts.Bindings, line 359, in _bindAndExec Module Products.CMFCore.FSPageTemplate, line 237, in _exec Module Products.CMFCore.FSPageTemplate, line 177, in pt_render Module Products.PageTemplates.PageTemplate, line 79, in pt_render Module zope.pagetemplate.pagetemplate, line 132, in pt_render Module zope.pagetemplate.pagetemplate, line 240, in call Module zope.tal.talinterpreter, line 271, in call Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 888, in do_useMacro Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 533, in do_optTag_tal Module zope.tal.talinterpreter, line 518, in do_optTag Module zope.tal.talinterpreter, line 513, in no_tag Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 954, in do_defineSlot Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 533, in do_optTag_tal Module zope.tal.talinterpreter, line 518, in do_optTag Module zope.tal.talinterpreter, line 513, in no_tag Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 858, in do_defineMacro Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 954, in do_defineSlot Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 533, in do_optTag_tal Module zope.tal.talinterpreter, line 518, in do_optTag Module zope.tal.talinterpreter, line 513, in no_tag Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 946, in do_defineSlot Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 533, in do_optTag_tal Module zope.tal.talinterpreter, line 518, in do_optTag Module zope.tal.talinterpreter, line 513, in no_tag Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 533, in do_optTag_tal Module zope.tal.talinterpreter, line 518, in do_optTag Module zope.tal.talinterpreter, line 513, in no_tag Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 821, in do_loop_tal Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 533, in do_optTag_tal Module zope.tal.talinterpreter, line 518, in do_optTag Module zope.tal.talinterpreter, line 513, in no_tag Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 533, in do_optTag_tal Module zope.tal.talinterpreter, line 518, in do_optTag Module zope.tal.talinterpreter, line 513, in no_tag Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 620, in do_insertText_tal Module Products.PageTemplates.Expressions, line 226, in evaluateText Module Products.PageTemplates.Expressions, line 264, in _handleText UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 20: ordinal not in range(128)

[mauritsvanrees]

Confirmed. I fixed something similar today (pull request #184 ) and I wondered if that fixed this error too, but unfortunately not. I am not going to fix it now, but let me at least leave some notes.

When tried in Plone 5, you see it goes wrong in a zope.i18n.translate call, which should not even take place according to the fg_result_embedded_view.pt template.

The request contains {'answer 1': '\xc3\xa4\xc3\xb6\xc3\xbc'}. Note that this looks like a dictionary, but it is an 'instance', which I think means an old-style class.

(Pdb) self.request.get(fname)
{'1': '\xc3\xa4\xc3\xb6\xc3\xbc'}
(Pdb) unicode(self.request.get(fname))
*** UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)
(Pdb) unicode(self.request.get(fname), encoding='utf-8')
*** TypeError: coercing to Unicode: need string or buffer, instance found
(Pdb) unicode(str(self.request.get(fname)), encoding='utf-8')
u'1: \xe4\xf6\xfc'
(Pdb) str(self.request.get(fname))
'1: \xc3\xa4\xc3\xb6\xc3\xbc'

Should be fixable, but it may need a helper method or browser view to get a value that does not break. It might need a fix in the LikertFields. I am not very familiar with those.

[mauritsvanrees]

This fix is in release 1.8.4 (today).