Writer converts everything to str

[dbrgn]

The Writer class converts all strings to str. This causes Problems with unicode strings.

64     def __call__(self, s, newline=True, stream=STDOUT):
65 
66         if newline:
67             s = tsplit(s, NEWLINES)
68             s = map(str, s)
69             indent = ''.join(self.shared['indent_strings'])
70 
71             s = (str('\n' + indent)).join(s)
72 
73         _str = ''.join((
74             ''.join(self.shared['indent_strings']),
75             str(s),
76             '\n' if newline else ''
77         ))
78         stream(_str)

While I can print a unicode string

>>> print u'äöü'
äöü

...I cannot do that when using puts.

>>> puts(u'äöü')
Traceback (most recent call last):
  File "test.py", line 7, in <module>
    puts(u'äöü')
  File "/home/danilo/Projects/clint/clint/textui/core.py", line 83, in puts
    Writer()(s, newline, stream=stream)
  File "/home/danilo/Projects/clint/clint/textui/core.py", line 68, in __call__
    s = map(str, s)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)

This breaks puts-ing any non-ascii characters, which is a big problem for me.

This is probably the reason for #47.

[myusuf3]

:+1:

[dbrgn]

Hey @kennethreitz, this is what I talked to you about this evening :) Would this still work if using unicode strings internally?

[hdemers]

Any updates on this issue? I'm also having problems, specifically in formatters.py line 85 where the same map with str is done:

_row = map(str, _row)

[jpiper]

This is a more widespread issue in clint with the handling of strings - particularly with regard to unicode strings. Some methods work on unicode strings and others fail . We need to go through with a fine toothcomb and try and work out a suitable way to fix it.

[gazpachoking]

This is a more widespread issue in clint with the handling of strings - particularly with regard to unicode strings. Some methods work on unicode strings and others fail . We need to go through with a fine toothcomb and try and work out a suitable way to fix it.

Yeah, I was just thinking about the best solution for this. Had several issues with implicit decoding using ascii, as well as other problems with explicit decoding using utf8 (which was the wrong encoding for the bytes I was giving.) I think the nicest solution would be to store all strings as strings internally, and only encode to bytes when outputting to the system. It would also be nice to be able to select a replace error handler, so that unicode strings can be outputted in consoles that do not support an encoding which can represent all the needed characters (windows, I'm looking at you.) I can take a crack at it, if we think that storing everything as strings internally is the right answer.

[flexd]

Any progress on this? :-)

[philadams-zz]

Running the unicode through colored works just fine (if that's an okay work around for anyone right now): puts(colored.red(u'äöü'))

[lsaffre]

Yes, gazpachoking I agree that all strings should internally stored as unicode strings, and decoding should occur only when they are being output to a stream. It is a very annoying thing. Why is there no progress on this issue? Is there some workaround other than coloring the text?

[Miserlou]

+1, this is a pretty shitty bug, ran into this while porting an app from Py2 to Py3. Blahh..

puts() should have safe unicode handling, and I suggest that there should also be a colored.default() that uses the default terminal font.

[pevik]

@philadams puts(colored.red(u'äöü')) is still not working on python2 windows with non-unicode encoding :-(.

[Miserlou]

@pevik - it seems like clint is abandoned, I switched to using the 'click' project instead, which has this issue fixed.