Unicode output from commands with unexpected

[bhyde]

I think inapproprate things happen when a command exits with both an unexpected return code and unicode it it's stdout.

bash-3.2$ python -V
Python 2.7.12
bash-3.2$ pip freeze | grep plumbum
plumbum==1.6.6
bash-3.2$ python
Python 2.7.12 (default, Jul 10 2017, 12:16:09) 
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from plumbum import local
>>> print local['bash']('-c', u"/bin/echo \u25cf ; exit 1", retcode=1)
●

>>> print local['bash']('-c', u"/bin/echo \u25cf ; exit 1")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/bhyde/.pyenv/versions/bh-venv-2.7.12/lib/python2.7/site-packages/plumbum/commands/base.py", line 95, in __call__
    return self.run(args, **kwargs)[1]
  File "/Users/bhyde/.pyenv/versions/bh-venv-2.7.12/lib/python2.7/site-packages/plumbum/commands/base.py", line 225, in run
    return p.run()
  File "/Users/bhyde/.pyenv/versions/bh-venv-2.7.12/lib/python2.7/site-packages/plumbum/commands/base.py", line 187, in runner
    return run_proc(p, retcode, timeout)
  File "/Users/bhyde/.pyenv/versions/bh-venv-2.7.12/lib/python2.7/site-packages/plumbum/commands/processes.py", line 206, in run_proc
    return _check_process(proc, retcode, timeout, stdout, stderr)
  File "/Users/bhyde/.pyenv/versions/bh-venv-2.7.12/lib/python2.7/site-packages/plumbum/commands/processes.py", line 23, in _check_process
    proc.verify(retcode, timeout, stdout, stderr)
  File "/Users/bhyde/.pyenv/versions/bh-venv-2.7.12/lib/python2.7/site-packages/plumbum/machines/base.py", line 22, in verify
    stdout, stderr)
plumbum.commands.processes.ProcessExecutionError: <exception str() failed>
>>> 

In ipython a more verbose backtrace:

In [2]: print local['bash']('-c', u"/bin/echo \u25cf ; exit 1")
---------------------------------------------------------------------------
ProcessExecutionError                     Traceback (most recent call last)
<ipython-input-2-0423c5fdb0d8> in <module>()
----> 1 print local['bash']('-c', u"/bin/echo \u25cf ; exit 1")

/Users/bhyde/.pyenv/versions/2.7.12/envs/bh-venv-2.7.12/lib/python2.7/site-packages/plumbum/commands/base.pyc in __call__(self, *args, **kwargs)
     93     def __call__(self, *args, **kwargs):
     94         """A shortcut for `run(args)`, returning only the process' stdout"""
---> 95         return self.run(args, **kwargs)[1]
     96 
     97     def _get_encoding(self):

/Users/bhyde/.pyenv/versions/2.7.12/envs/bh-venv-2.7.12/lib/python2.7/site-packages/plumbum/commands/base.pyc in run(self, args, **kwargs)
    223         """
    224         with self.bgrun(args, **kwargs) as p:
--> 225             return p.run()
    226 
    227 

/Users/bhyde/.pyenv/versions/2.7.12/envs/bh-venv-2.7.12/lib/python2.7/site-packages/plumbum/commands/base.pyc in runner()
    185             was_run[0] = True
    186             try:
--> 187                 return run_proc(p, retcode, timeout)
    188             finally:
    189                 del p.run  # to break cyclic reference p -> cell -> p

/Users/bhyde/.pyenv/versions/2.7.12/envs/bh-venv-2.7.12/lib/python2.7/site-packages/plumbum/commands/processes.pyc in run_proc(proc, retcode, timeout)
    204         stderr = stderr.decode(proc.custom_encoding, "ignore")
    205 
--> 206     return _check_process(proc, retcode, timeout, stdout, stderr)
    207 
    208 

/Users/bhyde/.pyenv/versions/2.7.12/envs/bh-venv-2.7.12/lib/python2.7/site-packages/plumbum/commands/processes.pyc in _check_process(proc, retcode, timeout, stdout, stderr)
     21 #===================================================================================================
     22 def _check_process(proc, retcode, timeout, stdout, stderr):
---> 23     proc.verify(retcode, timeout, stdout, stderr)
     24     return proc.returncode, stdout, stderr
     25 

/Users/bhyde/.pyenv/versions/2.7.12/envs/bh-venv-2.7.12/lib/python2.7/site-packages/plumbum/machines/base.pyc in verify(self, retcode, timeout, stdout, stderr)
     20             elif self.returncode != retcode:
     21                 raise ProcessExecutionError(getattr(self, "argv", None), self.returncode,
---> 22                     stdout, stderr)
     23 
     24 

<type 'str'>: (<type 'exceptions.UnicodeEncodeError'>, UnicodeEncodeError('ascii', u'\u25cf\n', 0, 1, 'ordinal not in range(128)'))

In [3]: 

[smheidrich]

Just a note: This seems to be the same thing that was mentioned here: https://github.com/tomerfiliba/plumbum/issues/359#issuecomment-358503165 And like most Unicode-related issues, it only occurs in Python 2, not 3. As @junpengl found out, the problem is caused by str() here: https://github.com/tomerfiliba/plumbum/blob/b1ac1182dc5557f1caff87dd4f1bc5b9388359eb/plumbum/commands/processes.py#L81-L83 Not sure yet how this could be fixed...

[erichulser]

I'd opened a bug before seeing these issues, I posted a suggested fix here: #486