rich
rich copied to clipboard
Textualize
[BUG] UnicodeEncodeError on Windows with ruler.
Describe the bug
The issue started appearing in my test suite, and I could boil it down to this reproducible example. I am using rich 12.5.1.
# Content of script.py.
from rich.console import Console
if __name__ == '__main__':
console = Console()
console.rule() # or console.print("─")
# Content of run.py.
import subprocess
result = subprocess.run(("python", "rich_script.py"), check=True)
and then I am executing pytest run.py, which results in
Click to expand
================================================= test session starts =================================================
platform win32 -- Python 3.10.5, pytest-7.1.2, pluggy-1.0.0
rootdir: C:\Users\TobiasR\git\pytask, configfile: tox.ini
plugins: anyio-3.6.1, cov-3.0.0, forked-1.4.0, xdist-2.5.0
collected 0 items / 2 errors
======================================================= ERRORS ========================================================
_______________________________________________ ERROR collecting run.py _______________________________________________
run.py:3: in <module>
result = subprocess.run(("python", "rich_script.py"), check=True)
C:\tools\miniconda3\envs\pytask\lib\subprocess.py:524: in run
raise CalledProcessError(retcode, process.args,
E subprocess.CalledProcessError: Command '('python', 'rich_script.py')' returned non-zero exit status 1.
--------------------------------------------------- Captured stderr ---------------------------------------------------
Traceback (most recent call last):
File "C:\Users\TobiasR\git\pytask\rich_script.py", line 6, in <module>
console.rule()
File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\console.py", line 1553, in rule
self.print(rule)
File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\console.py", line 1646, in print
with self:
File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\console.py", line 848, in __exit__
self._exit_buffer()
File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\console.py", line 806, in _exit_buffer
self._check_buffer()
File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\console.py", line 1999, in _check_buffer
legacy_windows_render(
File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\_windows_renderer.py", line 17, in legacy_windows_render
term.write_styled(text, style)
File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\_win32_console.py", line 442, in write_styled
self.write_text(text)
File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\_win32_console.py", line 403, in write_text
self.write(text)
File "C:\tools\miniconda3\envs\pytask\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-78: character maps to <undefined>
_______________________________________________ ERROR collecting run.py _______________________________________________
run.py:3: in <module>
result = subprocess.run(("python", "rich_script.py"), check=True)
C:\tools\miniconda3\envs\pytask\lib\subprocess.py:524: in run
raise CalledProcessError(retcode, process.args,
E subprocess.CalledProcessError: Command '('python', 'rich_script.py')' returned non-zero exit status 1.
--------------------------------------------------- Captured stderr ---------------------------------------------------
Traceback (most recent call last):
File "C:\Users\TobiasR\git\pytask\rich_script.py", line 6, in <module>
console.rule()
File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\console.py", line 1553, in rule
self.print(rule)
File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\console.py", line 1646, in print
with self:
File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\console.py", line 848, in __exit__
self._exit_buffer()
File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\console.py", line 806, in _exit_buffer
self._check_buffer()
File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\console.py", line 1999, in _check_buffer
legacy_windows_render(
File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\_windows_renderer.py", line 17, in legacy_windows_render
term.write_styled(text, style)
File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\_win32_console.py", line 442, in write_styled
self.write_text(text)
File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\_win32_console.py", line 403, in write_text
self.write(text)
File "C:\tools\miniconda3\envs\pytask\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-78: character maps to <undefined>
=============================================== short test summary info ===============================================
ERROR run.py - subprocess.CalledProcessError: Command '('python', 'rich_script.py')' returned non-zero exit status 1.
ERROR run.py - subprocess.CalledProcessError: Command '('python', 'rich_script.py')' returned non-zero exit status 1.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================== 2 errors in 0.72s ==================================================
The error does not occur when I
- switch to rich 12.4.4 or ...
- set PYTHONIOENCODING to utf-8 or ...
- execute
python run.py
Platform
Click to expand
The issue occurred only on Windows and I am running Powershell Core embedded in Windows Terminal.
python -m rich.diagnose
pip freeze | grep rich
╭───────────────────────── <class 'rich.console.Console'> ─────────────────────────╮
│ A high level console interface. │
│ │
│ ╭──────────────────────────────────────────────────────────────────────────────╮ │
│ │ <console width=120 ColorSystem.TRUECOLOR> │ │
│ ╰──────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ color_system = 'truecolor' │
│ encoding = 'utf-8' │
│ file = <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'> │
│ height = 30 │
│ is_alt_screen = False │
│ is_dumb_terminal = False │
│ is_interactive = True │
│ is_jupyter = False │
│ is_terminal = True │
│ legacy_windows = False │
│ no_color = False │
│ options = ConsoleOptions( │
│ size=ConsoleDimensions(width=120, height=30), │
│ legacy_windows=False, │
│ min_width=1, │
│ max_width=120, │
│ is_terminal=True, │
│ encoding='utf-8', │
│ max_height=30, │
│ justify=None, │
│ overflow=None, │
│ no_wrap=False, │
│ highlight=None, │
│ markup=None, │
│ height=None │
│ ) │
│ quiet = False │
│ record = False │
│ safe_box = True │
│ size = ConsoleDimensions(width=120, height=30) │
│ soft_wrap = False │
│ stderr = False │
│ style = None │
│ tab_size = 8 │
│ width = 120 │
╰──────────────────────────────────────────────────────────────────────────────────╯
╭── <class 'rich._windows.WindowsConsoleFeatures'> ───╮
│ Windows features available. │
│ │
│ ╭─────────────────────────────────────────────────╮ │
│ │ WindowsConsoleFeatures(vt=True, truecolor=True) │ │
│ ╰─────────────────────────────────────────────────╯ │
│ │
│ truecolor = True │
│ vt = True │
╰─────────────────────────────────────────────────────╯
╭────── Environment Variables ───────╮
│ { │
│ 'TERM': None, │
│ 'COLORTERM': None, │
│ 'CLICOLOR': None, │
│ 'NO_COLOR': None, │
│ 'TERM_PROGRAM': None, │
│ 'COLUMNS': None, │
│ 'LINES': None, │
│ 'JUPYTER_COLUMNS': None, │
│ 'JUPYTER_LINES': None, │
│ 'JPY_PARENT_PID': None, │
│ 'VSCODE_VERBOSE_LOGGING': None │
│ } │
╰────────────────────────────────────╯
platform="Windows"
Not the first time we've been bitten by Windows CP-1252 encoding.
The solution may be to force utf8
I can't reproduce this. Running the same Python version, Rich version, tried legacy/non-legacy terminals, tried forcing the encoding to cp1252 and utf-8, tried with an without running in a subprocess. Every time the rule renders without an error. I'm not sure where else the difference could lie 🤔
It's interesting that you've supplied the example as a subprocess - is that because you can only reproduce the issue when you run it in a subprocess?
I noticed in the "Platform" information you shared, legacy_windows = False, yet the code path which raises the error implies that legacy_windows = True. I wonder if for some reason, the console reports legacy Windows in the subprocess but not in the parent process? Regardless, I can't reproduce even when I force legacy and cp1252 on my end, so I'm stumped at the moment!
Also, comparing the version that works with you with the latest version, I can't see any relevant (encoding-related) changes at all: https://github.com/Textualize/rich/compare/v12.4.4...v12.5.1
Yes, this error only occurs when I am using pytest to execute the subprocess script with pytest run.py. python run.py works. So, for me, it seemed like pytest influences which console settings are detected in the subprocess.
I also had it after an update in my windows tests in github https://github.com/InfrastructureAsCode-ch/nornir_rich/runs/7483512482?check_suite_focus=true
I am using PYTHONIOENCODING: utf-8 at the moment and it works fine but for me on windows pytest . worked well. So I have the problam only in the workflow on GitHub
For tests, you probably want to set legacy_windows to True or False, so you aren't at the mercy of the environment. Ditto with the other parameters that auto-detect.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
I am seeing this in a azure pipelines run of a conda-forge feedstock. Should I open another issue? Relevant bits:
Python 3.8, rich 12.6.0-pyhd8ed1ab_0 from conda-forge.
ConsoleOptions
From this log
| | render_options = ConsoleOptions( | |
| | size=ConsoleDimensions(width=79, height=25), | |
| | legacy_windows=True, | |
| | min_width=1, | |
| | max_width=79, | |
| | is_terminal=False, | |
| | encoding='cp1252', | |
| | max_height=25, | |
| | justify=None, | |
| | overflow=None, | |
| | no_wrap=None, | |
| | highlight=None, | |
| | markup=None, | |
| | height=None | |
| | ) | |
when rendering this string
text = '. \u2554\u2584\u2593\u2593\u2593\u2593\u2584'
This is still an issue. In CP1252 encoding (which is what the :! operator in nvim uses) console.print(Rule()) and console.rule() throw an error when no title is provided. They work fine with a title provided.
