Textualize
[BUG] Writing JSON to STDOUT
Describe the bug A clear and concise description of what the bug is.
I have been attempting to use rich to output a very large JSON into STDOUT.
To Reproduce A minimal code example that reproduces the problem would be a big help if you can provide it. If the issue is visual in nature, consider posting a screenshot.
test_rich.py
from rich.console import Console
console = Console()
console.print_json(data=[{"hello": {"big wide": "world"}} for _ in range(50000)])
# Please note that my data is a decent bit larger
$ python test_rich.py | jq
parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 14, column 9
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "my_module/cli/__init__.py", line 21, in main
args.func(args)
File "my_module/cli/commands/search.py", line 143, in cli_search
write_file(results, **write_args)
File "my_module/cli/utils.py", line 115, in write_file
_write_screen(results_list)
File "my_module/cli/utils.py", line 82, in _write_screen
console.print_json(data=search_results)
File "./.venv/lib/python3.9/site-packages/rich/console.py", line 1643, in print_json
self.print(json_renderable)
File "./.venv/lib/python3.9/site-packages/rich/console.py", line 1615, in print
self._buffer.extend(new_segments)
File "./.venv/lib/python3.9/site-packages/rich/console.py", line 825, in __exit__
self._exit_buffer()
File "./.venv/lib/python3.9/site-packages/rich/console.py", line 784, in _exit_buffer
self._check_buffer()
File ./.venv/lib/python3.9/site-packages/rich/console.py", line 1872, in _check_buffer
self.file.write(text)
BrokenPipeError: [Errno 32] Broken pipe
Platform What platform (Win/Linux/Mac) are you running on? What terminal software are you using?
macOS, I have tried the normal Terminal, iTerm 2, and VSCode Integrated Terminal.
Diagnose
$ python -m rich.diagnose
╭─────────────────────── <class 'rich.console.Console'> ───────────────────────╮
│ A high level console interface. │
│ │
│ ╭──────────────────────────────────────────────────────────────────────────╮ │
│ │ <console width=80 ColorSystem.TRUECOLOR> │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ │
│ color_system = 'truecolor' │
│ encoding = 'utf-8' │
│ file = <_io.TextIOWrapper name='<stdout>' mode='w' │
│ encoding='utf-8'> │
│ height = 40 │
│ is_alt_screen = False │
│ is_dumb_terminal = False │
│ is_interactive = True │
│ is_jupyter = False │
│ is_terminal = True │
│ legacy_windows = False │
│ no_color = False │
│ options = ConsoleOptions( │
│ size=ConsoleDimensions(width=80, height=40), │
│ legacy_windows=False, │
│ min_width=1, │
│ max_width=80, │
│ is_terminal=True, │
│ encoding='utf-8', │
│ max_height=40, │
│ justify=None, │
│ overflow=None, │
│ no_wrap=False, │
│ highlight=None, │
│ markup=None, │
│ height=None │
│ ) │
│ quiet = False │
│ record = False │
│ safe_box = True │
│ size = ConsoleDimensions(width=80, height=40) │
│ soft_wrap = False │
│ stderr = False │
│ style = None │
│ tab_size = 8 │
│ width = 80 │
╰──────────────────────────────────────────────────────────────────────────────╯
$ python -m rich._windows
platform="Darwin"
WindowsConsoleFeatures(vt=False, truecolor=False)
$ pip freeze | grep rich
rich @ file:///Users/aidan/Library/Caches/pypoetry/artifacts/a2/2b/30/30eb6b7558c858219ff0094233e8dd6414d7733fe81ef92169292d774b/rich-10.12.0-py3-none-any.whl
Works fine here. The broken pipe suggests it is something to do with the mechanics of piping the data in to JQ.
Are you using a custom shell?
What happens if you don't pipe it in to jq?
Since I can't reproduce it, can you help me narrow down what the trigger is? What happens if you output a just 10 objects? Does it break if you don't pipe it in to jq?
Does your three line example break for you? I can see that the traceback has some additional code.
@thehappydinoa Any more information. Will close in a week if I can't reproduce this.
Hi @willmcgugan I have been unable to reproduce. My apologies but the best thing for my use case was to be able to revert to standard print statements. Thank you for all the help.
Did I solve your problem?
Consider sponsoring the ongoing work on Rich and Textual.
Or buy me a coffee to say thanks.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Hi @willmcgugan, this issue still persists and I think I have a minimal reproduction.
# file: foo.py
from rich.console import Console
console = Console()
console.print("foo")
console.print("bar")
console.print("baz")
Running the following bash command reproduces the error: python foo.py | grep -q "foo".
A few points to note:
- the
-qflag is needed for the reproduction - this is documented in the official Python docs here
Traceback:
Traceback (most recent call last):
File "/home/guacs/open-source/pdm/trial.py", line 7, in <module>
console.print("bar")
File "/home/guacs/open-source/pdm/.venv/lib/python3.11/site-packages/rich/console.py", line 1673, in print
with self:
File "/home/guacs/open-source/pdm/.venv/lib/python3.11/site-packages/rich/console.py", line 865, in __exit__
self._exit_buffer()
File "/home/guacs/open-source/pdm/.venv/lib/python3.11/site-packages/rich/console.py", line 823, in _exit_buffer
self._check_buffer()
File "/home/guacs/open-source/pdm/.venv/lib/python3.11/site-packages/rich/console.py", line 2065, in _check_buffer
self.file.flush()
BrokenPipeError: [Errno 32] Broken pipe
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe
EDIT: added traceback
Hmm yes just saw this regress after I switched to rich, but it can be remedied in the interim for integration in library code for example (who would otherwise see the same error if using regular print statements) with the following context manager.
import os
import sys
__all__ = ("pipe_cleanup",)
def pipe_cleanup() -> None:
"""Handle BrokenPipeError: redirect output to devnull"""
devnull = os.open(os.devnull, os.O_WRONLY)
os.dup2(devnull, sys.stdout.fileno())
sys.exit(1) # Python exits with error code 1 on EPIPE
- Note: this function is from the stdlib docs here (via this StackOverflow answer)
from contextlib import AbstractContextManager
from types import TracebackType
from .pipes import pipe_cleanup
__all__ = ("SuppressBrokenPipeError",)
class SuppressBrokenPipeError(AbstractContextManager):
def __exit__(
self,
__exc_type: type[BaseException] | None,
__exc_value: BaseException | None,
__traceback: TracebackType | None,
) -> bool | None:
if __exc_type is BrokenPipeError:
return pipe_cleanup()
else:
return super().__exit__(__exc_type, __exc_value, __traceback)
I incorporated this into my library like so:
from contextlib import contextmanager
from rich.console import Console
from .error_handlers import SuppressBrokenPipeError
__all__ = ("FugitConsole", "fugit_console")
class FugitConsole:
console: Console
use_pager: bool
page_with_styles: bool
def __init__(self, use_pager: bool = False, page_with_styles: bool = True):
self.console: Console = Console()
self.use_pager: bool = use_pager
self.page_with_styles: bool = page_with_styles
@contextmanager
def pager_available(self):
"""Uses console pagination if `DisplayConfig` switched this setting on."""
if self.use_pager:
with self.console.pager(styles=self.page_with_styles):
yield self
else:
yield self
def print(self, output: str, end="", style=None) -> None:
"""
Report output through the rich console, but don't style at all if rich was set to
no_color (so no bold, italics, etc. either), and avoid broken pipe errors when
piping to `head` etc.
"""
with_style = style if fugit_console.no_color else None
with SuppressBrokenPipeError():
fugit_console.console.print(output, end=end, style=with_style)
"""
Global `rich.console.Console` instance modified by a model validator upon initialisation
of `fugit.interfaces.display.DisplayConfig` or its subclass, the main `DiffConfig` model.
"""
fugit_console = FugitConsole()
I can then test that piping output of my package's CLI entrypoint to | head doesn't truncate with a BrokenPipeError afterwards.
