`ipywidgets.Output` context manager fails to capture `stdout` in `asyncio` loop after await

[basnijholt]

trafficstars

Description

[!NOTE] Seems related to "Interacting with output widgets from background threads"

When using an ipywidgets.Output widget as a context manager (with out:) inside an asyncio loop, stdout (e.g., from print()) is correctly captured by the Output widget only for the first iteration of the loop before an await occurs. In subsequent iterations, after an await (like asyncio.sleep()), print() statements that are still within the with out: block are no longer captured by the Output widget. Instead, they appear in the standard cell output area.

https://github.com/user-attachments/assets/feab4432-2cf7-4779-99f5-a156321dee36

Reproduce

1. Create a new JupyterLab (or Jupyter Notebook) notebook.

2. Ensure ipywidgets is installed (e.g., version 7.8.5 or 8.1.7).

3. Paste and run the following code in a cell:

   from ipywidgets import Output
   from IPython.display import display
   import time
   import asyncio
   import sys
   import ipywidgets
   import IPython
   
   print(f"Python version: {sys.version}")
   print(f"ipywidgets version: {ipywidgets.__version__}")
   print(f"IPython version: {IPython.__version__}")
   
   # Create and display the Output widget
   out = Output(layout={"border": "1px solid black", "min_height": "50px"})
   display(out)
   
   # Async function to update the Output widget
   async def update_output_widget_loop():
       i = 0
       while i < 5: # Loop a few times to demonstrate the issue
           await asyncio.sleep(1) # Simulate async work
           current_time = time.time()
           with out:
               # This print should always go to the 'out' widget
               print(f"Inside Output Widget (Attempt {i}): {current_time}")
   
           # For comparison, print to standard cell output
           print(f"Standard Cell Output (Attempt {i}): {current_time}")
           i += 1
   
   # Run the async task
   asyncio.create_task(update_output_widget_loop())
   

4. Observe the output:

   • The version information is printed to the standard cell output.
   • The Output widget (out) is displayed with a border.
   • After the first second:
     • The line "Inside Output Widget (Attempt 0): ..." correctly appears inside the bordered Output widget.
     • The line "Standard Cell Output (Attempt 0): ..." appears in the standard cell output area.
   • After the second second (and subsequent seconds):
     • Error: The line "Inside Output Widget (Attempt 1): ..." incorrectly appears in the standard cell output area, not inside the bordered Output widget. This continues for "Attempt 2", "Attempt 3", etc.
     • The Output widget remains either empty or only contains the output from "Attempt 0".

Expected behavior

All print() statements executed within the with out: block (i.e., "Inside Output Widget (Attempt X): ...") should consistently be captured and displayed inside the bordered Output widget (out) for every iteration of the asyncio loop. The Output widget's context manager should reliably redirect stdout for the duration of its scope, even across await points in an asyncio task.

Context

• ipywidgets version:
  • Tested with 7.8.5
  • Tested with 8.1.7 (Issue persists with both versions. The MRE will print the version being used when run.)
• Operating System and version: macOS 15.4.1
• Browser and version:
  • Brave Browser Version 1.78.97 Chromium: 136.0.7103.93 (Official Build) (arm64)
  • Firefox 138.0.1 (aarch64)

Python Version (from MRE): 3.13.1 (main, Dec 6 2024, 20:13:21) [Clang 18.1.8 ] IPython Version (from MRE): 9.2.0

Troubleshoot Output

/Users/marcellaholm/Work/pipefunc/.venv/bin/python: No module named pip
$PATH:
	/Users/marcellaholm/Work/pipefunc/.venv/bin
	/Users/marcellaholm/micromamba/condabin
	/Users/marcellaholm/.dotbins/macos/arm64/bin
	/opt/homebrew/bin
	/opt/homebrew/sbin
	/nix/var/nix/profiles/default/bin
	/Users/marcellaholm/.local/bin
	/usr/local/bin
	/System/Cryptexes/App/usr/bin
	/usr/bin
	/bin
	/usr/sbin
	/sbin
	/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin
	/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin
	/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin
	/Applications/iTerm.app/Contents/Resources/utilities
sys.path:
/Users/marcellaholm/Work/pipefunc/.venv/bin
/Users/marcellaholm/.local/share/uv/python/cpython-3.13.1-macos-aarch64-none/lib/python313.zip
/Users/marcellaholm/.local/share/uv/python/cpython-3.13.1-macos-aarch64-none/lib/python3.13
/Users/marcellaholm/.local/share/uv/python/cpython-3.13.1-macos-aarch64-none/lib/python3.13/lib-dynload
/Users/marcellaholm/Work/pipefunc/.venv/lib/python3.13/site-packages
/Users/marcellaholm/Work/pipefunc
sys.executable:
/Users/marcellaholm/Work/pipefunc/.venv/bin/python
sys.version:
3.13.1 (main, Dec  6 2024, 20:13:21) [Clang 18.1.8 ]
platform.platform():
macOS-15.4.1-arm64-arm-64bit-Mach-O
which -a jupyter:
/Users/marcellaholm/Work/pipefunc/.venv/bin/jupyter

Command Line Output

No relevant messages here.

Browser Output

Nothing relevant

If using JupyterLab

• JupyterLab version: 4.4.2

Installed Labextensions

at 13:57:36 |ψ❯  jupyter labextension list
JupyterLab v4.4.2
/Users/marcellaholm/Work/pipefunc/.venv/share/jupyter/labextensions
        jupyterlab-jupytext v1.4.4 enabled OK (python, jupytext)
        anywidget v0.9.18 enabled OK
        jupyterlab_pygments v0.3.0 enabled OK (python, jupyterlab_pygments)
        ipyparallel-labextension v9.0.1 enabled OK
        @pyviz/jupyterlab_pyviz v3.0.4 enabled OK
        @jupyter-notebook/lab-extension v7.4.2 enabled OK
        @jupyter-widgets/jupyterlab-manager v5.0.15 enabled OK (python, jupyterlab_widgets)

[jasongrout]

Seems related to "Interacting with output widgets from background threads"

Yes, I think you correctly identified the root cause, and my guess is that the workarounds mentioned in that documentation would work in this case as well.

[asaboor-gh]

In my observation, not just sys.stdout but also any kind of display data is lost too (in Jupyter logs in Lab). So I created JupyTimer to remove this type of issues in a monitor decorator that can time, throttle and denounce a function. Throttling is still easy but denouncing led me to dig this problem deeper. With JupyTimer, time monitoring is done on frontend, so you don't have to leave your main thread ever. And a bounce point is you can use a nice gui to control necessary stuff at runtime.

https://github.com/user-attachments/assets/fcdf4028-ad38-499d-a338-5b1744b657f4