ipykernel
ipykernel copied to clipboard
Prints from ray remote are coming in a previous notebook cell
Issue Description: When using ray in a notebook, the prints from a ray future are coming in previous cell instead of the cell where the future is executed
To Reproduce Refer minimal jupyter notebook example shown in the snapshot
Expected behavior The prints from ray future should come in the cell where the future is executed
Logs: Refer snapshot below
Desktop:
- OS: Ubuntu (WSL)
- OS Version: 22.04.3 LTS
- Python: 3.12
Package Version
aiosignal 1.3.2 asttokens 3.0.0 attrs 25.1.0 certifi 2025.1.31 charset-normalizer 3.4.1 click 8.1.8 comm 0.2.2 debugpy 1.8.12 decorator 5.2.1 executing 2.2.0 filelock 3.17.0 frozenlist 1.5.0 idna 3.10 ipykernel 6.29.5 ipython 8.32.0 ipywidgets 8.1.5 jedi 0.19.2 jsonschema 4.23.0 jsonschema-specifications 2024.10.1 jupyter_client 8.6.3 jupyter_core 5.7.2 jupyterlab_widgets 3.0.13 matplotlib-inline 0.1.7 msgpack 1.1.0 nest-asyncio 1.6.0 packaging 24.2 parso 0.8.4 pexpect 4.9.0 pip 25.0.1 platformdirs 4.3.6 prompt_toolkit 3.0.50 protobuf 5.29.3 psutil 7.0.0 ptyprocess 0.7.0 pure_eval 0.2.3 Pygments 2.19.1 python-dateutil 2.9.0.post0 PyYAML 6.0.2 pyzmq 26.2.1 ray 2.42.1 referencing 0.36.2 requests 2.32.3 rpds-py 0.23.1 setuptools 75.8.2 six 1.17.0 stack-data 0.6.3 tornado 6.4.2 traitlets 5.14.3 typing_extensions 4.12.2 urllib3 2.3.0 wcwidth 0.2.13 wheel 0.45.1 widgetsnbextension 4.0.13
Additional Information After downgrading ipykernel to 6.26.0, the normal behavior is observed. Refer snapshot below
I suspect you're hitting #1289, which has some more information. Unfortunately, there is no easy fix with a public API.
I've just figured out some hacky way to mitigate this issue.
In brief, use IPython Event hooks to generate a "logging background thread/coroutine" upon cell execution, and redirect Ray logs to that logging thread/coroutine just created in current cell.
It seems that after #1451 's release, Ray logs would display in the latest cell by default. Looking forward to this!
IPykernel 7.1.0 has been released with fix #1451. I think this should fix the problem here, it would be good to receive confirmation of this so we can close this issue.
IPykernel 7.1.0 has been released with fix #1451. I think this should fix the problem here, it would be good to receive confirmation of this so we can close this issue.
Fix confirmed for 7.1.0
Thanks for the confirmation @Clouder0.
Closing this as fixed by #1451.