ipykernel icon indicating copy to clipboard operation
ipykernel copied to clipboard

Prints from ray remote are coming in a previous notebook cell

Open scngupta-dsp opened this issue 9 months ago • 1 comments

Issue Description: When using ray in a notebook, the prints from a ray future are coming in previous cell instead of the cell where the future is executed

To Reproduce Refer minimal jupyter notebook example shown in the snapshot

Expected behavior The prints from ray future should come in the cell where the future is executed

Logs: Refer snapshot below

Image

Desktop:

  • OS: Ubuntu (WSL)
  • OS Version: 22.04.3 LTS
  • Python: 3.12

Package Version


aiosignal 1.3.2 asttokens 3.0.0 attrs 25.1.0 certifi 2025.1.31 charset-normalizer 3.4.1 click 8.1.8 comm 0.2.2 debugpy 1.8.12 decorator 5.2.1 executing 2.2.0 filelock 3.17.0 frozenlist 1.5.0 idna 3.10 ipykernel 6.29.5 ipython 8.32.0 ipywidgets 8.1.5 jedi 0.19.2 jsonschema 4.23.0 jsonschema-specifications 2024.10.1 jupyter_client 8.6.3 jupyter_core 5.7.2 jupyterlab_widgets 3.0.13 matplotlib-inline 0.1.7 msgpack 1.1.0 nest-asyncio 1.6.0 packaging 24.2 parso 0.8.4 pexpect 4.9.0 pip 25.0.1 platformdirs 4.3.6 prompt_toolkit 3.0.50 protobuf 5.29.3 psutil 7.0.0 ptyprocess 0.7.0 pure_eval 0.2.3 Pygments 2.19.1 python-dateutil 2.9.0.post0 PyYAML 6.0.2 pyzmq 26.2.1 ray 2.42.1 referencing 0.36.2 requests 2.32.3 rpds-py 0.23.1 setuptools 75.8.2 six 1.17.0 stack-data 0.6.3 tornado 6.4.2 traitlets 5.14.3 typing_extensions 4.12.2 urllib3 2.3.0 wcwidth 0.2.13 wheel 0.45.1 widgetsnbextension 4.0.13

Additional Information After downgrading ipykernel to 6.26.0, the normal behavior is observed. Refer snapshot below

Image

scngupta-dsp avatar Feb 27 '25 13:02 scngupta-dsp

I suspect you're hitting #1289, which has some more information. Unfortunately, there is no easy fix with a public API.

minrk avatar Mar 07 '25 12:03 minrk

I've just figured out some hacky way to mitigate this issue.

In brief, use IPython Event hooks to generate a "logging background thread/coroutine" upon cell execution, and redirect Ray logs to that logging thread/coroutine just created in current cell.

It seems that after #1451 's release, Ray logs would display in the latest cell by default. Looking forward to this!

Clouder0 avatar Oct 17 '25 22:10 Clouder0

IPykernel 7.1.0 has been released with fix #1451. I think this should fix the problem here, it would be good to receive confirmation of this so we can close this issue.

ianthomas23 avatar Oct 27 '25 09:10 ianthomas23

IPykernel 7.1.0 has been released with fix #1451. I think this should fix the problem here, it would be good to receive confirmation of this so we can close this issue.

Fix confirmed for 7.1.0

Clouder0 avatar Oct 27 '25 14:10 Clouder0

Thanks for the confirmation @Clouder0.

Closing this as fixed by #1451.

ianthomas23 avatar Oct 27 '25 15:10 ianthomas23