Inside a Python venv, the Raku kernel bombs in notebook session, but restarts (-ish)

[bazzaar]

The following are observations whilst running jupyter-notebook in a venv.

The background is that OpenSuse 15.4 has "alternate" Pythons, version 3.10 can be installed by the package manager (zypper/YAST) alongside version 3.7, the latter being the 'system' Python.

The more up-to-date version 3.10 python can then be used as the venv, with extra functionality provided by python packages (incl. Jupyter) being installed locally in the users home dir.

Manually activating the venv session, and then running jupyter-notebook ....

Opening a (new / pre-existing) Raku notebook in jupyter-notebook :

• The jupyter console shows that Raku fires up
• The Raku kernel then bombs (red button), though the notebook session continues
• the errors in the jupyter log are Raku-derived (messaging to undefined comm id)
• but then the Raku kernel automatically restarts in the session
• the still running notebook seems to fully operational and the kernel is stable
• the jupyter log warns of a root kernel, which it didn't the first time

Choosing to open a (new / pre-existing) Python notebook instead :

• The Python notebook kernel does not bomb

• However there are two 'startup' messages in the console log

  [IPKernelApp] ERROR | No such comm target registered: jupyter.widget.control
  [IPKernelApp] WARNING | No such comm: xxxxxxxxxx-xxxx-xxx-xxx-xxxxxxxxxx
  

• these messages don't get displayed in the raku scenario (above)

• the two messages seem to be a 'feature' of running jupyter under a venv

• they've been reported as issues for some time, but they apparently are not due to significant errors, and maybe aren't going to be fixed anytime soon.

Manually closing down jupyter, then deactivating the venv ....

I wonder if these two dubious messages are causing the raku kernel behaviour above?

[bduggan]

Thanks for the detailed report. Interesting. If I'm following...it sounds like some bespoke error handling by the python kernel. Could you add the specific error messages coming from the raku kernel that you are seeing?

[bazzaar]

Here's the raku-specific section of the jupyter log.

2022-09-01T10:53:00.334537+01:00 (6) info: iopub: sending execute_input message 2022-09-01T10:53:00.430829+01:00 (6) debug: shell got a message: kernel_info_request 2022-09-01T10:53:00.431460+01:00 (6) info: shell: sending kernel_info_reply message 2022-09-01T10:53:00.453474+01:00 (6) info: iopub: sending status message 2022-09-01T10:53:00.487660+01:00 (8) debug: ctl got a message: kernel_info_request 2022-09-01T10:53:00.498667+01:00 (6) debug: shell got a message: kernel_info_request 2022-09-01T10:53:00.499069+01:00 (6) info: shell: sending kernel_info_reply message 2022-09-01T10:53:00.514786+01:00 (6) info: iopub: sending status message 2022-09-01T10:53:00.541061+01:00 (6) debug: shell got a message: kernel_info_request 2022-09-01T10:53:00.541371+01:00 (6) info: shell: sending kernel_info_reply message 2022-09-01T10:53:00.560442+01:00 (6) info: iopub: sending status message 2022-09-01T10:53:00.581844+01:00 (6) debug: shell got a message: comm_open 2022-09-01T10:53:00.583664+01:00 (6) info: iopub: sending comm_close message 2022-09-01T10:53:00.609212+01:00 (6) debug: shell got a message: comm_msg 2022-09-01T10:53:00.609917+01:00 (6) debug: comm_msg for vvvvvvvv-wwww-xxxx-yyyy-zzzzzzzzzzzz 2022-09-01T10:53:00.611014+01:00 (6) debug: sending method request_states to vvvvvvvv-wwww-xxxx-yyyy-zzzzzzzzzzzz 2022-09-01T10:53:00.659629+01:00 (6) error: shell: No such method 'in' for invocant of type 'Any'. Did you mean 'min'? 2022-09-01T10:53:00.666729+01:00 (6) error: trace: in method throw at SETTING::src/core.c/Exception.pm6 line 64 in any throw_or_die at gen/moar/Metamodel.nqp line 60 in any report-method-not-found at gen/moar/BOOTSTRAP/v6c.nqp line 5117 in any at gen/moar/BOOTSTRAP/v6c.nqp line 5055 in method send-to-comm at /home/uuu/.raku/sources/.... (Jupyter::Kernel::Comms) line 32 in block at /home/uuu/.raku/sources/.... (Jupyter::Kernel) line 237 in block at /home/uuu/.raku/sources/.... (Jupyter::Kernel) line 131 in block at /home/uuu/.raku/sources/.... (Jupyter::Kernel) line 126 in block at /home/uuu/.raku/sources/.... (Jupyter::Kernel) line 124 in block at SETTING::src/core.c/Promise.pm6 line 295 in block at SETTING::src/core.c/ThreadPoolScheduler.pm6 line 899 in block at SETTING::src/core.c/ThreadPoolScheduler.pm6 line 266 in block at SETTING::src/core.c/ThreadPoolScheduler.pm6 line 247 in method run-one at SETTING::src/core.c/ThreadPoolScheduler.pm6 line 244 in block at SETTING::src/core.c/ThreadPoolScheduler.pm6 line 285

Was thinking similar, that if the "error" message (seen in the python scenario) doesn't have a valid comm id, then the variable doesn't get resolved by Raku, and so the "in" method fails. The python kernel might have a neat way of ignoring these dubious messages?

[bduggan]

Okay -- I made this into a warning, so that the kernel at least doesn't crash. Any better?

[bazzaar]

Unfortunately I still see the same behaviour, but maybe I'm doing something wrong?

• Here is the workflow (edited)

uuuu:~/Documents> source ~/.venv/py310env/bin/activate

(py310env) uuuu:~/Documents> zef install --force-install Jupyter::Kernel ===> Searching for: Jupyter::Kernel ===> Updating fez mirror: https://360.zef.pm/ ===> Updated fez mirror: https://360.zef.pm/ ===> Updating rea mirror: https://raw.githubusercontent.com/Raku/REA/main/META.json ===> Updated rea mirror: https://raw.githubusercontent.com/Raku/REA/main/META.json ===> Testing: Jupyter::Kernel:ver<0.0.18> ===> Testing [OK] for Jupyter::Kernel:ver<0.0.18> ===> Installing: Jupyter::Kernel:ver<0.0.18>

1 bin/ script [jupyter-kernel.raku] installed to: /home/uuuu/.raku/bin

(py310env) uuuu:~/Documents> jupyter-kernel.raku --generate-config --force Creating directory /home/uuuu/.local/share/jupyter/kernels/raku Writing kernel.json to /home/uuuu/.local/share/jupyter/kernels/raku/kernel.json Copying logo-32x32.png to /home/uuuu/.local/share/jupyter/kernels/raku Copying logo-64x64.png to /home/uuuu/.local/share/jupyter/kernels/raku Congratulations, configuration files have been successfully written! Happy Perling! <- jupyter console --kernel=raku

• here is the jupyter console

(py310env) uuuu:~/Documents> jupyter-notebook [I 22:25:17.637 NotebookApp] Loading IPython parallel extension [I 2022-09-03 22:25:18.062 LabApp] JupyterLab extension loaded from /home/uuuu/.venv/py310env/lib64/python3.10/site-packages/jupyterlab [I 2022-09-03 22:25:18.063 LabApp] JupyterLab application directory is /home/uuuu/.venv/py310env/share/jupyter/lab [I 22:25:18.067 NotebookApp] Serving notebooks from local directory: /home/uuuu/Documents [I 22:25:18.067 NotebookApp] Jupyter Notebook 6.4.12 is running at: [I 22:25:18.067 NotebookApp] http://localhost:8888/?token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx [I 22:25:18.067 NotebookApp] or http://127.0.0.1:8888/?token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx [I 22:25:18.067 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 22:25:18.092 NotebookApp]

To access the notebook, open this file in a browser:
    file:///home/uuuu/.local/share/jupyter/runtime/nbserver-eeeee-open.html
Or copy and paste one of these URLs:
    http://localhost:8888/?token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 or http://127.0.0.1:8888/?token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

[I 22:25:30.778 NotebookApp] Creating new notebook in [I 22:25:31.888 NotebookApp] Kernel started: zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz, name: raku Welcome to Rakudo™ v2022.07. Implementing the Raku® Programming Language v6.d. Built on MoarVM version 2022.07.

[I 22:25:32.979 NotebookApp] Adapting from protocol version 5.0 (kernel zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz) to 5.3 (client). [W 22:25:33.127 NotebookApp] 404 GET /static/components/codemirror/mode/raku/raku.js?v=wwwwwwwwwww (127.0.0.1) 7.970000ms referer=http://localhost:8888/notebooks/Untitled1.ipynb?kernel_name=raku [I 22:25:34.876 NotebookApp] KernelRestarter: restarting kernel (1/5), new random ports WARNING:root:kernel zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz restarted Welcome to Rakudo™ v2022.07. Implementing the Raku® Programming Language v6.d. Built on MoarVM version 2022.07.

• and here is the jupyter log ...

2022-09-03T22:25:33.156275+01:00 (7) debug: shell got a message: comm_msg 2022-09-03T22:25:33.157217+01:00 (7) debug: comm_msg for bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb 2022-09-03T22:25:33.158247+01:00 (7) debug: sending method request_states to bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb 2022-09-03T22:25:33.206534+01:00 (7) error: shell: No such method 'in' for invocant of type 'Any'. Did you mean 'min'? 2022-09-03T22:25:33.213315+01:00 (7) error: trace: in method throw at SETTING::src/core.c/Exception.pm6 line 64 in any throw_or_die at gen/moar/Metamodel.nqp line 60 in any report-method-not-found at gen/moar/BOOTSTRAP/v6c.nqp line 5117 in any at gen/moar/BOOTSTRAP/v6c.nqp line 5055 in method send-to-comm at /home/uuuu/.raku/sources/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv (Jupyter::Kernel::Comms) line 32 in block at /home/uuuu/.raku/sources/.... (Jupyter::Kernel) line 237 in block at /home/uuuu/.raku/sources/.... (Jupyter::Kernel) line 131 in block at /home/uuuu/.raku/sources/.... (Jupyter::Kernel) line 126 in block at /home/uuuu/.raku/sources/.... (Jupyter::Kernel) line 124 in block at SETTING::src/core.c/Promise.pm6 line 295 in block at SETTING::src/core.c/ThreadPoolScheduler.pm6 line 899 in block at SETTING::src/core.c/ThreadPoolScheduler.pm6 line 266 in block at SETTING::src/core.c/ThreadPoolScheduler.pm6 line 247 in method run-one at SETTING::src/core.c/ThreadPoolScheduler.pm6 line 244 in block at SETTING::src/core.c/ThreadPoolScheduler.pm6 line 285

2022-09-03T22:25:33.231596+01:00 (1) info: iopub: sending stream message 2022-09-03T22:25:35.611379+01:00 (1) info: starting jupyter kernel 2022-09-03T22:25:35.632372+01:00 (1) debug: read /home/uuuu/.local/share/jupyter/runtime/kernel-yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy.json 2022-09-03T22:25:35.632972+01:00 (1) debug: listening on tcp://127.0.0.1 2022-09-03T22:25:35.633975+01:00 (1) debug: setting up control on aaaaa 2022-09-03T22:25:35.668687+01:00 (1) debug: setting up shell on bbbbb 2022-09-03T22:25:35.669423+01:00 (1) debug: setting up iopub on ccccc 2022-09-03T22:25:35.669968+01:00 (1) debug: setting up hb on ddddd 2022-09-03T22:25:35.836046+01:00 (6) info: iopub: sending execute_input message 2022-09-03T22:25:37.019068+01:00 (6) debug: shell got a message: comm_info_request 2022-09-03T22:25:37.021909+01:00 (6) debug: comm_info_request ${:target_name("jupyter.widget")} 2022-09-03T22:25:37.022721+01:00 (6) info: shell: sending comm_info_reply message

[bduggan]

Just zef install Jupyter::Kernel won't get the latest version from git -- you'll want to

git clone github.com:bduggan/p6-jupyter-kernel
cd p6-jupyter-kernel
zef install --force-install .

If it works, then I'll release a new version :-)

[bduggan]

or zef install --force-install http://github.com/bduggan/p6-jupyter-kernel.git should work too

[bazzaar]

Ah, I did wonder if the update hadn't yet reached the repo. Thank for the zef tips, I'm still a bit of a novice on the dev workflow stuff :)

Pleased to report, the Raku kernel no longer crashes in the jupyter notebook session running within a Python venv ( tried both new / existing notebooks ).

Thankyou, @bduggan, for this excellent module.

[bazzaar]

for completeness, this is how the jupyter log looks now

2022-09-04T12:39:19.789387+01:00 (7) debug: shell got a message: comm_msg 2022-09-04T12:39:19.790070+01:00 (7) debug: comm_msg for vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv 2022-09-04T12:39:19.791137+01:00 (7) debug: sending method request_states to vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv 2022-09-04T12:39:19.791633+01:00 (7) warning: could not find comm for vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv, dropping message 2022-09-04T12:39:23.645296+01:00 (7) debug: shell got a message: comm_info_request 2022-09-04T12:39:23.647298+01:00 (7) debug: comm_info_request ${:target_name("jupyter.widget")} 2022-09-04T12:39:23.648263+01:00 (7) info: shell: sending comm_info_reply message

:)

[bduggan]

Great!