notebook
notebook copied to clipboard
jupyter
Connection failed, kernel does not exist
Describe the bug
I'm experiencing connection/kernel issues with my local copy of Jupyter. I'm not sure if this is the right place to report this, but I encounter the issue with both jupyter lab and jupyter notebook so it may be an issue with a common core/server dependency.
To Reproduce
- Launch Jupyter with
jupyter laborjupyter notebook - Create a new notebook with a simple
print('hello world')line - Try to run this line
- Nothing happens, errors in log
Expected behavior
I would Jupyter to find and use the kernel and run my code properly.
Screenshots
With jupyter lab I see the following log:
$ jupyter lab
[I 2022-05-25 17:58:18.321 ServerApp] jupyterlab | extension was successfully linked.
[I 2022-05-25 17:58:18.398 LabApp] JupyterLab extension loaded from /Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-13.1.6/py-jupyterlab-3.4.2-edi2q5wc43g627lu2gkmo2xtawiuv7jy/lib/python3.9/site-packages/jupyterlab
[I 2022-05-25 17:58:18.398 LabApp] JupyterLab application directory is /Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-13.1.6/py-jupyterlab-3.4.2-edi2q5wc43g627lu2gkmo2xtawiuv7jy/share/jupyter/lab
[I 2022-05-25 17:58:18.400 ServerApp] jupyterlab | extension was successfully loaded.
[I 2022-05-25 17:58:18.400 ServerApp] Serving notebooks from local directory: /Users/ajstewart/Downloads
[I 2022-05-25 17:58:18.400 ServerApp] Jupyter Server 1.17.0 is running at:
[I 2022-05-25 17:58:18.400 ServerApp] http://localhost:8888/lab?token=9375a706106d9afad83c4a544c502ad6237d5d3e21ddb9fe
[I 2022-05-25 17:58:18.400 ServerApp] or http://127.0.0.1:8888/lab?token=9375a706106d9afad83c4a544c502ad6237d5d3e21ddb9fe
[I 2022-05-25 17:58:18.400 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2022-05-25 17:58:18.404 ServerApp]
To access the server, open this file in a browser:
file:///Users/ajstewart/Library/Jupyter/runtime/jpserver-67212-open.html
Or copy and paste one of these URLs:
http://localhost:8888/lab?token=9375a706106d9afad83c4a544c502ad6237d5d3e21ddb9fe
or http://127.0.0.1:8888/lab?token=9375a706106d9afad83c4a544c502ad6237d5d3e21ddb9fe
[W 2022-05-25 17:58:22.067 LabApp] Could not determine jupyterlab build status without nodejs
[I 2022-05-25 17:58:22.717 ServerApp] Kernel started: 280c43a0-d3b9-4763-af38-8e49846055a3
[I 2022-05-25 17:58:25.714 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (1/5), new random ports
[I 2022-05-25 17:58:28.743 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (2/5), new random ports
[I 2022-05-25 17:58:31.787 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (3/5), new random ports
[I 2022-05-25 17:58:34.825 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (4/5), new random ports
[I 2022-05-25 17:58:37.868 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (5/5), new random ports
[W 2022-05-25 17:58:40.896 ServerApp] AsyncIOLoopKernelRestarter: restart failed
[W 2022-05-25 17:58:40.898 ServerApp] Kernel 280c43a0-d3b9-4763-af38-8e49846055a3 died, removing from map.
[W 2022-05-25 17:59:22.731 ServerApp] Timeout waiting for kernel_info reply from 280c43a0-d3b9-4763-af38-8e49846055a3
[W 2022-05-25 17:59:22.734 ServerApp] 404 GET /api/kernels/280c43a0-d3b9-4763-af38-8e49846055a3/channels?session_id=1fa21670-fa85-4f65-8c05-12b7f13f9272 (::1): Kernel does not exist: 280c43a0-d3b9-4763-af38-8e49846055a3
[W 2022-05-25 17:59:22.735 ServerApp] 404 GET /api/kernels/280c43a0-d3b9-4763-af38-8e49846055a3/channels?session_id=41ef9206-5db6-4404-88c4-37c5feed8356 (127.0.0.1): Kernel does not exist: 280c43a0-d3b9-4763-af38-8e49846055a3
[I 2022-05-25 17:59:56.360 ServerApp] Kernel started: 63cced8e-a8b3-417e-a20a-6a2d8be40be1
[I 2022-05-25 17:59:59.348 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (1/5), new random ports
[I 2022-05-25 18:00:02.374 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (2/5), new random ports
[I 2022-05-25 18:00:05.418 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (3/5), new random ports
[I 2022-05-25 18:00:08.451 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (4/5), new random ports
[I 2022-05-25 18:00:11.477 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (5/5), new random ports
[W 2022-05-25 18:00:14.523 ServerApp] AsyncIOLoopKernelRestarter: restart failed
[W 2022-05-25 18:00:14.525 ServerApp] Kernel 63cced8e-a8b3-417e-a20a-6a2d8be40be1 died, removing from map.
With jupyter notebook I see the following log:
$ jupyter notebook test.ipynb
[I 18:00:22.873 NotebookApp] Serving notebooks from local directory: /Users/ajstewart/Downloads
[I 18:00:22.873 NotebookApp] Jupyter Notebook 6.4.11 is running at:
[I 18:00:22.873 NotebookApp] http://localhost:8888/?token=002d2f8c5e2fe78ac0d44b9829a661f0f0fcb4f3fa269d8d
[I 18:00:22.873 NotebookApp] or http://127.0.0.1:8888/?token=002d2f8c5e2fe78ac0d44b9829a661f0f0fcb4f3fa269d8d
[I 18:00:22.873 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 18:00:22.879 NotebookApp]
To access the notebook, open this file in a browser:
file:///Users/ajstewart/Library/Jupyter/runtime/nbserver-67477-open.html
Or copy and paste one of these URLs:
http://localhost:8888/?token=002d2f8c5e2fe78ac0d44b9829a661f0f0fcb4f3fa269d8d
or http://127.0.0.1:8888/?token=002d2f8c5e2fe78ac0d44b9829a661f0f0fcb4f3fa269d8d
[W 18:00:25.247 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js (::1) 4.050000ms referer=http://localhost:8888/notebooks/test.ipynb
[I 18:00:25.328 NotebookApp] Kernel started: dc6a45c0-2768-48c9-bbec-ff8e340f3320, name: python3
[I 18:00:28.320 NotebookApp] KernelRestarter: restarting kernel (1/5), new random ports
[I 18:00:31.367 NotebookApp] KernelRestarter: restarting kernel (2/5), new random ports
[I 18:00:34.403 NotebookApp] KernelRestarter: restarting kernel (3/5), new random ports
[I 18:00:37.437 NotebookApp] KernelRestarter: restarting kernel (4/5), new random ports
[I 18:00:40.470 NotebookApp] KernelRestarter: restarting kernel (5/5), new random ports
[W 18:00:43.502 NotebookApp] KernelRestarter: restart failed
[W 18:00:43.504 NotebookApp] Kernel dc6a45c0-2768-48c9-bbec-ff8e340f3320 died, removing from map.
[W 18:01:25.337 NotebookApp] Timeout waiting for kernel_info reply from dc6a45c0-2768-48c9-bbec-ff8e340f3320
[E 18:01:25.344 NotebookApp] Error opening stream: HTTP 404: Not Found (Kernel does not exist: dc6a45c0-2768-48c9-bbec-ff8e340f3320)
[W 18:01:26.800 NotebookApp] 404 GET /api/kernels/dc6a45c0-2768-48c9-bbec-ff8e340f3320/channels?session_id=b7dbf7410a154f7d944fea051d6d4e17 (::1): Kernel does not exist: dc6a45c0-2768-48c9-bbec-ff8e340f3320
[W 18:01:26.803 NotebookApp] 404 GET /api/kernels/dc6a45c0-2768-48c9-bbec-ff8e340f3320/channels?session_id=b7dbf7410a154f7d944fea051d6d4e17 (::1) 7.810000ms referer=None
[W 18:01:29.604 NotebookApp] 404 GET /api/kernels/dc6a45c0-2768-48c9-bbec-ff8e340f3320/channels?session_id=b7dbf7410a154f7d944fea051d6d4e17 (::1): Kernel does not exist: dc6a45c0-2768-48c9-bbec-ff8e340f3320
[W 18:01:29.607 NotebookApp] 404 GET /api/kernels/dc6a45c0-2768-48c9-bbec-ff8e340f3320/channels?session_id=b7dbf7410a154f7d944fea051d6d4e17 (::1) 7.040000ms referer=None
[W 18:01:34.349 NotebookApp] 404 GET /api/kernels/dc6a45c0-2768-48c9-bbec-ff8e340f3320/channels?session_id=b7dbf7410a154f7d944fea051d6d4e17 (::1): Kernel does not exist: dc6a45c0-2768-48c9-bbec-ff8e340f3320
[W 18:01:34.351 NotebookApp] 404 GET /api/kernels/dc6a45c0-2768-48c9-bbec-ff8e340f3320/channels?session_id=b7dbf7410a154f7d944fea051d6d4e17 (::1) 5.120000ms referer=None
[W 18:01:42.372 NotebookApp] 404 GET /api/kernels/dc6a45c0-2768-48c9-bbec-ff8e340f3320/channels?session_id=b7dbf7410a154f7d944fea051d6d4e17 (::1): Kernel does not exist: dc6a45c0-2768-48c9-bbec-ff8e340f3320
[W 18:01:42.376 NotebookApp] 404 GET /api/kernels/dc6a45c0-2768-48c9-bbec-ff8e340f3320/channels?session_id=b7dbf7410a154f7d944fea051d6d4e17 (::1) 7.250000ms referer=None
I also see a popup in the browser:
Connection failed
A connection to the notebook server could not be established. The notebook will continue trying to reconnect. Check your network connection or notebook server configuration.
Desktop (please complete the following information):
- OS: macOS 12.4 (Apple M1 Pro, arm64)
- Browser: Chrome 101.0.4951.64 (Official Build) (arm64)
- Version: 6.4.11
Additional context
All Jupyter libraries are built from source using the Spack package manager. Here are all versions:
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
gnuconfig@2021-08-14
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
Let me know if there are any additional steps I can take to debug this or at least figure out which of these many dependencies has an issue. Also let me know if any of these dependencies should be updated.
@adamjstewart I see the same thing with my installation from July 12 :(
$ jupyter lab
[I 2022-08-02 14:53:16.144 ServerApp] jupyterlab | extension was successfully linked.
[I 2022-08-02 14:53:16.343 ServerApp] nbclassic | extension was successfully linked.
[I 2022-08-02 14:53:16.389 ServerApp] nbclassic | extension was successfully loaded.
[I 2022-08-02 14:53:16.390 LabApp] JupyterLab extension loaded from /opt/spack/var/spack/environments/jupyter/.spack-env/view/lib/python3.9/site-packages/jupyterlab
[I 2022-08-02 14:53:16.390 LabApp] JupyterLab application directory is /opt/spack/var/spack/environments/jupyter/.spack-env/._view/mn5r4lmlhzh73uxfmxvc5yll7vp6y5bj/share/jupyter/lab
[I 2022-08-02 14:53:16.391 ServerApp] jupyterlab | extension was successfully loaded.
[I 2022-08-02 14:53:16.392 ServerApp] Serving notebooks from local directory: /Users/s3j
[I 2022-08-02 14:53:16.392 ServerApp] Jupyter Server 1.13.5 is running at:
[I 2022-08-02 14:53:16.392 ServerApp] http://localhost:8888/lab?token=68062155432e96bf953ea2204bc69e4d37fdad70ba66b6b8
[I 2022-08-02 14:53:16.392 ServerApp] or http://127.0.0.1:8888/lab?token=68062155432e96bf953ea2204bc69e4d37fdad70ba66b6b8
[I 2022-08-02 14:53:16.392 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2022-08-02 14:53:16.394 ServerApp]
To access the server, open this file in a browser:
file:///Users/s3j/Library/Jupyter/runtime/jpserver-48643-open.html
Or copy and paste one of these URLs:
http://localhost:8888/lab?token=68062155432e96bf953ea2204bc69e4d37fdad70ba66b6b8
or http://127.0.0.1:8888/lab?token=68062155432e96bf953ea2204bc69e4d37fdad70ba66b6b8
[W 2022-08-02 14:53:19.385 LabApp] Could not determine jupyterlab build status without nodejs
[I 2022-08-02 14:53:38.177 ServerApp] Kernel started: f819880c-5002-4c9d-b3f0-7ae89ac024a9
[I 2022-08-02 14:53:41.166 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (1/5), new random ports
[I 2022-08-02 14:53:44.196 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (2/5), new random ports
[I 2022-08-02 14:53:47.219 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (3/5), new random ports
[I 2022-08-02 14:53:50.240 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (4/5), new random ports
[I 2022-08-02 14:53:53.265 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (5/5), new random ports
[W 2022-08-02 14:53:56.287 ServerApp] AsyncIOLoopKernelRestarter: restart failed
[W 2022-08-02 14:53:56.288 ServerApp] Kernel f819880c-5002-4c9d-b3f0-7ae89ac024a9 died, removing from map.
Given the cross-post to jupyter_client I'm assuming this is still an issue. These symptoms are indicative of the kernel process starting just enough to satisfy the start kernel request, but then that process terminates, at which time the auto-restart behavior is triggered, attempting the additional restart attempts.
I would recommend adding --debug when starting the notebook server and seeing if that produces any additional (and useful) information.
You might also try running the notebook via an application like papermill (e.g., papermill my-nb.ipynb my-nb-output.ipynb) to better isolate the issue since that would remove the web portion of things and may provide additional datapoints via its error reporting. You will not see the 5 restart attempts with this approach.
Thanks for the tip! Debug didn't reveal anything (at least to my naïve eyes), but here is the log: debug.txt
Will try papermill now.
Output from papermill:
$ papermill Untitled.ipynb Untitled-out.ipynb
Input Notebook: Untitled.ipynb
Output Notebook: Untitled-out.ipynb
Executing: 0%| | 0/1 [00:00<?, ?cell/s]Error occurred while starting new kernel client for kernel 3b09e6e4-0b86-4186-b645-f3416958cfaf: Kernel died before replying to kernel_info
Executing: 0%| | 0/1 [00:01<?, ?cell/s]
Traceback (most recent call last):
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-jpp2hrr5u57wgt2xqwv4tx62uafpsgxs/bin/papermill", line 8, in <module>
sys.exit(papermill())
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-click-8.1.3-mgslf22ooaphw5p6f4fzmpf2yck6v576/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-click-8.1.3-mgslf22ooaphw5p6f4fzmpf2yck6v576/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-click-8.1.3-mgslf22ooaphw5p6f4fzmpf2yck6v576/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-click-8.1.3-mgslf22ooaphw5p6f4fzmpf2yck6v576/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-click-8.1.3-mgslf22ooaphw5p6f4fzmpf2yck6v576/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-jpp2hrr5u57wgt2xqwv4tx62uafpsgxs/lib/python3.10/site-packages/papermill/cli.py", line 250, in papermill
execute_notebook(
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-jpp2hrr5u57wgt2xqwv4tx62uafpsgxs/lib/python3.10/site-packages/papermill/execute.py", line 113, in execute_notebook
nb = papermill_engines.execute_notebook_with_engine(
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-jpp2hrr5u57wgt2xqwv4tx62uafpsgxs/lib/python3.10/site-packages/papermill/engines.py", line 49, in execute_notebook_with_engine
return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-jpp2hrr5u57wgt2xqwv4tx62uafpsgxs/lib/python3.10/site-packages/papermill/engines.py", line 367, in execute_notebook
cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-jpp2hrr5u57wgt2xqwv4tx62uafpsgxs/lib/python3.10/site-packages/papermill/engines.py", line 436, in execute_managed_notebook
return PapermillNotebookClient(nb_man, **final_kwargs).execute()
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-jpp2hrr5u57wgt2xqwv4tx62uafpsgxs/lib/python3.10/site-packages/papermill/clientwrap.py", line 43, in execute
with self.setup_kernel(**kwargs):
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/python-3.10.10-n24ngvr7wkodkzdrghql5gtqbwk3cv76/lib/python3.10/contextlib.py", line 135, in __enter__
return next(self.gen)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-nbclient-0.7.2-iq72qbinrhcmmpuisb2o6mi2xdq3wr4g/lib/python3.10/site-packages/nbclient/client.py", line 598, in setup_kernel
self.start_new_kernel_client()
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-jupyter-core-5.1.0-zl3znlsenizlfwmu5xcexw6pig5kicpm/lib/python3.10/site-packages/jupyter_core/utils/__init__.py", line 160, in wrapped
return loop.run_until_complete(inner)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/python-3.10.10-n24ngvr7wkodkzdrghql5gtqbwk3cv76/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-nbclient-0.7.2-iq72qbinrhcmmpuisb2o6mi2xdq3wr4g/lib/python3.10/site-packages/nbclient/client.py", line 559, in async_start_new_kernel_client
await ensure_async(
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-jupyter-core-5.1.0-zl3znlsenizlfwmu5xcexw6pig5kicpm/lib/python3.10/site-packages/jupyter_core/utils/__init__.py", line 172, in ensure_async
result = await obj
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-jupyter-client-8.1.0-tqn5lvwmp7uy3rvekrvrliuzzavgdyq5/lib/python3.10/site-packages/jupyter_client/client.py", line 199, in _async_wait_for_ready
raise RuntimeError(msg)
RuntimeError: Kernel died before replying to kernel_info
Notebooks (renamed with .txt suffix to allow upload):
Thanks for the additional information and for trying out papermill. You're right, this didn't reveal a whole lot. On the bright side, both scenarios are producing identical results. The initial "handshake" message of kernel-info is not completing, and I suspect the issue is likely on the ipykernel/IPython side of things. All we can say is that the kernel starts and dies before it can respond to the kernel-info request (which we don't know it even received).
I don't know spack (or python package managers in general for that matter) but am wondering if you're able to set up a conda or pip env and eliminate spack from the equation. Although there were M1-related issues when it came about initially, I haven't heard of M1/arm64 continuing to be problematic and, therefore, a tad suspicious of spack (but mostly out of my naïveté).
Completely reasonable, Spack has a couple differences from other package managers.
First, like Conda, each package has to be updated manually, so it could be that one of the packages that had M1 issues hasn't been updated. If you know off the top of your head which packages had issues that would help a lot. Second, like Nix and Brew, Spack installs each package to its own separate installation prefix and (optionally) later symlinks them to a common shared environment. I believe we had issues with Jupyter plugins in the past that assumed everything was in the same directory, but I think we solved those via env vars.
@rchoudhary uses pip and assures me that pip-installed Jupyter works for him, so I suspect you're right that this issue is Spack-specific and a newer version of a package may help. Just not sure which packages to try to update.
Looking at the ipykernel dependencies, they seem to be a good summary of what packages are involved in kernel communications. These would also apply to the notebook server (should it be determined that issues are actually occurring in pre-kernel communication).
Updated ipykernel and all of its direct dependencies to the latest and greatest versions but I'm still seeing the same issue. Will try recursively updating the rest of papermill's dependencies and see if it works.
Ok. I don't expect the papermill-dependency exercise to yield anything since they are not common to those dependencies in the notebook/server code other than those same dependencies in ipykernel. That said, a datapoint is a datapoint.
It seems like trimming down the kernel side of things would be most informative, but I'm not familiar with those kinds of applications. You might try running ipython directly and make sure statements that reside in your notebook can be issued there. Assuming that works, then ipykernel introduces the ZMQ/sockets side of things and that's where there are a few moving parts. I think there's a QtConsole application that essentially interacts with a kernel directly - so that might be an option as well.
Yep, tried upgrading all dependencies (including zmq), still no luck.
ipython runs find, I can print "hello world" (that's all my notebook contains).
Thanks @adamjstewart. This (IMHO) implies the issue is probably at the sockets layer - which has some merit given that's a low-level area of the stack. I'm pinging @minrk in case he has some ideas.
I think one more thing worth trying is to enable debug logging on ipykernel itself. To do this, you'll need to extend the kernelspec file. To locate the kernelspec file run jupyter kernelspec list and identify your kernel name along with its location. In the specified the location will be a kernel.json file. Edit the file to include the --debug option. The contents should resemble this:
{
"argv": [
"python",
"-m",
"ipykernel_launcher",
"--debug",
"-f",
"{connection_file}"
],
"display_name": "Python 3 (ipykernel)",
"language": "python",
"metadata": {
"debugger": true
}
}
You do not need to restart the notebook server but will need to open the notebook so that a new kernel is launched. It should dump quite a lot of information into the console. I've included the output up to the reception of the kernel_info_request message.
ipykernel debug output...
[IPKernelApp] Using existing profile dir: '/Users/kbates/.ipython/profile_default'
[IPKernelApp] Searching path ['/Users/kbates/.ipython/profile_default', '/opt/miniconda3/envs/server-dev/etc/ipython', '/usr/local/etc/ipython', '/etc/ipython'] for config files
[IPKernelApp] Attempting to load config file: ipython_config.py
[IPKernelApp] Looking for ipython_config in /etc/ipython
[IPKernelApp] Looking for ipython_config in /usr/local/etc/ipython
[IPKernelApp] Looking for ipython_config in /opt/miniconda3/envs/server-dev/etc/ipython
[IPKernelApp] Looking for ipython_config in /Users/kbates/.ipython/profile_default
[IPKernelApp] Attempting to load config file: ipython_kernel_config.py
[IPKernelApp] Looking for ipython_kernel_config in /etc/ipython
[IPKernelApp] Looking for ipython_kernel_config in /usr/local/etc/ipython
[IPKernelApp] Looking for ipython_kernel_config in /opt/miniconda3/envs/server-dev/etc/ipython
[IPKernelApp] Looking for ipython_kernel_config in /Users/kbates/.ipython/profile_default
[IPKernelApp] Loading connection file /Users/kbates/Library/Application Support/Jupyter/runtime/kernel-a12ed905-3247-47ca-ad24-8b2e853bfbff.json
[IPKernelApp] Starting the kernel at pid: 72289
[IPKernelApp] shell ROUTER Channel on port: 49353
[IPKernelApp] stdin ROUTER Channel on port: 49355
[IPKernelApp] control ROUTER Channel on port: 49357
[IPKernelApp] iopub PUB Channel on port: 49354
[IPKernelApp] Heartbeat REP Channel on port: 49356
[IPKernelApp] Writing connection file: /Users/kbates/Library/Application Support/Jupyter/runtime/kernel-a12ed905-3247-47ca-ad24-8b2e853bfbff.json
[IPKernelApp] To connect another client to this kernel, use:
[IPKernelApp] --existing kernel-a12ed905-3247-47ca-ad24-8b2e853bfbff.json
[IPKernelApp] Seeing logger to stderr, rerouting to raw filedescriptor.
[IPKernelApp] Loading IPython extensions...
[IPKernelApp] Loading IPython extension: storemagic
[IPKernelApp]
*** MESSAGE TYPE:kernel_info_request***
[IPKernelApp] Content: {}
--->
[IPKernelApp] kernel_info_request: {'header': {'msg_id': '11653d02-4758-44b9-9163-89dbc0e35231_72270_0', 'msg_type': 'kernel_info_request', 'username': 'kbates', 'session': '11653d02-4758-44b9-9163-89dbc0e35231', 'date': datetime.datetime(2023, 3, 22, 22, 29, 32, 509676, tzinfo=tzutc()), 'version': '5.3'}, 'msg_id': '11653d02-4758-44b9-9163-89dbc0e35231_72270_0', 'msg_type': 'kernel_info_request', 'parent_header': {}, 'metadata': {}, 'content': {}, 'buffers': []}
This may or may not explain things, but I don't have a kernel.json file:
$ jupyter kernelspec list
Available kernels:
python3 /Users/Adam/spack/var/spack/environments/system/.spack-env/._view/7carnllnc4vuxozpkgg34llszt3okbr7/lib/python3.10/site-packages/ipykernel/resources
$ ls /Users/Adam/spack/var/spack/environments/system/.spack-env/._view/7carnllnc4vuxozpkgg34llszt3okbr7/lib/python3.10/site-packages/ipykernel/resources
logo-32x32.png logo-64x64.png logo-svg.svg
It's strange that jupyter kernelspec list is listing the resources directory here. I don't know what causes that and the path looks like it's the package installation area for ipykernel - not the location where an ipykernel instance (i.e., specification) would reside!
Is there a kernel.json file in the parent directory (/Users/Adam/spack/var/spack/environments/system/.spack-env/._view/7carnllnc4vuxozpkgg34llszt3okbr7/lib/python3.10/site-packages/ipykernel)? If so, continue the exercise to enable debug there.
Can you list what is in the parent directory regardless? I suspect this is indeed the package installation location.
Also, what happens when you run python -m ipykernel install --name my-test-kernel? If after doing so my-test-kernel shows up as expected in jupyter kernelspec list (likely installed in /usr/local/share/jupyter/kernels) could you try pointing your notebook at that kernel?
Is there a
kernel.jsonfile in the parent directory
Nope.
Can you list what is in the parent directory regardless?
__init__.py _version.py control.py embed.py inprocess kernelapp.py parentpoller.py resources zmqshell.py
__main__.py comm datapub.py eventloops.py iostream.py kernelbase.py pickleutil.py serialize.py
__pycache__ compiler.py debugger.py gui ipkernel.py kernelspec.py py.typed tests
_eventloop_macos.py connect.py displayhook.py heartbeat.py jsonutil.py log.py pylab trio_runner.py
Also, what happens when you run
python -m ipykernel install --name my-test-kernel?
[Errno 13] Permission denied: '/usr/local/share'
Fun times on macOS with SIP enabled. If I sudo mkdir, chown, and run again, it successfully installs a kernel:
Installed kernelspec my-test-kernel in /usr/local/share/jupyter/kernels/my-test-kernel
It does show up now. I added the debug flag to this kernel and ran papermill again:
$ papermill -k my-test-kernel Untitled.ipynb Untitled-out.ipynb
Input Notebook: Untitled.ipynb
Output Notebook: Untitled-out.ipynb
Executing: 0%| | 0/1 [00:00<?, ?cell/s][IPKernelApp] Using existing profile dir: '/Users/Adam/.ipython/profile_default'
[IPKernelApp] Searching path ['/Users/Adam/.ipython/profile_default', '/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/python-3.10.10-yadxcbmd2o7afi3pto7do5fd5aqjl6oo/etc/ipython', '/usr/local/etc/ipython', '/etc/ipython'] for config files
[IPKernelApp] Attempting to load config file: ipython_config.py
[IPKernelApp] Looking for ipython_config in /etc/ipython
[IPKernelApp] Looking for ipython_config in /usr/local/etc/ipython
[IPKernelApp] Looking for ipython_config in /Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/python-3.10.10-yadxcbmd2o7afi3pto7do5fd5aqjl6oo/etc/ipython
[IPKernelApp] Looking for ipython_config in /Users/Adam/.ipython/profile_default
[IPKernelApp] Attempting to load config file: ipython_kernel_config.py
[IPKernelApp] Looking for ipython_kernel_config in /etc/ipython
[IPKernelApp] Looking for ipython_kernel_config in /usr/local/etc/ipython
[IPKernelApp] Looking for ipython_kernel_config in /Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/python-3.10.10-yadxcbmd2o7afi3pto7do5fd5aqjl6oo/etc/ipython
[IPKernelApp] Looking for ipython_kernel_config in /Users/Adam/.ipython/profile_default
[IPKernelApp] Loading connection file /private/var/folders/jv/cgkfvslj6nq1l7cw0c8c_8gm0000gn/T/tmpw2zay9ml.json
[IPKernelApp] Starting the kernel at pid: 9626
[IPKernelApp] shell ROUTER Channel on port: 64546
[IPKernelApp] stdin ROUTER Channel on port: 64548
[IPKernelApp] control ROUTER Channel on port: 64550
[IPKernelApp] iopub PUB Channel on port: 64547
[IPKernelApp] Heartbeat REP Channel on port: 64549
[IPKernelApp] Writing connection file: /private/var/folders/jv/cgkfvslj6nq1l7cw0c8c_8gm0000gn/T/tmpw2zay9ml.json
[IPKernelApp] To connect another client to this kernel, use:
[IPKernelApp] --existing /private/var/folders/jv/cgkfvslj6nq1l7cw0c8c_8gm0000gn/T/tmpw2zay9ml.json
[IPKernelApp] Seeing logger to stderr, rerouting to raw filedescriptor.
Error occurred while starting new kernel client for kernel fd44057d-34db-421f-9baa-4f602f210dba: Kernel died before replying to kernel_info
Executing: 0%| | 0/1 [00:01<?, ?cell/s]
Traceback (most recent call last):
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-t7e6joq5jbsx4jtvtctq6d7zcohqyben/bin/papermill", line 8, in <module>
sys.exit(papermill())
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-click-8.1.3-phvm6ngokygjyb77gs5jxr3ueeyjt2jl/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-click-8.1.3-phvm6ngokygjyb77gs5jxr3ueeyjt2jl/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-click-8.1.3-phvm6ngokygjyb77gs5jxr3ueeyjt2jl/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-click-8.1.3-phvm6ngokygjyb77gs5jxr3ueeyjt2jl/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-click-8.1.3-phvm6ngokygjyb77gs5jxr3ueeyjt2jl/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-t7e6joq5jbsx4jtvtctq6d7zcohqyben/lib/python3.10/site-packages/papermill/cli.py", line 250, in papermill
execute_notebook(
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-t7e6joq5jbsx4jtvtctq6d7zcohqyben/lib/python3.10/site-packages/papermill/execute.py", line 113, in execute_notebook
nb = papermill_engines.execute_notebook_with_engine(
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-t7e6joq5jbsx4jtvtctq6d7zcohqyben/lib/python3.10/site-packages/papermill/engines.py", line 49, in execute_notebook_with_engine
return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-t7e6joq5jbsx4jtvtctq6d7zcohqyben/lib/python3.10/site-packages/papermill/engines.py", line 367, in execute_notebook
cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-t7e6joq5jbsx4jtvtctq6d7zcohqyben/lib/python3.10/site-packages/papermill/engines.py", line 436, in execute_managed_notebook
return PapermillNotebookClient(nb_man, **final_kwargs).execute()
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-t7e6joq5jbsx4jtvtctq6d7zcohqyben/lib/python3.10/site-packages/papermill/clientwrap.py", line 43, in execute
with self.setup_kernel(**kwargs):
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/python-3.10.10-yadxcbmd2o7afi3pto7do5fd5aqjl6oo/lib/python3.10/contextlib.py", line 135, in __enter__
return next(self.gen)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-nbclient-0.7.2-nqxvgunmul2o32f53yaoowwkpu2ej3ia/lib/python3.10/site-packages/nbclient/client.py", line 598, in setup_kernel
self.start_new_kernel_client()
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-jupyter-core-5.3.0-edicz5adwjeqyakfhzbgxvyokby4t4o3/lib/python3.10/site-packages/jupyter_core/utils/__init__.py", line 166, in wrapped
return loop.run_until_complete(inner)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/python-3.10.10-yadxcbmd2o7afi3pto7do5fd5aqjl6oo/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-nbclient-0.7.2-nqxvgunmul2o32f53yaoowwkpu2ej3ia/lib/python3.10/site-packages/nbclient/client.py", line 559, in async_start_new_kernel_client
await ensure_async(
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-jupyter-core-5.3.0-edicz5adwjeqyakfhzbgxvyokby4t4o3/lib/python3.10/site-packages/jupyter_core/utils/__init__.py", line 182, in ensure_async
result = await obj
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-jupyter-client-8.1.0-eh43c6atq5rxccxjlcgc4latl6xdvelz/lib/python3.10/site-packages/jupyter_client/client.py", line 199, in _async_wait_for_ready
raise RuntimeError(msg)
RuntimeError: Kernel died before replying to kernel_info
See anything useful there?
Thanks for the response (and your patience)! This is helpful, although I still cannot determine the cause of the kernel's abrupt termination.
I believe this issue should be investigated from the ipykernel side of things, but given this appears to only occur in a Spack env, it probably means you will need to troubleshoot further.
Based on the ipykernel debug output, we can determine that, in all likelihood, the kernel has completed the init_io() method since the "Seeing logger to stderr" message is logged there in what appears to be general behavior (not an error condition). The init_io() method is called from initialize() and following initialize() is the start() method.
Barring other ideas, I think it would be interesting to instrument additional logging in initialize() and start() and see where that takes you. (You should probably confirm the issue still occurs using a custom-built ipykernel prior to making any changes.)
@blink1073 - does it make sense to transfer this issue to ipykernel for now? I suppose it could be an issue in jupyter_client but it seems like it should be looked at from ipykernel at the moment. I think the related issue in jupyter_client can be closed since we would likely transfer this issue there if it's determined to be a jupyter_client issue.
Also, do you happen to have any ideas here?
I'd suggest trying to install juptyer_client<8 to rule out any changes from that. After that, something akin to a git bisect on the version of ipykernel might be useful. I don't have the bandwidth to investigate further.
I'd suggest trying to install
juptyer_client<8to rule out any changes from that.
Same issue.
After that, something akin to a git bisect on the version of
ipykernelmight be useful.
Papermill is sufficient to reproduce the issue, and it doesn't depend on ipykernel in any way, so it can't be the cause of the problem.
Papermill is sufficient to reproduce the issue, and it doesn't depend on ipykernel in any way, so it can't be the cause of the problem.
How did you come to the conclusion that ipykernel is not the issue? It is that process that is not starting completely enough to respond to kernel_info_request messages. Papermill (via nbclient) launches a kernel (i.e., ipykernel process) that interacts with that process to execute all the cells in the given notebook. It (nbclient via papermill) essentially feeds the notebook cell's code to the kernel via messaging.
I suspect the problem is environmental in nature and it's probably most direct to continue using debug logging to determine where the ipykernel process is (silently!) falling on its face.
Papermill (via nbclient) launches a kernel (i.e., ipykernel process) that interacts with that process to execute all the cells in the given notebook.
Papermill depends on nbclient, but nbclient doesn't depend on ipykernel: https://github.com/jupyter/nbclient/blob/main/pyproject.toml
I do happen to have ipykernel installed (6.16.0), but if it's required to use nbclient, then the dependency is missing from its pyproject.toml.
ipykernel is not a direct dependency in the Python sense. nbclient doesn't call ipykernel methods directly. It issues messages to configured ZMQ sockets (that the two processes know about via the shared connection information). This decoupling enables the ability to have different kernel implementations for various languages, etc., because this communication goes over "generic" sockets.
However, in this case, nbclient is not able to start issuing cell contents as messages because the initial handshaking that occurs during a kernel's startup has not been completed successfully.
This seems to be working now with the latest versions of all packages!
i have the same issue running on a fresh mamba env with python 3.11.4
jupyter 1.0.0 py311h38be061_8 conda-forge
jupyter-lsp 2.2.0 pyhd8ed1ab_0 conda-forge
jupyter_client 8.3.0 pyhd8ed1ab_0 conda-forge
jupyter_console 6.6.3 pyhd8ed1ab_0 conda-forge
jupyter_core 5.3.1 py311h38be061_0 conda-forge
jupyter_events 0.7.0 pyhd8ed1ab_2 conda-forge
jupyter_server 2.7.0 pyhd8ed1ab_0 conda-forge
jupyter_server_terminals 0.4.4 pyhd8ed1ab_1 conda-forge
jupyterlab 4.0.4 pyhd8ed1ab_0 conda-forge
jupyterlab_pygments 0.2.2 pyhd8ed1ab_0 conda-forge
jupyterlab_server 2.24.0 pyhd8ed1ab_0 conda-forge
jupyterlab_widgets 3.0.8 pyhd8ed1ab_0 conda-forge
ipyevents 2.0.1 pyhd8ed1ab_0 conda-forge
ipykernel 6.25.1 pyh71e2992_0 conda-forge
ipython 8.14.0 pyh41d4057_0 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
i have the exact same behaviour, with 5 restarts then kernel is said to have died. can't even execute a cell in a jupyter notebook :(
Similar issue here. I get the line "StdErr from Kernel Process [IPKernelApp] Seeing logger to stderr, rerouting to raw filedescriptor"
And jupyter notebook cells hang forever
