Share running kernels between processes

[pplonski]

Is it possible to run a kernel in one process and have access to it in the second process?

Example code:

first process (file start.py):

import time
from jupyter_client import MultiKernelManager 
mkm = MultiKernelManager()
mkm.start_kernel(kernel_name="python3", kernel_id="some-kernel-id")
time.sleep(1000)

second process (file getkernel.py):

from jupyter_client import MultiKernelManager 
mkm = MultiKernelManager()
km = mkm.get_kernel("some-kernel-id")

Is it possible to run processes in the terminal:

python start.py &
sleep 3
python getkernel.py

and have access to kernel some-kernel-id in getkernel.py process?

[davidbrochart]

It's not possible like that, because both kernel managers don't share their state. You can see it if you list the kernels in the second process:

print(mkm.list_kernel_ids())
# shows []

You would need to connect to the same kernel using a new set of ZMQ sockets. This means that if you start the kernel in the first process, the second process should access the connection file that was created in the first process.

[pplonski]

@davidbrochart thank you for your reply!

How to get information about the connection file? Is there any get_connection_file() method in KernelManager?

from jupyter_client import MultiKernelManager 
mkm = MultiKernelManager()
km = mkm.get_kernel("some-kernel-id")

# how to get connection file information?
connection_file = km.get_connection_file()

[davidbrochart]

You should get the connection file from the process that started the kernel:

from jupyter_client import MultiKernelManager 
mkm = MultiKernelManager()
mkm.start_kernel(kernel_name="python3", kernel_id="some-kernel-id")
km = mkm.get_kernel("some-kernel-id")
km.connection_file
# 'kernel-some-kernel-id.json'

The connection file was written in the current directory with the name kernel-some-kernel-id.json.

[kevin-bates]

~The connection file is not always written locally so cannot be assumed to be present if you want to support kernel provisioners other than the default LocalProvisioner.~ As a result, I'd recommend km.get_connection_info(), which essentially provides the content of what would be in a local connection file.

~The LocalProvisioner does write out the connection info into a file located in the JUPYTER_RUNTIME_DIR~, so if you know the kernel_id and you only need to support locally-provisioned kernels, then you could search the runtime directory matching on the kernel_id to load the kernel's connection info into another process.

EDIT: Actually, I just realized that the KernelManager writes out the connection information it receives from the provisioner, so this subject is actually provisioner-agnostic - which is great.

I also realized that David's comment about the file's location is correct, although I think that might be a bug. If we don't write the connection file to well-known location, then its nearly impossible for other applications to use the same information. (Perhaps I'm missing something here as well.)

[pplonski]

@davidbrochart @kevin-bates thank you for responses!

I run David's code and the connection file is not created. I've run jupyter --runtime-dir to check the path for kernel jsons. But there is no file kernel-some-kernel-id.json in the path.

I've created two files process1.py and process2.py. And tried to manually save the connection file and load in the second process. I don't know how to get KernelManager from KernelClient and how to check if connection is made. (I need KernelManager to execute the notebook).

# process1.py
import time
import json
from jupyter_client import MultiKernelManager

mkm = MultiKernelManager()
mkm.start_kernel(kernel_name="python3", kernel_id="some-kernel-id")
km = mkm.get_kernel("some-kernel-id") # the kernel connection file is not created
print(km.connection_file)
print(km.get_connection_info())

# manually save connection info
with open("my-kernel.json", "w") as fout:
    info = km.get_connection_info()
    info["key"] = info["key"].decode("utf-8")
    print(info)
    fout.write(json.dumps(info))
time.sleep(60) # sleep so second process can connect

# process2.py
import json
from jupyter_client import KernelClient

info = json.load(open("my-kernel.json"))

kc = KernelClient()
kc.load_connection_info(info)
print(kc)
# how to check if KernelClient is connected?
# hot to get access to KernelManager?
print(kc.history()) # it throws error

[davidbrochart]

In the second process you need either a blocking or an asynchronous kernel client. I don't think you need a kernel manager in the second process as the kernel is already managed in the first process.

# process2.py
import json
from jupyter_client import BlockingKernelClient

info = json.load(open("kernel-some-kernel-id.json"))

kc = BlockingKernelClient()
kc.load_connection_info(info)
kc.execute("with open('abc', 'w') as f: f.write('def')")
# file abc is created

[pplonski]

@davidbrochart thank you! It works.

# process1.py
import time
import json
from jupyter_client import MultiKernelManager

mkm = MultiKernelManager()
mkm.start_kernel(kernel_name="python3", kernel_id="some-kernel-id")
km = mkm.get_kernel("some-kernel-id")
print(km.connection_file)
print(km.get_connection_info())


with open("my-kernel.json", "w") as fout:
    info = km.get_connection_info()
    info["key"] = info["key"].decode("utf-8")
    print(info)
    fout.write(json.dumps(info))

kc = km.blocking_client()
kc.execute_interactive("a = 13") # define variable `a`

time.sleep(60)

# process2.py
import json
from jupyter_client import BlockingKernelClient

info = json.load(open("my-kernel.json"))

kc = BlockingKernelClient()
kc.load_connection_info(info)
kc.execute_interactive("print(a)") # prints variable `a` from the process1.py

I first run process1.py and then in the second terminal process2.py. The process2.py prints variable defined in the process1.py. Thank you!

Anyway, the kernel connection info is not created automatically. I write it manually. Is it a bug?

[davidbrochart]

@davidbrochart thank you! It works.

:tada:

Anyway, the kernel connection info is not created automatically. I write it manually. Is it a bug?

Could be, it would be great if you could investigate and maybe come up with a PR!

[pplonski]

I would be super happy to help. Any tips where to start investigation? Which files to look? Maybe there are some tests that I can check/create?

[kevin-bates]

Thanks for looking into this and I apologize for my confusing comments.

The provisioner returns the connection information to the kernel manager here where it should be written to the file.

[pplonski]

@kevin-bates @davidbrochart I found that the kernel information is saved but in different directory. I was looking for kernel files in /home/piotr/.local/share/jupyter/runtime (the directory from jupyter --runtime-dir) but the files where in the local directory (the same directory when I start processes).

Thank you very much for your help!!!

[kevin-bates]

@pplonski - for completeness, the "Notebook Server" (i.e., notebook or jupyter_server) will set the connection_dir attribute of the MultiKernelManager which then gets composed with the connection filename when starting the kernel. As a result, kernels launched via Jupyter Notebook/Lab will have their connection files posted into the runtime_dir, while kernels launched via applications like nbclient (which don't use MultiKernelManager) will have their connection files written in the current working directory, unless they set KernelManager.connection_file explicitly.

Since it sounds like you want some coordination to occur between processes, you might want to look at setting connection_file yourself or control where the current working directory is located.

[pplonski]

Thank you @kevin-bates for help!

[pplonski]

I have a few more questions:

1. How long kernel lives? If I start a kernel in process1.py how long it will be available? Is it live only when process1.py is running? Can I somehow list all running kernels?
2. I would like to use the nbconvert Python API to execute the notebook. It accepts the KernelManager in ExecutePreprocessor.preprocess() method. I am able to construct BlockingKernelClient() from the kernel connection file, but how to get a KernelManager?

[davidbrochart]

1. You need to ask your kernel manager to shutdown the kernel, otherwise the kernel continues living even after process1.py has exited. You can get the running kernels from a multi-kernel manager.
2. The kernel manager you created the kernel with lives in process1.py. You would need somehow to pass to process2.py the process ID of the kernel launched in process1.py, and recreate a kernel manager from it. I don't think we have the logic for that, but that would make a good PR, if you feel like it :smile:

[kevin-bates]

The kernel manager you created the kernel with lives in process1.py. You would need somehow to pass to process2.py the process ID of the kernel launched in process1.py, and recreate a kernel manager from it. I don't think we have the logic for that, but that would make a good PR, if you feel like it 😄

One of the things built into provisioners is the ability to get and load information about the provisioner and the connection to the kernel it is provisioning. The idea is that a KernelManager will call get_provisioner_info() after launching the kernel and persist this information somewhere. A different KernelManager could then call load_provisioner_info() using the previously persisted information to "hydrate" the provisioner and enable communication from the "current" KernelManager. There are probably some loose ends to tie together to get "hydration" working but those methods provide the basis.

This is primarily applicable to remote kernels where the node on which jupyter_client was running has gone down, yet the kernel, being remote, is still running, but it could be used for processes locally (and assuming the first process doesn't bring the kernel down with it via signal handling, etc.)

[pplonski]

Thank you @davidbrochart and @kevin-bates for responses!

I've created two files based on your responses:

process1.py

import nbformat
import json
from jupyter_client import MultiKernelManager
from nbconvert.preprocessors import ExecutePreprocessor

mkm = MultiKernelManager()
mkm.start_kernel(kernel_name="python3", kernel_id="some-kernel")
km = mkm.get_kernel("some-kernel") # the kernel connection file is not created

print(km.get_connection_info())

nb1 = nbformat.v4.new_notebook()

nb1["cells"] = [nbformat.v4.new_code_cell("a = 13")]

ep = ExecutePreprocessor()
ep.preprocess(nb1, km=km)

print(nb1.cells)

Output from running process1.py:

{'transport': 'tcp', 'ip': '127.0.0.1', 'shell_port': 53149, 'iopub_port': 48901, 'stdin_port': 39891, 'hb_port': 45391, 'control_port': 41813, 'signature_scheme': 'hmac-sha256', 'key': b'973a1936-730bbc4d4c1f0b6cfeb75828'}
[{'id': '9cec141e', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-07-12T15:07:02.121607Z', 'iopub.execute_input': '2022-07-12T15:07:02.121856Z', 'iopub.status.idle': '2022-07-12T15:07:02.126288Z', 'shell.execute_reply': '2022-07-12T15:07:02.125667Z'}}, 'execution_count': 1, 'source': 'a = 13', 'outputs': []}]

process2.py

import nbformat
import time
import json
from jupyter_client import KernelManager
from nbconvert.preprocessors import ExecutePreprocessor

info = json.load(open("kernel-some-kernel.json"))

print(info)

km = KernelManager()
print(km.get_connection_info())

km.load_connection_info(info)

print(km.get_connection_info())

print("alive", km.is_alive())

nb1 = nbformat.v4.new_notebook()

nb1["cells"] = [nbformat.v4.new_code_cell("print(a)")]

ep = ExecutePreprocessor()
ep.preprocess(nb1, km=km)
print(nb1.cells)

Output from running process2.py after process1.py:

{'shell_port': 53149, 'iopub_port': 48901, 'stdin_port': 39891, 'control_port': 41813, 'hb_port': 45391, 'ip': '127.0.0.1', 'key': '973a1936-730bbc4d4c1f0b6cfeb75828', 'transport': 'tcp', 'signature_scheme': 'hmac-sha256', 'kernel_name': ''}
{'transport': 'tcp', 'ip': '127.0.0.1', 'shell_port': 0, 'iopub_port': 0, 'stdin_port': 0, 'hb_port': 0, 'control_port': 0, 'signature_scheme': 'hmac-sha256', 'key': b'fcc59c2e-28711c789aa7cb6ddba99301'}
{'transport': 'tcp', 'ip': '127.0.0.1', 'shell_port': 53149, 'iopub_port': 48901, 'stdin_port': 39891, 'hb_port': 45391, 'control_port': 41813, 'signature_scheme': 'hmac-sha256', 'key': b'973a1936-730bbc4d4c1f0b6cfeb75828'}
alive False
Traceback (most recent call last):
  File "process2.py", line 25, in <module>
    ep.preprocess(nb1, km=km)
  File "/home/piotr/sandbox/mercury/mercury/menv/lib/python3.8/site-packages/nbconvert/preprocessors/execute.py", line 89, in preprocess
    self.preprocess_cell(cell, resources, index)
  File "/home/piotr/sandbox/mercury/mercury/menv/lib/python3.8/site-packages/nbconvert/preprocessors/execute.py", line 110, in preprocess_cell
    cell = self.execute_cell(cell, index, store_history=True)
  File "/home/piotr/sandbox/mercury/mercury/menv/lib/python3.8/site-packages/nbclient/util.py", line 85, in wrapped
    return just_run(coro(*args, **kwargs))
  File "/home/piotr/sandbox/mercury/mercury/menv/lib/python3.8/site-packages/nbclient/util.py", line 60, in just_run
    return loop.run_until_complete(coro)
  File "/home/piotr/sandbox/mercury/mercury/menv/lib/python3.8/site-packages/nest_asyncio.py", line 81, in run_until_complete
    return f.result()
  File "/usr/lib/python3.8/asyncio/futures.py", line 175, in result
    raise self._exception
  File "/usr/lib/python3.8/asyncio/tasks.py", line 280, in __step
    result = coro.send(None)
  File "/home/piotr/sandbox/mercury/mercury/menv/lib/python3.8/site-packages/nbclient/client.py", line 1025, in async_execute_cell
    await self._check_raise_for_error(cell, cell_index, exec_reply)
  File "/home/piotr/sandbox/mercury/mercury/menv/lib/python3.8/site-packages/nbclient/client.py", line 919, in _check_raise_for_error
    raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)
nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
print(a)
------------------

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-1-bca0e2660b9f> in <module>()
----> 1 print(a)

NameError: name 'a' is not defined
NameError: name 'a' is not defined

Looks like I can load connection info with load_connection_info() method. It shows correct key from connection but somehow the KernelManager is not connected, it returns False for is_alive() method.

From what I've checked the KernelManager derives from ConnectionFileMixin so just setting connection info should make it work. What am I missing?

[davidbrochart]

somehow the KernelManager is not connected, it returns False for is_alive() method.

It is currently considered alive if it launched the kernel.

[pplonski]

@davidbrochart thank you! I've checked and it is impossible to have two KernelManager in two different processes that are pointing to the same kernel. Each KernelManager has a process (controlling running kernel) that is created with subprocess.Popen and it cant be shared between processes.

It might be an option to create some mock Provisioner that will take process ID of running kernel. It might be possible if Provisioner process is only needed to start and kill a kernel process.