Turning an app with embedded Python into a Jupyter kernel

[s-m-e]

I am doing a few experiments on apps with Python integration via embedded Python, i.e. QGIS (and FreeCAD). The objective is to turn them into Jupyter kernels. Both apps come with their own Python console, but I'd like to run their GUI and Jupyter lab side-by-side while Jupyter's kernel is actually the embedded Python interpreter of the GUI app. I essentially want to use Jupyter for interacting with the apps instead of the apps' own consoles.

I started with QGIS (and on Windows, because I was curious ...). For "implementation details" see below.

QGIS launches, but from Jupyter's perspective, the kernel keeps starting. It never "finishes" starting. Interestingly, I can actually re-start the kernel, i.e. QGIS, from within Jupyter just fine. Either way, code can not be executed.

• Completely ignoring my "implementation": Is what I described even possible?
• I am trying to make sense of ipykernel (and ipython for that matter), but it is not trivial to get started. Does my "implementation" make any sense or do I have to approach things differently altogether?
• Alternatively, could I turn an already running process (pure Python or with embedded Python) into a "kernel" by attaching to it from some kind of a wrapper process (via some form of IPC) which is the actual kernel from Jupyter's perspective?

---

This is what I have so far:

kernel.json, which injects code at startup via PYQGIS_STARTUP:

{
 "argv": [
  "C:/Users/demo/mambaforge/envs/cluster/Library/bin\\qgis.exe",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "QGIS",
 "language": "python",
 "env": {
  "PYQGIS_STARTUP": "C:/Users/demo/mambaforge/envs/cluster/share/jupyter/kernels/qgis/launch.py"
 }
}

launch.py which is supposed to launch the kernelapp. argv is a bit of an issue because QGIS does not expose it via sys, hence the ugly hack. I think it should forward the args to the right place in traitlets:

from threading import Thread
import os
from time import time
import sys

from qgis.core import QgsApplication

from ipykernel import kernelapp as app

LOG_FN = 'C:/Users/demo/mambaforge/envs/cluster/share/jupyter/kernels/qgis/log.out'

def log_out(msg):
    with open(LOG_FN, mode = 'a') as f:
        f.write('%d | %s\n' % (round(time()), msg))

def launch_ipython():
    log_out('Argv...')
    argv = QgsApplication.arguments().copy() # HACK: sys.argv not available
    log_out(str(argv))
    log_out('App...')
    app.launch_new_instance(argv = argv) # Blocks ... ?
    log_out('Done?')

sys._ipython = Thread(target = launch_ipython) # HACK for later access
sys._ipython.start()

log.out from a single kernel start. Looks like two instances, threads or processes are getting started:

1621087681 | Argv...
1621087681 | ['C:/Users/demo/mambaforge/envs/cluster/Library/bin\\qgis.exe', '-m', 'ipykernel_launcher', '-f', 'C:\\Users\\demo\\AppData\\Roaming\\jupyter\\runtime\\kernel-b5e49e78-f742-49c1-b75d-6425c4fbee6a.json']
1621087681 | App...
1621087681 | Argv...
1621087681 | ['C:/Users/demo/mambaforge/envs/cluster/Library/bin\\qgis.exe', '-m', 'ipykernel_launcher', '-f', 'C:\\Users\\demo\\AppData\\Roaming\\jupyter\\runtime\\kernel-b5e49e78-f742-49c1-b75d-6425c4fbee6a.json']
1621087681 | App...

[SylvainCorlay]

Hey @s-m-e this we are definitely interested in this application (embedding a Jupyter kernel into a desktop application).

It turns out we have done it already for Slicer3D (a medical imaging Qt desktop app developed by Kitware). You may be interested in the following article, which dives into this example: https://blog.jupyter.org/slicerjupyter-a-3d-slicer-kernel-for-interactive-publications-6f2ad829f635. Actually, FreeCAD and Blender are mentioned in the post as possible applications that could benefit from this approach.

Long story short, the approach is to use xeus-python.

• Xeus-python is a python kernel based on xeus (a C++ implementation of the kernel protocol).
• Xeus-python relies on IPython (just like ipykernel) so that it supports all IPython magics, rich display mechanism, widgets etc.

however, xeus-python differs from ipykernel in that it has a pluggable concurrency model. In the case of Slicer, which is a Qt application, we override that concurrency model to make use of the Qt event loop, so that polling the kernel sockets does not block the application (and reversely).

[s-m-e]

It turns out we have done it already for Slicer3D

Thanks a lot for the pointer.

we override that concurrency model to make use of the Qt event loop

I recently came across this concept via qasync. If I am not mistaken, ipython supports a similar concept.

[SylvainCorlay]

If I am not mistaken, ipython supports a similar concept.

That is meant to run the GUI event loop of matplotlib backends.