Wrong kernel status for async calls

[trungleduc]

Description

On JupyterLite the kernel status is not reported correctly when calling an async function using the xeus-python kernel. It works well with pyodide kernel

https://github.com/user-attachments/assets/8a09b7b6-03fe-4f6c-a2af-bd8e214e8f52

Reproduce

import asyncio
async def wait():
    print('start')
    await asyncio.sleep(5)
    print('end')

await wait()

Expected behavior

The kernel is busy for 5 seconds with the above snippet.

Context

• JupyterLite version:
• Operating System and version:
• Browser and version:

Browser Output

Paste the output from your browser Javascript console here.

[DerThorsten]

The reason for this behaviour is the following. In xeus(-python), when we see an awaited toplevel promise, we await that promise here. So that is, when we already recive the message for the next cell, we block the execution of the next cell via that await.

~Afaik this might be fixed on a ipython / pure python level.~ I think we can fix this directely in xeus-python. There is a callback we need to call, once the cell is fully executed, but we call it to early.

To fix this, we need to move the code which generated the async task for the cell to the c++ side, and

async def _helper_function_for_cell(...)

        # the cells code is run here, this contains awaits.
        # [...]
        await something(...)
        # [...]
       
        # once the code for the cell is done we need to call the xeus "done" callback here
        xeus_done_callback(...)

Currently, the logic to generate thism async helper task is somehere in the ipython odebase. Maybe @martinRenou knows where

[DerThorsten]

For some context / proof of concept. In the xeus-javascript side we already do what I suggested above. There we call the "done" calback from the js side . With that, the kernel is reporting the correct "busy" state for async cells