"Segmentation fault" when using hot reload

[MarcSkovMadsen]

We are developing a Panel application with many pages served. The application utilizes duckdb as database.

We have now added a new page validation_report.py. When we serve this page with --dev option

panel serve validation_report.py --dev

we see

Traceback (most recent call last):
  File "/home/jovyan/repos/mt-pm-reporting/.venv/bin/panel", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/jovyan/repos/mt-pm-reporting/.venv/lib/python3.11/site-packages/panel/command/__init__.py", line 95, in main
    raise e
  File "/home/jovyan/repos/mt-pm-reporting/.venv/lib/python3.11/site-packages/panel/command/__init__.py", line 92, in main
    ret = parsed_args.invoke(parsed_args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/repos/mt-pm-reporting/.venv/lib/python3.11/site-packages/panel/command/serve.py", line 736, in invoke
    super().invoke(args)
  File "/home/jovyan/repos/mt-pm-reporting/.venv/lib/python3.11/site-packages/bokeh/command/subcommands/serve.py", line 988, in invoke
    server.run_until_shutdown()
  File "/home/jovyan/repos/mt-pm-reporting/.venv/lib/python3.11/site-packages/bokeh/server/server.py", line 203, in run_until_shutdown
    self._loop.start()
  File "/home/jovyan/repos/mt-pm-reporting/.venv/lib/python3.11/site-packages/tornado/platform/asyncio.py", line 205, in start
    self.asyncio_loop.run_forever()
  File "/home/jovyan/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 597, in run_forever
    self._check_running()
  File "/home/jovyan/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 589, in _check_running
    raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running
/home/jovyan/repos/mt-pm-reporting/.venv/lib/python3.11/site-packages/panel/io/server.py:350: RuntimeWarning: coroutine 'Server.stop.<locals>.stop_autoreload' was never awaited
  pass # Ignore if the event loop is still running
ORA-24550: signal received: [si_signo=11] [si_errno=0] [si_code=1] [si_int=0] [si_ptr=(nil)] [si_addr=0x4]
kpedbg_dmp_stack()+396<-kpeDbgCrash()+204<-kpeDbgSignalHandler()+113<-skgesig_sigactionHandler()+258<-__sighandler()<-method_dealloc()+16<-_ZN12_rust_notify10RustNotify5watch17he81fa7bd07045a9aE()+6518
Segmentation fault (core dumped)

Workarounds

The work arounds below don't trigger the problem.

Don't use hot reload

Serving with out --dev flag does not trigger the problem.

Use hot reload but set the event loop

import asyncio
from typing import Any

import pandas as pd
import panel as pn

... REST OF IMPORTS

asyncio.set_event_loop(asyncio.new_event_loop())

[philippjfr]

Can you try defining a --setup script and then do this:

import panel.io.reload

panel.io.reload.IGNORED_MODULES.append('duckdb')

[MarcSkovMadsen]

A colleague made me aware that every time the server fails like above a core.... file is created.

I have noticed these for a long time. But did not realize they are created by failing --dev server. I would like this not to happen. The files are very large and risk ending up in my git commit.

[hoxbro]

Python should never segfault, so if we fix the segfault, the core dump will automatically disappear.

[MarcSkovMadsen]

Minimum Reproducible Example

Create a python package segfaultwith the file

src/segfault/init.py

from adlfs import AzureBlobFileSystem
import pandas as pd

CONNECTION_STRING = 'DefaultEndpointsProtocol=https;AccountName=...core.windows.net'
PATH = '.....parquet'


def read_parquet():
    file_system = AzureBlobFileSystem(connection_string=CONNECTION_STRING)
    kwargs = {'path': PATH, 'engine': 'pyarrow', 'filesystem': file_system}
    return pd.read_parquet(**kwargs)

Install the python package for editing

pip install -e .

serve the script.py file

import panel as pn

from segfault import read_parquet
# from .utils import read_parquet

if pn.state.served:
    read_parquet()

I don't understand why the read_parquet function has to be in package and not in a module file utils.py right next to script.py. But that is the case for me.

[MarcSkovMadsen]

Maybe related to https://github.com/holoviz/panel/issues/6934#issue-2370107143.

[hoxbro]

I can't recreate it. But I don't have the same data as you. Could you try with:

import pandas as pd
import s3fs


def read_parquet():
    fs = s3fs.S3FileSystem(anon=True, asynchronous=True)
    url = "s3://assets.holoviews.org/data/airline_flights.parq"
    kwargs = {"path": url, "engine": "pyarrow", "filesystem": fs}
    return pd.read_parquet(**kwargs)

As it is a segfault, can you report the versions and how they are installed (it looks like it is UV)? Can you try with another Python version, maybe even locally, if it is possible?

If you remove the engine kwargs, does it not segfault?

[philippjfr]

I don't understand why the read_parquet function has to be in package and not in a module file utils.py right next to script.py. But that is the case for me.

Modules get reloaded, packages installed into site-packages do not.

[philippjfr]

Hmm, actually I misread with pip install -e . it's not in site-packages, so both are reloaded.

[MarcSkovMadsen]

I will try thanks.

Is it not already possible to infer from the stacktrace above where the problem is and where a fix could be applied?

I'm thinking that here

  File "/home/jovyan/repos/mt-pm-reporting/.venv/lib/python3.11/site-packages/panel/command/serve.py", line 736, in invoke
    super().invoke(args)

Some safeguarding is needed in case the loop is already started?

[MarcSkovMadsen]

Isn't the main problem/ issue that with --dev option the file served is run before the server is started? See https://panel.holoviz.org/tutorials/basic/caching.html#exercise-enable-dev.

And that is not the case without --dev option.

[philippjfr]

Do you have the same problem with --warm. That also runs the app before the server starts.