reloadium Segmentation fault while using Django with python-devtools

Describe the bug

Using Django==3.2.13 this is a massive project in general. Loaded 416 watched modules so far from paths.

When I used python-devtools got a segfault with:

Thread 44 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffa1329700 (LWP 13697)]
_PyMem_DebugMalloc (ctx=0x555555b59380 <_PyMem_Debug+96>, nbytes=40) at Objects/obmalloc.c:2329
warning: Source file is more recent than executable.
2329        _PyMem_DebugCheckGIL(__func__);

bt output:

(gdb) bt
#0  _PyMem_DebugMalloc (ctx=0x555555b59380 <_PyMem_Debug+96>, nbytes=40) at Objects/obmalloc.c:2329
#1  0x00005555556029a6 in PyObject_Malloc (size=<optimized out>) at Objects/obmalloc.c:685
#2  0x00005555556ed825 in _PyObject_GC_Alloc (use_calloc=use_calloc@entry=0, basicsize=<optimized out>) at Modules/gcmodule.c:2225
#3  0x00005555556ee7be in _PyObject_GC_Malloc (basicsize=<optimized out>) at Modules/gcmodule.c:2252
#4  0x00005555556ee7eb in _PyObject_GC_New (tp=0x7ffff6e893c0) at Modules/gcmodule.c:2264
...
#56155 0x00005555555bec20 in PyVectorcall_Call (callable=callable@entry=0x7fffa2563ef0, tuple=tuple@entry=0x7ffff73e3050, kwargs=kwargs@entry=0x7fffa24d6a10) at Objects/call.c:230
#56156 0x00005555555bef65 in _PyObject_Call (tstate=0x7fffbc002c30, callable=callable@entry=0x7fffa2563ef0, args=args@entry=0x7ffff73e3050, kwargs=kwargs@entry=0x7fffa24d6a10) at Objects/call.c:265
#56157 0x00005555555befe3 in PyObject_Call (callable=callable@entry=0x7fffa2563ef0, args=args@entry=0x7ffff73e3050, kwargs=kwargs@entry=0x7fffa24d6a10) at Objects/call.c:292
--Type <RET> for more, q to quit, c to continue without paging--
#56158 0x000055555567ccea in do_call_core (tstate=tstate@entry=0x7fffbc002c30, func=func@entry=0x7fffa2563ef0, callargs=callargs@entry=0x7ffff73e3050, kwdict=kwdict@entry=0x7fffa24d6a10) at Python/ceval.c:5092
#56159 0x0000555555687562 in _PyEval_EvalFrameDefault (tstate=0x7fffbc002c30, f=0x7fffcc1e3990, throwflag=<optimized out>) at Python/ceval.c:3552
#56160 0x00005555555be8db in _PyEval_EvalFrame (throwflag=0, f=0x7fffcc1e3990, tstate=0x7fffbc002c30) at ./Include/internal/pycore_ceval.h:40
#56161 function_code_fastcall (tstate=0x7fffbc002c30, co=<optimized out>, args=0x7fffe013eec0, args@entry=0x7fffe013eeb8, nargs=1, globals=<optimized out>) at Objects/call.c:329
#56162 0x00005555555bf232 in _PyFunction_Vectorcall (func=<optimized out>, stack=0x7fffe013eeb8, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:366
#56163 0x000055555567caff in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=0x7fffe013eeb8, callable=0x7ffff5d2beb0, tstate=0x7fffbc002c30) at ./Include/cpython/abstract.h:118
#56164 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=0x7fffe013eeb8, callable=0x7ffff5d2beb0) at ./Include/cpython/abstract.h:127
#56165 call_function (tstate=tstate@entry=0x7fffbc002c30, pp_stack=pp_stack@entry=0x7fffa1328a60, oparg=oparg@entry=1, kwnames=kwnames@entry=0x0) at Python/ceval.c:5044
#56166 0x0000555555687052 in _PyEval_EvalFrameDefault (tstate=0x7fffbc002c30, f=0x7fffe013ed40, throwflag=<optimized out>) at Python/ceval.c:3476
#56167 0x00005555555be8db in _PyEval_EvalFrame (throwflag=0, f=0x7fffe013ed40, tstate=0x7fffbc002c30) at ./Include/internal/pycore_ceval.h:40
#56168 function_code_fastcall (tstate=0x7fffbc002c30, co=<optimized out>, args=0x7fffcc01b260, args@entry=0x7fffcc01b258, nargs=1, globals=<optimized out>) at Objects/call.c:329
#56169 0x00005555555bf232 in _PyFunction_Vectorcall (func=<optimized out>, stack=0x7fffcc01b258, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:366
#56170 0x000055555567caff in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=0x7fffcc01b258, callable=0x7ffff5d2e230, tstate=0x7fffbc002c30) at ./Include/cpython/abstract.h:118
#56171 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=0x7fffcc01b258, callable=0x7ffff5d2e230) at ./Include/cpython/abstract.h:127
#56172 call_function (tstate=tstate@entry=0x7fffbc002c30, pp_stack=pp_stack@entry=0x7fffa1328c80, oparg=oparg@entry=1, kwnames=kwnames@entry=0x0) at Python/ceval.c:5044
#56173 0x0000555555687052 in _PyEval_EvalFrameDefault (tstate=0x7fffbc002c30, f=0x7fffcc01b0e0, throwflag=<optimized out>) at Python/ceval.c:3476
#56174 0x00005555555be8db in _PyEval_EvalFrame (throwflag=0, f=0x7fffcc01b0e0, tstate=0x7fffbc002c30) at ./Include/internal/pycore_ceval.h:40
#56175 function_code_fastcall (tstate=0x7fffbc002c30, co=<optimized out>, args=0x7fffa1328dc0, args@entry=0x7fffa1328db8, nargs=1, globals=<optimized out>) at Objects/call.c:329
#56176 0x00005555555bf232 in _PyFunction_Vectorcall (func=<optimized out>, stack=0x7fffa1328db8, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:366
#56177 0x0000555555783cdd in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0x7fffa1328db8, callable=0x7ffff5d2bf50, tstate=0x7fffbc002c30) at ./Include/cpython/abstract.h:118
#56178 method_vectorcall (method=<optimized out>, args=0x7ffff73e3068, nargsf=<optimized out>, kwnames=0x0) at Objects/classobject.c:61
#56179 0x00005555555bec20 in PyVectorcall_Call (callable=callable@entry=0x7fffa145bef0, tuple=tuple@entry=0x7ffff73e3050, kwargs=kwargs@entry=0x0) at Objects/call.c:230
#56180 0x00005555555bef65 in _PyObject_Call (tstate=0x7fffbc002c30, callable=0x7fffa145bef0, args=0x7ffff73e3050, kwargs=0x0) at Objects/call.c:265
#56181 0x00005555555befe3 in PyObject_Call (callable=<optimized out>, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:292
#56182 0x000055555574238b in t_bootstrap (boot_raw=boot_raw@entry=0x7fffa249c9a0) at ./Modules/_threadmodule.c:1040
#56183 0x00005555556da7ec in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:236
#56184 0x00007ffff7f91609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#56185 0x00007ffff7d5c133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

bt full:

(gdb) bt full
#0  _PyMem_DebugMalloc (ctx=0x555555b59380 <_PyMem_Debug+96>, nbytes=40) at Objects/obmalloc.c:2329
        __func__ = "_PyMem_DebugMalloc"
#1  0x00005555556029a6 in PyObject_Malloc (size=<optimized out>) at Objects/obmalloc.c:685
No locals.
#2  0x00005555556ed825 in _PyObject_GC_Alloc (use_calloc=use_calloc@entry=0, basicsize=<optimized out>) at Modules/gcmodule.c:2225
        tstate = <optimized out>
        gcstate = 0x555555baa118
        size = <optimized out>
        g = <optimized out>
        __PRETTY_FUNCTION__ = "_PyObject_GC_Alloc"
        op = <optimized out>
#3  0x00005555556ee7be in _PyObject_GC_Malloc (basicsize=<optimized out>) at Modules/gcmodule.c:2252
No locals.
#4  0x00005555556ee7eb in _PyObject_GC_New (tp=0x7ffff6e893c0) at Modules/gcmodule.c:2264
        op = <optimized out>

No syms for reloader:

objdump --syms /home/shako/REPOS/.../env/lib/python3.9/site-packages/reloadium/reloader.cpython-39-x86_64-linux-gnu.so

/home/shako/REPOS/.../env/lib/python3.9/site-packages/reloadium/reloader.cpython-39-x86_64-linux-gnu.so:     file format elf64-x86-64

SYMBOL TABLE:
no symbols

nm /home/shako/REPOS/.../env/lib/python3.9/site-packages/reloadium/reloader.cpython-39-x86_64-linux-gnu.so
nm: /home/shako/REPOS/.../env/lib/python3.9/site-packages/reloadium/reloader.cpython-39-x86_64-linux-gnu.so: no symbols

To Reproduce Steps to reproduce the behavior:

enable python-devtools by following this: https://python-devtools.helpmanual.io/usage/#manual-install My sitecustomize.py path: /usr/local/lib/python3.9/sitecustomize.py
Fire up django with given version
add sum dummy simple view
add debug() statements as in: https://python-devtools.helpmanual.io/usage/#debug
make it to reload
then remove debug() statements
make it to reload
expect segfault

Expected behavior No segfault obviously

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: Ubuntu
OS version: 20.04.4 LTS
Reloadium package version: 0.9.0
PyCharm plugin version: No plugin running from terminal
Editor: PyCharm
Run mode: run

Additional context Add any other context about the problem here.

Aug 05 '22 07:08 ShahriyarR

Possibly fixed in reloadium 0.9.1 (can't reproduce on python 3.9.7 from pyenv). Could you confirm it's working for you as well?

Aug 09 '22 07:08 dkrystki

There is no segfault but I got a new fail as:

Update Import: datetime(1)
Update Imported: Decimal(2)
Update Imported: Optional(3)
Update Import: pytz(6)
Update Imported: relativedelta(7)
Update Imported: apps(8)
Update Imported: settings(9)
Update Imported: GenericRelation(10)
Update Imported: ContentType(11)
Update Imported: ArrayField(12)
Update Imported: ObjectDoesNotExist(13)
Update Imported: PermissionDenied(13)
Update Imported: ValidationError(13)
Update Imported: MinValueValidator(14)
Update Imported: models(15)
Update Imported: Q(16)
Update Imported: Sum(16)
Update Imported: post_delete(17)
Update Imported: post_save(17)
Update Imported: pre_save(17)
Update Imported: atomic(18)
Update Imported: receiver(19)
Update Imported: get_object_or_404(20)
Update Imported: reverse(21)
Update Imported: timezone(22)
Update Imported: cached_property(23)
Update Imported: escape(24)
Update Imported: mark_safe(25)
Update Imported: slugify(26)
Update Imported: AUTO_INVOICE_HELP_TEXT(28-43)
Traceback (most recent call last):
ValueError: empty names on ImportFrom

Oops, it seems like you made a mistake.
Fix your changes and save to reload.

In fact I just add single space in the file and I am pretty sure there is no mistake there.

reloadium -h
INFO
  version      0.9.1

Aug 09 '22 13:08 ShahriyarR

Interesting, definitely a bug. Can't reproduce it though. Could you turning off caching (startup performance will suffer for sure) and try again?

2022-08-10_19:07:46

Aug 10 '22 09:08 dkrystki

@dkrystki Well I have no plugin installed. I run reloadium on the terminal. Do you have a commandline flag for disabling the cache?

Aug 11 '22 07:08 ShahriyarR

@ShahriyarR Right, you have to set RW_CACHE=False enviromental flag then.

Aug 11 '22 07:08 dkrystki

okay, export RW_CACHE=False reloadium run manage.py runserver 0.0.0.0:8000 --noreload

The result of changes:

Reloadium experienced an error.
Please submit a github issue to let us know at https://github.com/reloadware/reloadium
Last changes could not be hot reloaded.
Please undo to continue
Reloadium experienced an error.
Please submit a github issue to let us know at https://github.com/reloadware/reloadium
Last changes could not be hot reloaded.
Please undo to continue
Update Module: xxxxx.schedule.views.service.rcrs_conjobs
Update Method: ServiceRcrCRUD.get_form(65-318)

Reloadium experienced an error.
Please submit a github issue to let us know at https://github.com/reloadware/reloadium
Last changes could not be hot reloaded.
Please undo to continue
Reloadium experienced an error.
Please submit a github issue to let us know at https://github.com/reloadware/reloadium
Last changes could not be hot reloaded.
Please undo to continue

Aug 11 '22 07:08 ShahriyarR

Is get_form method a regular method (non classmethod etc)?

Aug 11 '22 07:08 dkrystki

Ok I found the culprit, it's the cached_property decorator causing this. Will be fixed in the next release.

Aug 11 '22 12:08 dkrystki

Fixed in reloadium 0.9.2 (PyCharm plugin 0.8.5)

Sep 05 '22 15:09 dkrystki