msgspec icon indicating copy to clipboard operation
msgspec copied to clipboard

Fix segmentation fault on Python 3.13

Open shadchin opened this issue 1 month ago • 5 comments

Fix for https://github.com/jcrist/msgspec/issues/910 and https://github.com/jcrist/msgspec/issues/868

The problem is only with Python 3.13, in Python 3.14 PyObject_GC_New does this initialization itself, see https://github.com/python/cpython/pull/123192

shadchin avatar Nov 27 '25 11:11 shadchin

I'm curious though how you tracked it down to this?

It was difficult :) The short version:

From gdb bt found following calls:

4  make_dict_from_instance_attributes
#5  _PyObject_MaterializeManagedDict_LockHeld
#6  _PyObject_MaterializeManagedDict
#7  ensure_managed_dict

In next step, I found that values was not initialized:

(gdb) frame 4
#4  0x00000000023d7c59 in make_dict_from_instance_attributes (interp=0x2fcc2e8 <_PyRuntime+104400>, keys=0x7ffff4e24f20, values=0x7ffff54dc070) at /home/shadchin/arc/arcadia/contrib/tools/python3/Objects/dictobject.c:6736
6736                track += _PyObject_GC_MAY_BE_TRACKED(val);
(gdb) info locals
val = <unknown at remote 0xcdcdcdcdcdcdcdcd>
i = 0
used = 1
track = 0
size = 30
res = 0x7ffff1b535b0
(gdb) print values
$1 = (PyDictValues *) 0x7ffff54dc070
(gdb) print keys
$2 = (PyDictKeysObject *) 0x7ffff4e24f20
(gdb) x/10gx values
0x7ffff54dc070: 0xcdcdcdcdcdcdcdcd      0xcdcdcdcdcdcdcdcd
0x7ffff54dc080: 0xcdcdcdcdcdcdcdcd      0xcdcdcdcdcdcdcdcd
0x7ffff54dc090: 0xcdcdcdcdcdcdcdcd      0xcdcdcdcdcdcdcdcd
0x7ffff54dc0a0: 0xcdcdcdcdcdcdcdcd      0xcdcdcdcdcdcdcdcd
0x7ffff54dc0b0: 0xcdcdcdcdcdcdcdcd      0xcdcdcdcdcdcdcdcd

Then I had to study what managed dict/PyDictValues and Struct_alloc. I found that it's true for 3.13 that we don't initialize PyDictValues, looked at the CPython code, how it does it, found the _PyObject_InitInlineValues call.

shadchin avatar Nov 27 '25 13:11 shadchin

Do we know why 3.14 is fine?

ofek avatar Nov 27 '25 15:11 ofek

@shadchin seems like _PyObject_InitInlineValues isn't available?

https://github.com/jcrist/msgspec/actions/runs/19736064121/job/56566395047?pr=960#step:8:228

provinzkraut avatar Nov 27 '25 15:11 provinzkraut

Do we know why 3.14 is fine?

Python 3.14+ does this initialization itself - https://github.com/python/cpython/blob/main/Python/gc.c#L2377-L2379

shadchin avatar Nov 27 '25 17:11 shadchin

@shadchin seems like _PyObject_InitInlineValues isn't available?

https://github.com/jcrist/msgspec/actions/runs/19736064121/job/56566395047?pr=960#step:8:228

We have our own build system, fix is working for us, but I forgot to check on a clean environment, sorry. I tried to make a fix but it didn't help. There's a conflict with write_u64.

I'll go upstream with it.

shadchin avatar Nov 27 '25 18:11 shadchin

I took a different approach and it helped)

Maybe something else will prompt in upstream.

shadchin avatar Dec 13 '25 16:12 shadchin