python
[mypyc] Pickle test failure with Python 3.14.0a1
After being a bit late with Python 3.13, I figured to start testing 3.14 early this time.
With Python 3.14.0a1 the following fails:
# native.py
from typing import Any
def dec(x: Any) -> Any:
return x
@dec
class D:
x: int
class E(D):
y: int
# driver.py
from native import D, E
import pickle
assert not hasattr(D, '__mypyc_attrs__')
assert E.__mypyc_attrs__ == ('y', '__dict__')
e = E()
e.x = 10
e.y = 20
assert e.__getstate__() == {'y': 20, '__dict__': {'x': 10}}
e2 = pickle.loads(pickle.dumps(e))
assert e is not e2 and e.x == e2.x and e.y == e2.y
rm -rf build
rm *.so
python3.14 -m mypyc native.py
python3.14 -c 'import driver.py'
The error message
$ python -c "import driver.py"
Traceback (most recent call last):
File "<string>", line 1, in <module>
import driver.py
File ".../driver.py", line 14, in <module>
assert e is not e2 and e.x == e2.x and e.y == e2.y
^^^^
AttributeError: 'E' object has no attribute 'x'
I bisected this to https://github.com/python/cpython/pull/123192 upstream.
-- This is also part of the mypyc test suite
pytest -n0 mypyc/test/test_run.py::TestRun::run-classes.test::testPickling
I suspect it's something funky when calling PyObject_SetAttr(obj, "__dict__", {"x": 10}), which is what mypyc will do here when unpickling
With a debug Python build you also get a segfault following the AttributeError:
Assertion failed: (0), function _PyObject_InlineValuesConsistencyCheck, file dictobject.c, line 7470.
zsh: abort python driver.py
In lldb:
* frame #4: 0x000000010031b524 python`_PyObject_InlineValuesConsistencyCheck.cold.3 at dictobject.c:7470:5 [opt]
frame #5: 0x00000001000d3e54 python`_PyObject_InlineValuesConsistencyCheck(obj=<unavailable>) at dictobject.c:7470:5 [opt]
frame #6: 0x00000001000d3b94 python`_PyObject_SetManagedDict(obj=0x0000000100bc7420, new_dict=0x0000000000000000) at dictobject.c:7114:5 [opt]
frame #7: 0x00000001000d3fb8 python`PyObject_ClearManagedDict(obj=<unavailable>) at dictobject.c:7176:9 [opt]
frame #8: 0x0000000100c67088 native.cpython-314d-darwin.so`E_dealloc [inlined] E_clear at __native.c:50:5 [opt]
frame #9: 0x0000000100c67024 native.cpython-314d-darwin.so`E_dealloc at __native.c:59:5 [opt]
frame #10: 0x00000001000ebe0c python`_Py_Dealloc(op=0x0000000100bc7420) at object.c:2933:5 [opt]
frame #11: 0x00000001000d41c0 python`dictkeys_decref [inlined] Py_DECREF(filename=<unavailable>, lineno=476, op=0x0000000100bc7420) at refcount.h:367:9 [opt]
frame #12: 0x00000001000d4174 python`dictkeys_decref [inlined] Py_XDECREF(op=0x0000000100bc7420) at refcount.h:476:9 [opt]
frame #13: 0x00000001000d4170 python`dictkeys_decref(interp=<unavailable>, dk=0x0000000100eda3b0, use_qsbr=false) at dictobject.c:460:17 [opt]
frame #14: 0x00000001000cfc2c python`dict_dealloc(self=0x0000000100a6c7d0) at dictobject.c:0 [opt]
frame #15: 0x00000001000ebe0c python`_Py_Dealloc(op=0x0000000100a6c7d0) at object.c:2933:5 [opt]
...
A possible workaround could be to disable the Py_TPFLAGS_INLINE_VALUES here:
https://github.com/python/mypy/blob/f44a60dd9d02ce496561c08ded134d5e2e3bc8ca/mypyc/lib-rt/misc_ops.c#L299-L304
#if PY_MINOR_VERSION == 11
// This is a hack. Python 3.11 doesn't include good public APIs to work with managed
// dicts, which are the default for heap types. So we try to opt-out until Python 3.12.
t->ht_type.tp_flags &= ~Py_TPFLAGS_MANAGED_DICT;
+#elif PY_MINOR_VERSION == 14
+ t->ht_type.tp_flags &= ~Py_TPFLAGS_INLINE_VALUES;
With that the test case passes again (and no other test is failing because of it). However, I'm unsure that's the correct solution. With https://github.com/python/cpython/pull/123192 Py_TPFLAGS_INLINE_VALUES is automatically set for type->tp_itemsize == 0 which is the case here.
From what I can tell the testPickling case also had issues when adding support for 3.12.
- https://github.com/python/mypy/pull/15471
- https://github.com/python/mypy/pull/15574
/CC @JukkaL
This was fixed in https://github.com/python/cpython/pull/134859 which will be included in Python 3.14.0b3.