msgspec
msgspec copied to clipboard
jcrist
Fix annotations support on 3.14
With this change, the tests run for me on a local build of Python 3.14. There are a lot of failures related to sys.getrefcount() but that seems to be an unrelated issue.
Closes #810. Fixes #651. Fixes #795.
I get a copule of SystemError: error return without exception set:
___________ TestTypedDict.test_total_partially_optional[json-False] ____________
self = <test_common.TestTypedDict object at 0x7f25e7f160a0>
proto = <module 'msgspec.json' from '/builddir/build/BUILD/python-msgspec-0.19.0-build/BUILDROOT/usr/lib64/python3.14/site-packages/msgspec/json.py'>
use_typing_extensions = False
@pytest.mark.parametrize("use_typing_extensions", [False, True])
def test_total_partially_optional(self, proto, use_typing_extensions):
if use_typing_extensions:
tex = pytest.importorskip("typing_extensions")
cls = tex.TypedDict
else:
cls = TypedDict
class Base(cls):
a: int
b: str
class Ex(Base, total=False):
c: str
dec = proto.Decoder(Ex)
x = {"a": 1, "b": "two", "c": "extra"}
> assert dec.decode(proto.encode(x)) == x
E SystemError: <built-in method decode of msgspec.json.Decoder object at 0x7f25e89b85e0> returned NULL without setting an exception
tests/test_common.py:2106: SystemError
__________ TestTypedDict.test_total_partially_optional[msgpack-False] __________
self = <test_common.TestTypedDict object at 0x7f25e7fcab70>
proto = <module 'msgspec.msgpack' from '/builddir/build/BUILD/python-msgspec-0.19.0-build/BUILDROOT/usr/lib64/python3.14/site-packages/msgspec/msgpack.py'>
use_typing_extensions = False
@pytest.mark.parametrize("use_typing_extensions", [False, True])
def test_total_partially_optional(self, proto, use_typing_extensions):
if use_typing_extensions:
tex = pytest.importorskip("typing_extensions")
cls = tex.TypedDict
else:
cls = TypedDict
class Base(cls):
a: int
b: str
class Ex(Base, total=False):
c: str
dec = proto.Decoder(Ex)
x = {"a": 1, "b": "two", "c": "extra"}
> assert dec.decode(proto.encode(x)) == x
E SystemError: error return without exception set
tests/test_common.py:2106: SystemError
___________ TestTypedDict.test_required_and_notrequired[json-False] ____________
self = <test_common.TestTypedDict object at 0x7f25e813e180>
proto = <module 'msgspec.json' from '/builddir/build/BUILD/python-msgspec-0.19.0-build/BUILDROOT/usr/lib64/python3.14/site-packages/msgspec/json.py'>
use_typing_extensions = False
@pytest.mark.parametrize("use_typing_extensions", [False, True])
def test_required_and_notrequired(self, proto, use_typing_extensions):
if use_typing_extensions:
module = "typing_extensions"
else:
module = "typing"
ns = pytest.importorskip(module)
if not hasattr(ns, "Required"):
pytest.skip(f"{module}.Required is not available")
source = f"""
from __future__ import annotations
from {module} import TypedDict, Required, NotRequired
class Base(TypedDict):
a: int
b: NotRequired[str]
class Ex(Base, total=False):
c: str
d: Required[bool]
"""
with temp_module(source) as mod:
dec = proto.Decoder(mod.Ex)
x = {"a": 1, "b": "two", "c": "extra", "d": False}
> assert dec.decode(proto.encode(x)) == x
E SystemError: <built-in method decode of msgspec.json.Decoder object at 0x7f25e91a0590> returned NULL without setting an exception
tests/test_common.py:2144: SystemError
__________ TestTypedDict.test_required_and_notrequired[msgpack-False] __________
self = <test_common.TestTypedDict object at 0x7f25e8199550>
proto = <module 'msgspec.msgpack' from '/builddir/build/BUILD/python-msgspec-0.19.0-build/BUILDROOT/usr/lib64/python3.14/site-packages/msgspec/msgpack.py'>
use_typing_extensions = False
@pytest.mark.parametrize("use_typing_extensions", [False, True])
def test_required_and_notrequired(self, proto, use_typing_extensions):
if use_typing_extensions:
module = "typing_extensions"
else:
module = "typing"
ns = pytest.importorskip(module)
if not hasattr(ns, "Required"):
pytest.skip(f"{module}.Required is not available")
source = f"""
from __future__ import annotations
from {module} import TypedDict, Required, NotRequired
class Base(TypedDict):
a: int
b: NotRequired[str]
class Ex(Base, total=False):
c: str
d: Required[bool]
"""
with temp_module(source) as mod:
dec = proto.Decoder(mod.Ex)
x = {"a": 1, "b": "two", "c": "extra", "d": False}
> assert dec.decode(proto.encode(x)) == x
E SystemError: error return without exception set
tests/test_common.py:2144: SystemError
Do you also get those?
I didn't how, are you running the test suite exactly?
I get these failures which all seem related to getrefcount calls:
FAILED tests/test_common.py::TestGenericStruct::test_generic_struct_info_cached[json] - assert 3 == 4
FAILED tests/test_common.py::TestGenericStruct::test_generic_struct_info_cached[msgpack] - assert 3 == 4
FAILED tests/test_common.py::TestGenericDataclassOrAttrs::test_generic_info_cached[dataclass-json] - assert 3 == 4
FAILED tests/test_common.py::TestGenericDataclassOrAttrs::test_generic_info_cached[dataclass-msgpack] - assert 3 == 4
FAILED tests/test_convert.py::TestConvert::test_custom_input_type_works_with_any - assert 2 == 3
FAILED tests/test_convert.py::TestConvert::test_custom_input_type_works_with_custom - assert 2 == 3
FAILED tests/test_convert.py::TestConvert::test_custom_input_type_works_with_dec_hook - assert 1 == 2
FAILED tests/test_convert.py::TestInt::test_int_subclass - assert 2 == 3
FAILED tests/test_convert.py::TestBinary::test_bytes_subclass - AssertionError: assert 1 == 2
FAILED tests/test_convert.py::TestEnum::test_int_enum_int_subclass - assert 1 == 2
FAILED tests/test_json.py::TestDatetime::test_decode_timezone_cache - assert 2 == 3
FAILED tests/test_json.py::TestStruct::test_decode_struct - AssertionError: assert 2 == 3
FAILED tests/test_msgpack.py::TestTypedDecoder::test_decode_memoryview_zerocopy[bytes] - AssertionError: assert 2 == 3
FAILED tests/test_msgpack.py::TestTypedDecoder::test_decode_memoryview_zerocopy[memoryview] - AssertionError: assert 2 == 3
FAILED tests/test_msgpack.py::TestTypedDecoder::test_vartuple_lengths[1] - AssertionError: assert 2 == 3
FAILED tests/test_msgpack.py::TestTypedDecoder::test_vartuple_lengths[31] - AssertionError: assert 2 == 3
FAILED tests/test_msgpack.py::TestTypedDecoder::test_vartuple_lengths[32] - AssertionError: assert 2 == 3
FAILED tests/test_msgpack.py::TestTypedDecoder::test_vartuple_lengths[255] - AssertionError: assert 2 == 3
FAILED tests/test_msgpack.py::TestTypedDecoder::test_vartuple_lengths[256] - AssertionError: assert 2 == 3
FAILED tests/test_msgpack.py::TestTypedDecoder::test_vartuple_lengths[65535] - AssertionError: assert 2 == 3
FAILED tests/test_msgpack.py::TestTypedDecoder::test_vartuple_lengths[65536] - AssertionError: assert 2 == 3
FAILED tests/test_struct.py::test_struct_reference_counting - assert 2 == 3
To run it I do:
$ python -VV
Python 3.14.0b1+ (heads/3.14:2a089244f0d, May 26 2025, 08:38:42) [Clang 15.0.0 (clang-1500.3.9.4)]
$ python -m pytest -s -m "not mypy and not pyright"
That's the latest tip of the 3.14 branch.
The failures you post feel like they wouldn't be related to retrieving annotations; the code I'm changing is just in gathering annotations at class creation time, and whatever is happening in those tests is after the class is already created.
Oh actually this is because of a bug in b1 that I fixed (https://github.com/python/cpython/issues/133701); the fix will be in b2 which is about to go out.
I can reproduce it with the following change in the test:
$ git diff
diff --git a/tests/test_common.py b/tests/test_common.py
index 38898be..1c4c969 100644
--- a/tests/test_common.py
+++ b/tests/test_common.py
@@ -2099,6 +2099,7 @@ class TestTypedDict:
class Ex(Base, total=False):
c: str
+ Ex.__annotations__ = {"c": "str"}
dec = proto.Decoder(Ex)
I do think that indicates a bug in msgspec; presumably it shouldn't crash even if people mess with the __annotations__ manually. I'll see if I can submit a fix.
There are a lot of failures related to sys.getrefcount() but that seems to be an unrelated issue.
I opened https://github.com/jcrist/msgspec/pull/854
I've confirmed via local testing that this PR fixes the Python 3.14 compatibility issue I noted in https://github.com/lmstudio-ai/lmstudio-python/issues/153
Ping @jcrist - we're getting close to the final release date of Python 3.14. It'd be nice to have this merged to unblock further work to add 3.14 support here and in downstream packages that use msgspec.
Another ping here. I know @kumaraditya303 has a followup for this to add support for the free-threaded build which he plans to send in as soon as this PR is merged.
Is @kumaraditya303's free-thread branch at https://github.com/kumaraditya303/msgspec/tree/thread-safe?
There are a lot of failures related to sys.getrefcount() but that seems to be an unrelated issue.
Those are also to be expected for Python 3.14. There is a new GC that will count a number of references, so any unit tests depending on sys.getrefcount() will have to deal with different values before and after 3.14. I've seen lots of if...else... branches in other libraries with similar tests.
#854 is a draft PR to tackle the refcount tests just by allowing more relaxed measurements in general.
Is @kumaraditya303's free-thread branch at https://github.com/kumaraditya303/msgspec/tree/thread-safe?
Yes, I created https://github.com/jcrist/msgspec/pull/877 for adding free-threading support.
I'm in talks with Jim about co-maintenance and should hopefully hear back in the next few days. If all goes well I plan to merge and release everything rapidly.
Here's the issue tracking the performance regression https://github.com/jcrist/msgspec/issues/880