Python 3.13 support

[iherasymenko]

When trying to install ddtrace-py with the latest Python 3.13 RC builds, the installation fails with the following error.

building 'ddtrace.profiling.collector._memalloc' extension
creating build/temp.linux-aarch64-cpython-313/ddtrace/profiling/collector
gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -fPIC -I/tmp/tmperfe434l/.venv/include -I/usr/local/include/python3.13 -c ddtrace/profiling/collector/_memalloc.c -o build/temp.linux-aarch64-cpython-313/ddtrace/profiling/collector/_memalloc.o
gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -fPIC -I/tmp/tmperfe434l/.venv/include -I/usr/local/include/python3.13 -c ddtrace/profiling/collector/_memalloc_heap.c -o build/temp.linux-aarch64-cpython-313/ddtrace/profiling/collector/_memalloc_heap.o
gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -fPIC -I/tmp/tmperfe434l/.venv/include -I/usr/local/include/python3.13 -c ddtrace/profiling/collector/_memalloc_tb.c -o build/temp.linux-aarch64-cpython-313/ddtrace/profiling/collector/_memalloc_tb.o
gcc -shared build/temp.linux-aarch64-cpython-313/ddtrace/profiling/collector/_memalloc.o build/temp.linux-aarch64-cpython-313/ddtrace/profiling/collector/_memalloc_heap.o build/temp.linux-aarch64-cpython-313/ddtrace/profiling/collector/_memalloc_tb.o -L/usr/local/lib -o build/lib.linux-aarch64-cpython-313/ddtrace/profiling/collector/_memalloc.cpython-313-aarch64-linux-gnu.so
building 'ddtrace.internal._threads' extension
creating build/temp.linux-aarch64-cpython-313/ddtrace/internal
g++ -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -fPIC -I/tmp/tmperfe434l/.venv/include -I/usr/local/include/python3.13 -c ddtrace/internal/_threads.cpp -o build/temp.linux-aarch64-cpython-313/ddtrace/internal/_threads.o -std=c++17 -Wall -Wextra
ddtrace/internal/_threads.cpp: In constructor ‘GILGuard::GILGuard()’:
ddtrace/internal/_threads.cpp:23:14: error: ‘_Py_IsFinalizing’ was not declared in this scope; did you mean ‘Py_IsFinalizing’?
   23 |         if (!_Py_IsFinalizing())
      |              ^~~~~~~~~~~~~~~~
      |              Py_IsFinalizing
ddtrace/internal/_threads.cpp: In constructor ‘AllowThreads::AllowThreads()’:
ddtrace/internal/_threads.cpp:45:14: error: ‘_Py_IsFinalizing’ was not declared in this scope; did you mean ‘Py_IsFinalizing’?
   45 |         if (!_Py_IsFinalizing())
      |              ^~~~~~~~~~~~~~~~
      |              Py_IsFinalizing
ddtrace/internal/_threads.cpp: In destructor ‘AllowThreads::~AllowThreads()’:
ddtrace/internal/_threads.cpp:50:14: error: ‘_Py_IsFinalizing’ was not declared in this scope; did you mean ‘Py_IsFinalizing’?
   50 |         if (!_Py_IsFinalizing())
      |              ^~~~~~~~~~~~~~~~
      |              Py_IsFinalizing

[sanchda]

@iherasymenko thank you for the report! We're still addressing 3.13 support for the library. For the time being, I think it is safe to treat 3.13 as unsupported by ddtrace. At the current time we do not have a precise timeline (sorry about that).

I don't really have a specific workaround to suggest here, since we have a lot of native code that needs to be made away of changes to the runtime API in 3.13.

[sanchda]

Oops, just realized I mischaracterized this as a profiler issue, but I see that the compilation site is actually coming from a different component. Re-routing this issue, but unfortunately I think my overall message is still accurate.

[sanchda]

@emmettbutler directing to you in case you have anything more optimistic to say on the topic of 3.13 support.

[leandrodamascena]

Hi, do you have any updates on this issue?! I really want to use ddtrace with Python 3.13.

I was reading through the Python 3.13 API changes and the necessary changes to make ddtrace aware of Python 3.13 and I see some important changes.

_Py_IsFinalizing Part of the Stable ABI since version 3.13.

_Py_IsFinalizing is part of the Stable ABI and now its Py_IsFinalizing, so, the following code (we can optimize, of course) can keep backward compatibility (https://github.com/DataDog/dd-trace-py/blob/main/ddtrace/internal/_threads.cpp) and add support for Python 3.13

#if PY_VERSION_HEX < 0x30d0000
    if (!_Py_IsFinalizing()) {
#else
    if (!Py_IsFinalizing()) {
#endif

Py_BUILD_CORE must be define when importing internal

Accessing some internal headers now requires defining Py_BUILD_CORE. However, if I'm not reading wrong, this define is only meant to be used when building CPython itself. So, I think adding Py_BUILD_CORE define (+ python version check) on top of the file https://github.com/DataDog/dd-trace-py/blob/b849a2a1197265a43a4960f49d302f4eccaa3470/ddtrace/appsec/_iast/_stacktrace.c can fix the problem.

Building datadog/profiling/stack_v2

I see some errors like fatal error: 'internal/pycore_opcode.h' file not found, but I don't know if this is a problem when cmake is linking the headers or not. I'm still debugging.

I see some other debugs when compiling it locally, but I don't know if that's my environment or not. I can't find in the repository any guidance to set up my local environment correctly.

Please let me know if I can help in any other way to add this support in a timely manner.

[P403n1x87]

@leandrodamascena thanks for your interest in getting ddtrace to work with CPython 3.13. We are currently working towards this, and you have already identified many of the places where support needs to be extended. Besides that, we also need a new release of the bytecode library, which is on its way. We currently don't have a timescale for when all the work will be done, but we are confident we can get there within a few weeks 🤞

[leandrodamascena]

@leandrodamascena thanks for your interest in getting ddtrace to work with CPython 3.13. We are currently working towards this, and you have already identified many of the places where support needs to be extended. Besides that, we also need a new release of the bytecode library, which is on its way. We currently don't have a timescale for when all the work will be done, but we are confident we can get there within a few weeks 🤞

Thanks for letting me know @P403n1x87! Feel free to reach out if you need someone to do some testing with Python 3.13

[evanstjabadi]

@P403n1x87, Nice updated shared.

Besides that, we also need a new release of the bytecode library, which is https://github.com/MatthieuDartiailh/bytecode/pull/146.

It looks like bytecode is now cleared for Python 3.13 🎉

[dima-aronov-dev]

Looking forward for the python 3.13 support :)

[olibook]

any update ?

[kpaperski]

Hi, do you have any timeline for when ddtrace will be available for Python 3.13?

[mik3y]

FWIW we turned off datadog traces due to this issue; turns out we can live without it. Python 3.13 being 2+ months old, this was the only thing blocking our upgrade, which was needed for other reasons.

[Mulugruntz]

this was the only thing blocking our upgrade

Same here.

[williamglasse]

Same here This is the only dependency blocking our upgrade to Python 13 Are there any suitable workarounds? (Other than not using ddtrace)

[Goganasan86]

still waiting(

[ishaan-jaff]

bump on this - any update ?

it's critical for us cc @Kyle-Verhoog

[miketheman]

From https://github.com/DataDog/dd-trace-py/issues/11833#issuecomment-2568043362

We are planning to include wheels for python 3.13 in the next release (v2.19.0). The official release should go out within the next two weeks.

Here's our first release candidate that ships wheels for 3.13: v2.19.0rc1 (release). We'd love to get feedback.

[averagechris]

fwiw the release candidate is working well for me on python 3.13. Thanks everyone!

[kpaperski]

For me 2.19.0 release candidate is working as well! Thank you and waiting for official release

[Mulugruntz]

https://github.com/DataDog/dd-trace-py/releases/tag/v2.19.0 has been released 5 days ago. It would have been nice to update this issue :)

To be noted that https://github.com/DataDog/dd-trace-py/releases/tag/v2.19.1 was released 4 days ago as well.

Can we consider this issue resolved now?

Thank you to all the DD team!

[ngoduykhanh]

I've tried the latest release (2.19.1), which works well 👍 . Thank you, team!

[rblaine95]

It seem that ddtrace doesn't support 3.13.1:

[2025-01-27 16:08:04] [DEBUG] datadog.autoinstrumentation(pid: 1): Found incompatible runtime: cpython 3.13.1. Supported runtimes: {'cpython': {'min': Version(version=(3, 7), constraint=''), 'max': Version(version=(3, 13), constraint='')}}

[sanchda]

@rblaine95 which version of ddtrace are you using?

[rblaine95]

@sanchda We're using DDTrace 2.20.0 and also tested 2.19.2

Edit: Just a quick note, this is with ddtrace auto instrumentation in kubernetes with helm

[owensrd]

Hi, we are also looking to move to 3.13 and this is the only remaining thing holding us off. Running the 3.13 docker image and ddtrace 2.20.0 we are seeing this error:

2025-01-27 19:03:40.589Z 2025-01-27 19:03:40,587 ERROR [ddtrace.bootstrap.preload] [preload.py:70] [dd.trace_id=0 dd.span_id=0] - failed to enable profiling 2025-01-27 19:03:40.589Z Traceback (most recent call last): 2025-01-27 19:03:40.589Z File "/usr/local/lib/python3.13/site-packages/ddtrace/bootstrap/preload.py", line 68, in 2025-01-27 19:03:40.589Z import ddtrace.profiling.auto # noqa: F401 2025-01-27 19:03:40.589Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-27 19:03:40.589Z File "/usr/local/lib/python3.13/site-packages/ddtrace/internal/module.py", line 295, in _exec_module 2025-01-27 19:03:40.589Z self.loader.exec_module(module) 2025-01-27 19:03:40.589Z ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ 2025-01-27 19:03:40.589Z File "/usr/local/lib/python3.13/site-packages/ddtrace/profiling/init.py", line 5, in 2025-01-27 19:03:40.589Z from .profiler import Profiler # noqa:F401 2025-01-27 19:03:40.589Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-27 19:03:40.589Z File "/usr/local/lib/python3.13/site-packages/ddtrace/internal/module.py", line 295, in _exec_module 2025-01-27 19:03:40.589Z self.loader.exec_module(module) 2025-01-27 19:03:40.589Z ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ 2025-01-27 19:03:40.589Z File "/usr/local/lib/python3.13/site-packages/ddtrace/profiling/profiler.py", line 28, in 2025-01-27 19:03:40.589Z from ddtrace.profiling.collector import stack 2025-01-27 19:03:40.589Z File "/usr/local/lib/python3.13/site-packages/ddtrace/internal/module.py", line 234, in _create_module 2025-01-27 19:03:40.589Z return self.loader.create_module(spec) 2025-01-27 19:03:40.589Z ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ 2025-01-27 19:03:40.589Z ImportError: /usr/local/lib/python3.13/site-packages/ddtrace/profiling/collector/stack.cpython-313-x86_64-linux-gnu.so: undefined symbol: _PyThread_CurrentExceptions

Are there any workarounds for this or just wait for another release to come?

[jamesbeith]

Also getting a crash with undefined symbol: _PyThread_CurrentExceptions running on Python 3.13.1 with ddtrace==2.20.0.

failed on setup with "ImportError: /venv/lib/python3.13/site-packages/ddtrace/profiling/collector/stack.cpython-313-x86_64-linux-gnu.so: undefined symbol: _PyThread_CurrentExceptions"
tests/integration/common/interfaces/common/test_management_commands.py:40: in datadog_profiler_results
    from ddtrace.profiling.exporter import http
/venv/lib/python3.13/site-packages/ddtrace/internal/module.py:295: in _exec_module
    self.loader.exec_module(module)
/venv/lib/python3.13/site-packages/_pytest/assertion/rewrite.py:184: in exec_module
    exec(co, module.__dict__)
/venv/lib/python3.13/site-packages/ddtrace/profiling/exporter/http.py:19: in <module>
    from ddtrace.profiling import exporter
/venv/lib/python3.13/site-packages/ddtrace/internal/module.py:295: in _exec_module
    self.loader.exec_module(module)
/venv/lib/python3.13/site-packages/_pytest/assertion/rewrite.py:184: in exec_module
    exec(co, module.__dict__)
/venv/lib/python3.13/site-packages/ddtrace/profiling/__init__.py:5: in <module>
    from .profiler import Profiler  # noqa:F401
/venv/lib/python3.13/site-packages/ddtrace/internal/module.py:295: in _exec_module
    self.loader.exec_module(module)
/venv/lib/python3.13/site-packages/_pytest/assertion/rewrite.py:184: in exec_module
    exec(co, module.__dict__)
/venv/lib/python3.13/site-packages/ddtrace/profiling/profiler.py:28: in <module>
    from ddtrace.profiling.collector import stack
/venv/lib/python3.13/site-packages/ddtrace/internal/module.py:234: in _create_module
    return self.loader.create_module(spec)
E   ImportError: /venv/lib/python3.13/site-packages/ddtrace/profiling/collector/stack.cpython-313-x86_64-linux-gnu.so: undefined symbol: _PyThread_CurrentExceptions

[noelleleigh]

I see there are no wheels available for Windows and Python 3.13 as of ddtrace 2.20.0. Will those be available at some point?

[emmettbutler]

@rblaine95 Here's a fix for the lib-injection early exit https://github.com/DataDog/dd-trace-py/pull/12283

[emmettbutler]

@owensrd @jamesbeith The profiling product currently does not work with Python 3.13. It will be updated to do so in a future release.

I intended this information to be in a release note (https://github.com/DataDog/dd-trace-py/pull/11792), but it seems not to be included in any.

Here's the original release note for ease of access, with some updates reflecting the current state of main:

    The following limitations currently apply to support for Python 3.13:
      - ``ddtrace`` is not supported on Windows with Python 3.13
      - Appsec Threat Detection is not tested against Django, Flask, or FastAPI with 3.13
      - Automatic Service Naming is not tested with 3.13
      - The following products are not tested with 3.13:
        - Code Coverage
        - Appsec IAST
        - Data Streams Monitoring
        - CI Visibility
        - Continuous Profiling
      - The following integrations are not tested with 3.13:
        - anthropic
        - consul
        - freezegun
        - gevent
        - google_generativeai
        - gunicorn
        - langchain
        - mysqlclient
        - opentracing
        - psycopg
        - psycopg2
        - pymysql
        - pytest
        - pytest-bdd
        - pytest-benchmark
        - sanic
        - selenium
        - sqlite3
        - starlette
        - tornado
        - vertexai

cc @sanchda

[usamamashkoor]

I am facing this error

I see there are no wheels available for Windows and Python 3.13 as of ddtrace 3.0.0. Will How I can solve this issue?

[marcelomarkus]

any news about 3.13?