feat: support Python 3.11

[Yun-Kim]

Description

Python 3.11 has made some new changes to their C internal API that we need to address in a couple of places, mostly in our profiler.

I've created some initial work adding Python 3.11 to Riot and C API fixes. The fixes include:

• Python 3.11 moved _PyFloat_Pack8() into their internal C API and replaced it with PyFloat_Pack8() to their public C API (Link to CPython issue). We use it once in ddtrace.internal.pack_template.h where I've added a Python version check to use the correct corresponding function.
• Python 3.11 now creates PyFrameObject lazily and cannot be directly accesed through a PyThreadState object, and instead recommends using a helper function PyThreadState_GetFrame() which was earlier introduced in Python 3.9 (Link to issue).
• Python 3.11 removed redundant data fields from their exception representation (exc.exc_type and exc.exc_traceback) and now recommends using the Py_TYPE() and PyException_GetTraceback() helpers to access those values from the only remaining field exc.exc_value. (Link to CPython change, Link to Cython fix)

Checklist

• [ ] Title must conform to conventional commit.
• [ ] Add additional sections for feat and fix pull requests.
• [ ] Ensure tests are passing for affected code.
• [ ] Library documentation and/or Datadog's documentation site is updated. Link to doc PR in description.

Motivation

Design

Testing strategy

Relevant issue(s)

Testing strategy

Reviewer Checklist

• [ ] Title is accurate.
• [ ] Description motivates each change.
• [ ] No unnecessary changes were introduced in this PR.
• [ ] PR cannot be broken up into smaller PRs.
• [ ] Avoid breaking API changes unless absolutely necessary.
• [ ] Tests provided or description of manual testing performed is included in the code or PR.
• [ ] Release note has been added for fixes and features, or else changelog/no-changelog label added.
• [ ] All relevant GitHub issues are correctly linked.
• [ ] Backports are identified and tagged with Mergifyio.
• [ ] Add to milestone.

[P403n1x87]

Just a note that the bytecode module doesn't support Python 3.11 yet, so the debugger and internal test suites won't pass.

[Yun-Kim]

Just a note that the bytecode module doesn't support Python 3.11 yet, so the debugger and internal test suites won't pass.

As discussed, the bytecode library currently does not support Python 3.11, and is used by the internal, debugger, graphene, graphql components/integrations. Until then, I'll exclude those test suites from running Python 3.11.

[jd]

I've replied to @Yun-Kim already, just leaving a trace here: there's an infinite loop in the memory profiler with Python 3.11 which I was able to identify. I'll work on a fix.

[jd]

@Yun-Kim I've a fix for the memory part, but it seems there's another issue with the changes on the stack collector

Exception in thread ddtrace.profiling.collector.stack:StackCollector:
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.11-dev/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/root/project/ddtrace/internal/periodic.py", line 73, in run
    self._target()
  File "/root/project/ddtrace/profiling/collector/__init__.py", line 42, in periodic
    for events in self.collect():
                  ^^^^^^^^^^^^^^
  File "ddtrace/profiling/collector/stack.pyx", line 500, in ddtrace.profiling.collector.stack.StackCollector.collect
    all_events = stack_collect(

  File "ddtrace/profiling/collector/stack.pyx", line 382, in ddtrace.profiling.collector.stack.stack_collect
    frames, nframes = _traceback.traceback_to_frames(exc_traceback, max_nframes)

  File "ddtrace/profiling/collector/_traceback.pyx", line 23, in ddtrace.profiling.collector._traceback.traceback_to_frames
    cpdef traceback_to_frames(traceback, max_nframes):

  File "ddtrace/profiling/collector/_traceback.pyx", line 35, in ddtrace.profiling.collector._traceback.traceback_to_frames
    frame = tb.tb_frame

AttributeError: type object 'type' has no attribute 'tb_frame'

Not sure what it is yet. The code looks ok to me, but there might be something we missed.

[jd]

memalloc should be fixed with: https://github.com/DataDog/dd-trace-py/pull/4304

[Yun-Kim]

@Yun-Kim I've a fix for the memory part, but it seems there's another issue with the changes on the stack collector

Exception in thread ddtrace.profiling.collector.stack:StackCollector:
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.11-dev/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/root/project/ddtrace/internal/periodic.py", line 73, in run
    self._target()
  File "/root/project/ddtrace/profiling/collector/__init__.py", line 42, in periodic
    for events in self.collect():
                  ^^^^^^^^^^^^^^
  File "ddtrace/profiling/collector/stack.pyx", line 500, in ddtrace.profiling.collector.stack.StackCollector.collect
    all_events = stack_collect(

  File "ddtrace/profiling/collector/stack.pyx", line 382, in ddtrace.profiling.collector.stack.stack_collect
    frames, nframes = _traceback.traceback_to_frames(exc_traceback, max_nframes)

  File "ddtrace/profiling/collector/_traceback.pyx", line 23, in ddtrace.profiling.collector._traceback.traceback_to_frames
    cpdef traceback_to_frames(traceback, max_nframes):

  File "ddtrace/profiling/collector/_traceback.pyx", line 35, in ddtrace.profiling.collector._traceback.traceback_to_frames
    frame = tb.tb_frame

AttributeError: type object 'type' has no attribute 'tb_frame'

Not sure what it is yet. The code looks ok to me, but there might be something we missed.

@jd Interesting, from my understanding I've merely replaced the exception type & traceback with utility functions instead of accessing them directly, but it appears that the traceback object is actually a type object? Do you think there might've been some implicit switching of the exception tuple's ordering?

[Yun-Kim]

memalloc should be fixed with: #4304

@jd Thank you for the memalloc fix! 🎉

[mergify[bot]]

@Yun-Kim this pull request is now in conflict 😩

[mergify[bot]]

@Yun-Kim this pull request is now in conflict 😩

[majorgreys]

While this is not full support with wheels, we should update https://ddtrace.readthedocs.io/en/stable/versioning.html#supported-runtimes.

[Yun-Kim]

@P403n1x87 @jd Since Dynamic Instrumentation and Profiling aren't yet compatible with Python 3.11, I thought it would be better to not allow users to run these using Python 3.11 instead of causing random crashes during runtime. To this end I've added runtime checks to raise a runtime error if either Dynamic Instrumentation/Profiling are started using Python 3.11. WDYT?

[mergify[bot]]

@Yun-Kim this pull request is now in conflict 😩

[mergify[bot]]

@Yun-Kim this pull request is now in conflict 😩

[Yun-Kim]

@mergifyio backport 1.6

[mergify[bot]]

backport 1.6

✅ Backports have been created

• #4387 feat(tracer): add Python 3.11 support (backport #4125) has been created for branch 1.6