Python 3.12 support

[kakkoyun]

Hey @benfred, first of all, I'm a massive fan of the project! I have learned a lot about Python by reading this codebase [1].

I have seen that Python 3.12 tests are failing https://github.com/benfred/py-spy/pull/618 and also, starting from ~~3.12~~ 3.13, Python provides a struct with offsets for out-of-process tools https://github.com/python/cpython/pull/106598

I would like to ask about the state (or plan) for the Python 3.12 support and offer help if you would like to accept contributions for this part.

[1]: As a testament to that, I have created this project to use eBPF for profiling Python code.

[kakkoyun]

and also, starting from ~3.12~ 3.13, Python provides a struct with offsets for out-of-process tools python/cpython#106598

I was overly enthusiastic about the changes. Turns out they're merged into the main and scheduled to be released for the 3.13 release.

 git branch -r --contains b444bfb0a325dea8c29f7b1828233b00fbf4a1cb
  origin/HEAD -> origin/main
  origin/main

Regardless, I'm still eager to help with 3.12 support.

[kakkoyun]

Relevant: _PyRuntime.tstate_current removed for the sake of storing it in a thread-local storage https://github.com/python/cpython/issues/103323

[benfred]

Thanks for the links @kakkoyun ! I've started on this - and have a very basic implementation running locally. There are currently a couple of issues that I still need to sort out on this though before we can fully support python 3.12:

• [ ] GIL state has been moved to thread local storage like you mentioned - and detecting which thread holds the GIL no longer works
• [x] Thread names aren't getting picked up
• [ ] Line numbers are incorrect
• [ ] There seems to be a new <interpreter trampoline> frame in the stack - that we might want to filter out

As an example here is the current py-spy dump output on the tests/scripts/longsleep.py test case:

Process 43243: python tests/scripts/longsleep.py
Python v3.12.0 (/home/ben/miniconda3/envs/py12/bin/python3.12)
Thread 43243 (idle)
    longsleep (longsleep.py:-359)
    <module> (longsleep.py:-3)
    <interpreter trampoline> (<shim>:1)

You can see most of the issues here (aside from the missing GIL) - the line numbers are clearly wrong, its not picking up the thread name etc , but aside from that its at least starting to get the right output.

I'm hoping to get everything aside from the GIL detection working in the next couple of days, and get a release out with this (and some other fixes).

[kakkoyun]

Thanks for the detailed update, @benfred. Please don't hesitate to let me know if I can be helpful.

I can try to take a shot at the GIL detection problem.

[benfred]

Initial PR is up here https://github.com/benfred/py-spy/pull/642 - still needs a bunch of work (including line numbers being wrong), but its a start at least

[mdczaplicki]

Hey there, any update on this?

[ianozsvald]

I'm working on our 3rd edition of O'Reilly's High Performance Python book. I've covered Py-Spy in the previous edition and I will continue to show it. We're using Python 3.12 for the book (it'll still be relevant when we publish early in 2025). Currently I'm noting that Py-Spy doesn't support 3.12, it'd be great to know if there's a timeline for support. I'd happily change that note in the book once 3.12 is supported. Thanks for your continued support on this tool, it can be very handy!

[kakkoyun]

@benfred I finally figured out how to read the thread state from the TLS. (i.e. https://github.com/parca-dev/parca-agent/pull/2553)

I hope to find some free cycles in the upcoming weeks to port changes to here.

[mg-christian-esser]

Any progress on this issue?

[kakkoyun]

I haven't started it yet. I recently changed jobs, and I haven't been able to find any free-cycles yet. If anyone else has more time, I'd be happy to help with the reviews and guidance.

[battaglia01]

What is left to do? Just the line numbers?

[olejorgenb]

Since there's so many duplicates I'll risk "spamming" this issue with the alternatives I posted here (TIL: use https://github.com/P403n1x87/austin for now)