[Question] Merge this POC to CPython

[redradist]

I am curious if this test implementation will be merged in CPython ?

[stonebig]

So say we all, but the timing is right: lot of people and money are working on python 5x speed-up goals.

[satishdash]

This should get into CPython. I'd love to have it in 3.11 with the fastness and this parallelism. Thanks @colesbury for such a detailed work. Appreciate all the efforts. Hope it's rewarded with a merge soon.

[Superskyyy]

I really appreciate the dedications to this work and it will be so nice to see it merge back to upstream. We should definitely push on this once things mature.

[redradist]

@colesbury Looks like you are doing lots of work for updating to up-to-stream CPython ... Will this nogil be added to CPython 3.11 ?

[colesbury]

@redradist No, it won't be added to CPython 3.11. CPython 3.11 will have a "beta" release soon, which means a feature freeze (bug fixes only).

At the Python language summit this week I suggested integrating some of the "nogil" ideas into CPython 3.12 behind a compile flag. It's not clear if that will happen -- there's still a lot of open questions.

Having additional real use cases may help with adoption upstream. So if you have a program or use case that would benefit from removing the GIL, please try it out and report back both any problems you encounter and the results.

[chulucninh09]

Hi. It is really nice to see such a dedicated work for this. I want to contribute my real use case as follow:

Taken the inspiration from LMAX Disruptor written in Java, we (devs in a stock brokerage company) are developing in-memory queue system similar to Disruptor that heavily utilize shared-memory and concurrent model to achieve low-enough latency.

1. Producer continuously put data into the shared buffer/shared list, but we construct it to be ring-buffer liked
2. Consumers continuously poll for data from the shared buffer and consumer it
3. No lock required if there is single producer
4. Producer increase its offset when producing new data to shared buffer. Consumers observe it by polling to get producer's offset value (while True loop)
5. Producer also check for consumers offset to avoid wrapping the slowest consumers

To implement this, currently we are using multiprocessing module with SharedMemory class, each Consumer/Producer lives on separated process. Our producer and consumers are long-running processes.

Currently, due to the GIL, we are limited at communicating between producer and consumers (most likely sharing offset position). Consumers and producer using multiprocessing.Value to share offset value, and that is not the efficient way to observe those values.

In multithreading model, I believe those operations will be more natural in the code, less work-around, and also we get more performance since threads share the same context, no need to pickle/copy data around.

[goophps]

I've been looking forward to it.

[Techno-Fox]

and 3.12 is in development. I know that the summit it could add a compile option for nogil, but are there any updates on it?

[manueldeprada]

Is there any updated page where we can track what is being incorporated into Python 3.12??

[DimitrisJim]

Is there any updated page where we can track what is being incorporated into Python 3.12??

The what's new document keeps track of these things, the current one for 3.12 is available (and continuously updated).

[manueldeprada]

Is there any updated page where we can track what is being incorporated into Python 3.12??

The what's new document keeps track of these things, the current one for 3.12 is available (and continuously updated).

Thank you. I see it's nowhere in the roadmap for 3.12. I was asking more for something like a link where we can follow the status and discussion of the NOGIL integration into Python.

Right now, I could only find some minutes from core-dev meetings and discussions on the mailing list.

It would be great if there was a central thread somewhere that tracked the discussion and status of the NOGIL integration into CPython.

[colesbury]

@manueldeprada I'm currently working on a PEP (Python enhancement proposal). When it's ready, it will be posted to https://discuss.python.org/c/peps/ and I'll share a link to it here.

[colesbury]

Here's the PEP: https://peps.python.org/pep-0703/

[stonebig]

Here's the PEP: https://peps.python.org/pep-0703/

For python-3.12 ?

[colesbury]

Yes, but I wouldn’t read too much into that.

On Tue, Jan 10, 2023 at 1:58 PM stonebig @.***> wrote:

Here's the PEP: https://peps.python.org/pep-0703/

For python-3.12 ?

— Reply to this email directly, view it on GitHub https://github.com/colesbury/nogil/issues/7#issuecomment-1377777045, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFAD6TWX2CRISPVH4ATBA3WRW5PFANCNFSM5GEU4QBQ . You are receiving this because you were mentioned.Message ID: @.***>

[ibrandiay]

Hi, has your interesting PEP finally been accepted for integration into Python 3.12?

[manueldeprada]

Hi, has your interesting PEP finally been accepted for integration into Python 3.12?

After a lot of drama back and forth, it has been accepted by the SC🎊🎊

https://discuss.python.org/t/a-steering-council-notice-about-pep-703-making-the-global-interpreter-lock-optional-in-cpython/30474

Most discussion threads have been closed, the only I see now is: https://discuss.python.org/t/python-abis-and-pep-703/34018

but there is not a central active thread as far as I know. Probably, it will make it into 3.13😁😁