cubed icon indicating copy to clipboard operation
cubed copied to clipboard

Manually manage the GIL with Cython

Open alxmrs opened this issue 1 year ago • 3 comments

I suspect that the performance of the ThreadPoolExecutor would substantially increase if we strategically placed cython with nogil calls.

  • https://cython.readthedocs.io/en/latest/src/userguide/parallelism.html
  • https://stackoverflow.com/questions/49047255/cython-nogil-with-threadpoolexecutor-not-giving-speedups
  • https://stackoverflow.com/questions/56537989/usage-of-threadpoolexecutor-in-conjunction-with-cythons-nogil

There are drawbacks to process based parallelism; namely, there is a high overhead and IPC latency compared to multithreading.

alxmrs avatar Jul 14 '24 12:07 alxmrs