multi-core-python icon indicating copy to clipboard operation
multi-core-python copied to clipboard
verified publisher icon ericsnowcurrently

per-interpreter: memory allocators

Open ericsnowcurrently opened this issue 7 years ago • 6 comments

[maybe split into more granular tasks]

  1. move the global state back to Include/internal (see below)
  2. move PyRuntimeState.obj to PyInterpreterState.obj
  3. move PyRuntimeState.mem to PyInterpreterState.mem

Notable concerns:

  • very delicate code
  • allocators needed before main interpreter created (?)
  • keep a global allocator?

The global state for this was moved back out of Include/internal to fix a bug. That PR encapsulates the relevant global state and C-API pretty well.

ericsnowcurrently avatar Aug 24 '18 21:08 ericsnowcurrently

Can we have per interpreter heap and allocators and a flag telling whether we are in the main interpreter? The overhead doesn't sounds terrible.

qizzzh avatar Jun 13 '19 05:06 qizzzh

"per interpreter allocators" is exactly what this issue is for.

What do you mean by "per interpreter heap"? The C heap wouldn't make sense so I'm guessing you're talking about the Python "heap". If so, that's pretty much unrelated to what we're working on here. Consider starting a thread on the python-ideas or python-dev mailing lists.

Where are you looking for a flag about the main interpreter? In the C code all the info is there already. From Python the information is exposed in PEP 554.

ericsnowcurrently avatar Jun 15 '19 00:06 ericsnowcurrently

A workaround is to disable pymalloc. It can even be done by setting PYTHONMALLOC env var to "malloc" for example. libc malloc() is thread safe.

vstinner avatar Mar 19 '20 02:03 vstinner

@vstinner, a workaround for what? per-interpreter allocators or the bug for which we rolled back moving the allocators into PyRuntimeState?

ericsnowcurrently avatar Mar 27 '20 16:03 ericsnowcurrently

@ericsnowcurrently: "@vstinner, a workaround for what?"

pymalloc is not thread safe. If you want to experiment subinterpreters running in parallel in different threads, you can force the usage of the libc malloc()/free() by setting PYTHONMALLOC env var to "malloc".

vstinner avatar Mar 27 '20 18:03 vstinner

A workaround is to disable pymalloc. It can even be done by setting PYTHONMALLOC env var to "malloc" for example. libc malloc() is thread safe.

I proposed the following issue and PR to implement this workaround as a configure build option:

  • https://bugs.python.org/issue40514
  • https://github.com/python/cpython/pull/19926

vstinner avatar May 05 '20 13:05 vstinner