Learn lessons from the PyParallel project

[thedrow]

I believe our sub-interpreters work in PEP 554 can be greatly enhanced with the ideas of the PyParallel project.

Since we're no longer going to share the GIL between sub-interpreters this creates an optimization opportunity we haven't considered.

I believe we can create a Python interpreter which is:

1. Multi-Core

2. Doesn't require serialization on reads from other threads.

3. Unlike PyParallel, does not crash on writes between threads.

The changes made by the PyParallel allow us to read objects from other threads without holding the GIL. They do so by removing the GC on parallel threads.

We could in theory create sub-interpreters which don't have a GC at all and use the ideas in their diffs by creating a flag short_lived=True on Interpreter.run() which does so. Since we're initializing a new interpreter from scratch I don't think (but not sure) we have to take a snapshot and rollback to the previous state of the memory. If we were to release the sub-interpreter, we'd release all the memory associated to it.

For that we need to make the GC and GIL optional on sub-interpreters.

Implementing the parallel module on top of this mechanism using asyncio should be feasible since sharing a file descriptor between sub-interpreters is possible.

Writing is a matter of creating a pair of channels behind the scenes.

Suppose we have two threads A & B: Thread A declares a variable foo which is set to 1. Thread B executes foo = 2. This creates a RecvChannel on Thread A's interpreter object and a SendChannel on Thread B. Instead of emitting:

dis.dis("foo = 2")
  1           0 LOAD_CONST               0 (1)
              2 STORE_NAME               0 (foo)
              4 LOAD_CONST               2 (None)
              6 RETURN_VALUE

We will emit an opcode which creates a SendChannel on the interpreter if it doesn't exist and calls send with the value 2 on the key foo.

To update, we need to halt the execution of Thread A. If thread A was a coroutine we could halt its execution, context switch into our code which calls our RecvChannel.recv(), update the value and context switch back.

Naturally this doesn't work with lists, sets or any other collection. For those we'd have to create special collections which wraps the collection and provide a differential view.

This may or may not be a lot like STM (cc @arigato, @cfbolz) but it sounds more feasible now with the work we're doing on sub-interpreters.

@tpn is the creator of the PyParallel experiment so we can ask him questions if we needed to.

Goal 3 can be deferred to a later time. As for Goal 2, we can consider implementing it now and either crash on writes or simply make the variables local to the sub-interpreter whenever we write to them.

[crusaderky]

Unless there's some fundamental optimization I'm missing, the idea of having short-lived subinterpreters is fundamentally flawed. Starting a new interpreter can take hundreds of milliseconds, because you have to reload all modules from the OS disk cache. With subinterpreters, it's going to be only marginally faster than starting a new process with the 'spawn' multiprocessing context. So if you create a subinterpreter, make it execute a short task, and then kill it, it's going to be slower than having a long-running pool that is simply sent pickled data, just like a multiprocessing pool works but minus the IPC overhead.

I'm also not sure I grasp how variable sharing is supposed to work. How do you tell apart a private variable from a shared one?

[thedrow]

Oh in that case can't you copy the loaded modules to the new interpreter instead?

If we're allocating a variable in our sub-interpreter, it should be in a dictionary held by the interpreter state. If it isn't, it belongs to another interpreter.

[ericsnowcurrently]

These are great ideas, but ones that should be explored after we have the fundamental functionality available. We can circle back to this later. Feel free to run with it now though. :smile:

[tpn]

(Just wanted to add a quick note saying that I'm delighted to see PyParallel included in these discussions; I think that there were some very interesting outcomes from the design decisions that I took in that project that should be considered (even if to reject) when discussing multi-core Python options.)

[thedrow]

@tpn I wish you could simply contribute the API for parallel contexts without the extra module. That will allow us to experiment with it and see how we can improve multi-core python further.

Can we chat about this?

[thedrow]

@ericsnowcurrently I'll need you to complete #34, #29 and possibly others first. We just have to make sure a GC and a GIL are optional for a sub-interpreter. Is that doable soon?

[ericsnowcurrently]

@thedrow, #34 (GIL) is blocked by pretty much everything else and will happen pretty much last. #29 (GC) is already done.

[ericsnowcurrently]

@tpn, FWIW, I drew inspiration from PyParallel (particularly the focus on isolation). :smile:

[thedrow]

Can we introduce a disable_gc keyword argument? Should I open a separate issue about that?

[ericsnowcurrently]

A keyword argument for what?

[thedrow]

For disabling the garbage collector when starting a new sub-interpreter.