colmena
colmena copied to clipboard
Support launching Task Server as Thread, not Process
Avoid copying between the Task Server and Thinker may outweigh the need to avoid the GIL in some cases. I have the FuncXTaskServer clearly in mind here, where the only think it does is communicate to another server.
As pointed out by @gpauloski , shared memory between the Task Server/Thinker will be especially useful as we can start resolving object proxies as soon as they are received by the task server.
We will need to implement Thread Queue support (#36) before addressing this issue.
An idea on implementation that will also open up #119 : Do not subclass Process but, instead, provide a start_server
method which takes a queue as an argument. This will fix the issues with have with multiprocessing queues and pickling.