colmena icon indicating copy to clipboard operation
colmena copied to clipboard

Support launching Task Server as Thread, not Process

Open WardLT opened this issue 3 years ago • 1 comments

Avoid copying between the Task Server and Thinker may outweigh the need to avoid the GIL in some cases. I have the FuncXTaskServer clearly in mind here, where the only think it does is communicate to another server.

As pointed out by @gpauloski , shared memory between the Task Server/Thinker will be especially useful as we can start resolving object proxies as soon as they are received by the task server.

We will need to implement Thread Queue support (#36) before addressing this issue.

WardLT avatar Jan 27 '22 12:01 WardLT

An idea on implementation that will also open up #119 : Do not subclass Process but, instead, provide a start_server method which takes a queue as an argument. This will fix the issues with have with multiprocessing queues and pickling.

WardLT avatar Nov 28 '23 14:11 WardLT