website icon indicating copy to clipboard operation
website copied to clipboard

Switch from "fork" to "spawn" for multiprocessing in tests

Open pradh opened this issue 7 months ago • 1 comments

Some kind of mysterious cloud build platform change broke "fork" start method for python multiprocessing. The "spawn" start method is apparently a safer alternative, since it doesn't inherit parent state into child (sharing state can cause deadlocks or mess up behavior, perhaps like we see now) -- doc.

However, "spawn" is finicky about pickling of methods and objects that get passed to the run() method.

So, this change does three things to switch over to "spawn" mode:

  1. Fix the multiprocessing.Process call we make to start NL server in a way that makes spawn happy
  2. The Web server gets started by an external library (flask_testing) in a way that makes spawn unhappy. So, stop using that (generally unmaintained) library, and instead start web server in our code.
  3. Given our support for spawn, we no longer set fork, so avoid the mac-os specific setting.

pradh avatar Jul 12 '24 07:07 pradh