neat-python icon indicating copy to clipboard operation
neat-python copied to clipboard

neat.distributed: use socket module instead of multiprocessing.managers

Open bennr01 opened this issue 6 years ago • 2 comments

I finally found time to complete the rewrite of distributed.py. This PR replaces the usage of the multiprocessing.managers module with the socket module. Here are some advantages of this change:

  • improved stability and performance (tests will no longer fail due to timeout errors)
  • we can now use our own serialization functions instead of relying on pickle, which may be useful if we implement xml/json/... serialization)

These changes are mostly compatible with the old API, however the type of the authkey argument has changed from bytes to str/unicode.

Also, this closes #101.

Edit: It seems like one of the tests still fails randomly. The travis build prior to the PR still succeeded. Edit 2: Seems like the travis failure was caused by some process timeout. After restarting the build, the tests pass, but coverage is still reduced. However, most of the missing coverage seems to be from error catching code, which cant easily be triggered manually (e.g. handling of invalid network messages).

bennr01 avatar Feb 05 '18 17:02 bennr01

Coverage Status

Coverage decreased (-1.0%) to 95.171% when pulling 2ed502b8a46ae339b1371e2a6b8974e710164b9c on bennr01:distributed_socket into c2b79c88667a1798bfe33c00dd8e251ef8be41fa on CodeReclaimers:master.

coveralls avatar Feb 05 '18 17:02 coveralls

Apologies for the long silence--I've been overwhelmed with other stuff and just kind of let this fall idle. I appreciate the patch and will take a look as soon as I get some free time. Thanks!

CodeReclaimers avatar Jun 23 '18 15:06 CodeReclaimers