disco-dop icon indicating copy to clipboard operation
disco-dop copied to clipboard

replace multiprocessing.Pool

Open andreasvc opened this issue 12 years ago • 0 comments

multiprocessing pools work fine unless any kind of error condition arises...

  • [ ] properly detect segmentation faults, out of memory, &c. concurrent.futures does this, but doesn't take an initializer ... (works on both python 3.3 and backported python 2 version)
  • [x] handle ctrl-C (resolved in python 3.3+ for both multiprocessing and concurrent.futures; cf. http://bugs.python.org/issue9205 )
  • [ ] robustness: retry failed jobs (but this might make things worse in certain cases, so better to design jobs so that half-finished jobs can be resumed later)
  • [ ] do not rely on copy-on-write memory inherited from parent process; load data in child processes.
  • [x] Full Python traceback from children. parser.workerfunc() decorator does this.

andreasvc avatar Dec 23 '12 16:12 andreasvc