disco-dop
disco-dop copied to clipboard
replace multiprocessing.Pool
multiprocessing pools work fine unless any kind of error condition arises...
- [ ] properly detect segmentation faults, out of memory, &c.
concurrent.futures
does this, but doesn't take aninitializer
... (works on both python 3.3 and backported python 2 version) - [x] handle ctrl-C (resolved in python 3.3+ for both
multiprocessing
andconcurrent.futures
; cf. http://bugs.python.org/issue9205 ) - [ ] robustness: retry failed jobs (but this might make things worse in certain cases, so better to design jobs so that half-finished jobs can be resumed later)
- [ ] do not rely on copy-on-write memory inherited from parent process; load data in child processes.
- [x] Full Python traceback from children.
parser.workerfunc()
decorator does this.