Kneckter
Multiprocessing pool assumes shared memory
The code uses a multiprocessing pool and tries to share memory through global variables. This is an invalid assumption and fails on macos for me.
pool = multiprocessing.pool.ThreadPool(processes=8)
Switching to the threadpool is a quick hack
What as the error you ran into? I know that the sched_getaffinity part of pool = multiprocessing.Pool(processes=len(os.sched_getaffinity(0))) is Linux-specific so it won't work on MacOS or Windows. I never went back and coded lookups for those but you did hardcode it in your example so you probably hit that issue.
I haven't looked into the ThreadPool before; Also wrote this when I first tried multiprocessing.
The error was related to the use of shared global variables which should not be used with multi processes. It’s a shared nothing model
you can explicitly use some shared memory objects or pipes for communicating or pass back results to map at the end of execution
a thread has everything shared
The affinity for pool size isn’t supported non Linux but I just hard coded a valid for that!