NREL
Support Multiprocessing
As discovered in #187, pickling the road map rtree is currently failing which disables us from leveraging multiprocessing for a batch of traces. Until the pickling issue is resolved we could explore workarounds to allow traces to be processed in parallel. Road maps could be very large and so a solution that allows the road network data to be shared would be ideal.
@jhoshiko - I seem to recall that you've run into a similar issue in the past. Do you have any notes from trying to map match large road networks in parallel that might be useful in this context?
@nreinicke In the past, I would initialize a copy of the road network on each worker, scaling the number of workers based on available memory and the size of the road network. I haven't personally experimented with matching to a road network in shared memory, but I agree that it would be ideal. I think multiprocessing/joblib might have options for instantiating objects into shared memory, but I haven't personally tested anything.
