Parallelization error

[smmaurer]

The test_parallel_lottery_choices unit test is failing in certain environments.

My local machine, MacOS 10.15:

• py36 and py37: works
• py38 and py39: error

GitHub Actions (link):

• ubuntu: all python versions work
• mac: looks like same pattern as on my machine
• windows: all python versions fail

@mxndrwgrdnr, any chance you've run into this? I'll look into it more when i have a chance..

Sample error message

probs = <function probs.<locals>.probs_callable at 0x7fa810bd0b80>

    def test_parallel_lottery_choices(obs, alts, mct, probs):
        """
        Test that parallel lottery choices can run and that there
        aren't any duplicate choices
    
        """
        num_cpus = multiprocessing.cpu_count()
        batch_size = int(np.ceil(len(obs) / num_cpus))
>       choices = parallel_lottery_choices(
            obs, alts, mct, probs, chooser_batch_size=batch_size)

tests/test_simulation.py:226: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../anaconda/envs/temp38/lib/python3.8/site-packages/choicemodels/tools/simulation.py:462: in parallel_lottery_choices
    proc.start()
../../../../anaconda/envs/temp38/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
../../../../anaconda/envs/temp38/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
../../../../anaconda/envs/temp38/lib/python3.8/multiprocessing/context.py:284: in _Popen
    return Popen(process_obj)
../../../../anaconda/envs/temp38/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
../../../../anaconda/envs/temp38/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
../../../../anaconda/envs/temp38/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <Process name='Process-2' parent=19435 initial>
file = <_io.BytesIO object at 0x7fa8007189f0>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'mct.<locals>.mct_callable'

../../../../anaconda/envs/temp38/lib/python3.8/multiprocessing/reduction.py:60: AttributeError