Example problem crashing due to un-picklable objects

[lukevolpatti]

I've been trying to run the a2dr example program from the readme but it doesn't seem to work on my computer (MacOS with Python 3.9). The error message I get is as follows:

Traceback (most recent call last):
  File "test2.py", line 22, in <module>
    drs_result = a2dr(prox_list, A_list, b, anderson=False)
  File "/usr/local/lib/python3.8/site-packages/a2dr/solver.py", line 261, in a2dr
    procs[-1].start()
  File "/usr/local/Cellar/[email protected]/3.8.13/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/Cellar/[email protected]/3.8.13/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/local/Cellar/[email protected]/3.8.13/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/usr/local/Cellar/[email protected]/3.8.13/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/Cellar/[email protected]/3.8.13/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/Cellar/[email protected]/3.8.13/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/Cellar/[email protected]/3.8.13/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'precondition.<locals>.proto.<locals>.<lambda>'

I have tried running the program on previous versions of Python, and found that it works on Python 3.7 but fails in a similar fashion on Python 3.8. The Python changelogs for version 3.8 note the following update to the inbuilt multiprocessing library, which I think is the cause of the problem: "On macOS, the spawn start method is now used by default.". The relevant discussion about this update is here, and they mention there that there is potential loss of functionality when dealing with non-pickable objects when switching from fork to spawn for starting new processes.

From the error message, it seems as though the precondition function in a2dr/a2dr/precondition.py might be the culprit. Indeed, just removing the lambda function from the readme example isn't enough to fix the error. Is there a way to implement precondition without lambda functions to see if that fixes things? I suppose an alternative would be switching from spawning back to forking, though I don't know what the side effects of that would be (there's some talk on that in the previously linked discussion).

[LudoDaemen]

I managed to work around the issue by starting the script/notebook with the following fragment :

import multiprocessing multiprocessing.set_start_method('fork')

[junziz]

Thanks a lot Luke for sharing this! Sorry it has been extremely busy at Amazon in the past two months and so I haven't got time to resolve this issue. We will test your solution and update the code as soon as possible. Thanks again!