Commands can't be defined in __main__ module

[vringar]

Due to limitations in dill, classes defined in the __main__ module can't be pickled. Since we send all of our commands through a multiprocess.queue which uses dill internally we can't send Commands defined in the main module.

See https://github.com/uqfoundation/multiprocess/issues/22 and https://github.com/uqfoundation/multiprocess/issues/5 for progress on allowing us to specify a different serializer in multiprocess.

[englehardt]

Possibly relevant: https://stackoverflow.com/questions/60583118/serialize-subclass-of-abstract-class-in-python-3-7-1 https://stackoverflow.com/questions/59058588/pyspark-got-typeerror-can-t-pickle-abc-data-objects

[mmckerns]

Due to limitations in dill, classes defined in the __main__ module can't be pickled.

This isn't true.

Python 3.7.14 (default, Sep 10 2022, 11:17:06) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> class Foo(object):
...   def __init__(self, x):
...     self.x = x
...   def bar(self, y):
...     return self.x + y
... 
>>> f = Foo(1)
>>> 
>>> import multiprocess as mp
>>> mp.Pool().map(f.bar, [1,2,3,4])
[2, 3, 4, 5]
>>> dill.loads(dill.dumps(f)).bar(1)
2
>>> dill.loads(dill.dumps(Foo))(1).bar(1)
2

Can you be more specific? If you are interested in abc classes and methods, some can be handled and some can't. dill currently has issues with _abc_data objects, but there's a PR that is pending for that.

[vringar]

If I copy the contents of LinkCountingCommand over to the demo.py and run python demo.py, I get the following error:

Traceback (most recent call last):
  File "/home/stefan/.conda/envs/openwpm/lib/python3.10/site-packages/multiprocess/queues.py", line 248, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/home/stefan/.conda/envs/openwpm/lib/python3.10/site-packages/multiprocess/reduction.py", line 54, in dumps
    cls(buf, protocol, *args, **kwds).dump(obj)
  File "/home/stefan/.conda/envs/openwpm/lib/python3.10/site-packages/dill/_dill.py", line 620, in dump
    StockPickler.dump(self, obj)
  File "/home/stefan/.conda/envs/openwpm/lib/python3.10/pickle.py", line 487, in dump
    self.save(obj)
  File "/home/stefan/.conda/envs/openwpm/lib/python3.10/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/stefan/.conda/envs/openwpm/lib/python3.10/pickle.py", line 687, in save_reduce
    save(cls)
  File "/home/stefan/.conda/envs/openwpm/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/stefan/.conda/envs/openwpm/lib/python3.10/site-packages/dill/_dill.py", line 1838, in save_type
    _save_with_postproc(pickler, (_create_type, (
  File "/home/stefan/.conda/envs/openwpm/lib/python3.10/site-packages/dill/_dill.py", line 1140, in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
  File "/home/stefan/.conda/envs/openwpm/lib/python3.10/pickle.py", line 692, in save_reduce
    save(args)
  File "/home/stefan/.conda/envs/openwpm/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/stefan/.conda/envs/openwpm/lib/python3.10/pickle.py", line 902, in save_tuple
    save(element)
  File "/home/stefan/.conda/envs/openwpm/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/stefan/.conda/envs/openwpm/lib/python3.10/site-packages/dill/_dill.py", line 1251, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/home/stefan/.conda/envs/openwpm/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/home/stefan/.conda/envs/openwpm/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/home/stefan/.conda/envs/openwpm/lib/python3.10/pickle.py", line 578, in save
    rv = reduce(self.proto)
TypeError: cannot pickle '_abc._abc_data' object

So yeah, I guess we are blocked on the _abc_data PR. Can you link that PR?