BUG: pickle throws error "Can't pickle .. objects" when using C-Implementation

[schlichtanders]

Dear wrapt developer,

when using something like

from wrapt import ObjectProxy
class A(ObjectProxy):
    pass
a = A([1,2,3])

import pickle
pickle.dump(a, open("objectproxy.pkl", "wb"))

I get the following pickle TypeError

Traceback (most recent call last):
  File "C:\tools\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2881, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-14-2926618be977>", line 1, in <module>
    pickle.dump(a, open("objectproxy.pkl", "wb"))
TypeError: can't pickle A objects

however if I wrap the ObjectProxy again, adding the python pickle magics which are in fact already defined for Python ObjectProxy

class ObjectProxy2(ObjectProxy):

    def __reduce__(self):
        return identity, (self.__wrapped__,)

    def __reduce_ex__(self, protocol):
        return identity, (self.__wrapped__,)

everything works.

I.e. this seems to be related to the C implementation and might in fact be due to the same source as the bug reported in #101

Hopefully this can be solved, thanks a lot for listening

environment

• Operating System: Windows 10 64bit
• Python: 3.6
• wrapt: 1.10.10

[GrahamDumpleton]

Part of the issue is that one can't really assume a general case as to how pickling should be handled if it is done on a proxy object. Depending on the use case you may or may not want the original wrapped object to pickled. You may instead need the proxy object to also be pickled in some way. So adding in proxy methods for the pickle methods as a default may not be the right idea. That it errors at the moment may therefore in some ways be good as it means you need to determine what is the appropriate behaviour.

Anyway, will leave this issue open so can revisit the issue later.

[schlichtanders]

thanks for your response, what would be the issue with pickling the Proxy object as a whole and unpickling it as a whole? Can you explain?

[GrahamDumpleton]

The #develop branch has been updated now to raise a NotImplementedError from __reduce__() and __reduce_ex__() with clear message that __reduce_ex__() needs to be overridden in the custom object proxy if it is necessary that pickle work for wrapper. It is not proxied to wrapped object automatically, as that doesn't include the proxy object itself, which may be necessary depending on use case.

[zbarry]

To anyone encountering @GrahamDumpleton's new exception in your own code, here's a rough skeleton for an ObjectProxy subclass that can be serialized with Pickle:

class MyProxy(wrapt.ObjectProxy):

    def __init__(self, object_to_wrap, arbitrary_parameter):

        super().__init__(object_to_wrap)

        self._self_arbitrary_parameter = arbitrary_parameter

    @property
    def arbitrary_parameter(self):
        return self._self_arbitrary_parameter

    def __reduce_ex__(self, protocol):
        return type(self), (self.__wrapped__, self.arbitrary_parameter)

I just ripped / refactored this out of some of my own code. Let me know if anything funky happens with it.

[yarikoptic]

haven't done full analysis yet or tried to come up with a workaround, but we are encountering now NotImplementedError: object proxy must define __reduce_ex__() on OSX and Windows (not on linux) with python >= 3.8 in DataLad now while using wrapt.decorator

[yarikoptic]

ping on this issue, any hope to get it addressed?

[GrahamDumpleton]

@yarikoptic As a general solution for wrapt.ObjectProxy am not sure there is a good solution. There is an outside chance that something may be able to be done using wrapt.FunctionWrapper which is used just for decorated functions, so depends on your use case.

If you want to experiment with whether you can get it to work for plain decorated functions, using wrapt version 1.13.0rc3 on PyPi you can try customising the FunctionWrapper used by wrapt.decorator to override the special pickle dunderscore methods.

So something like:

import wrapt

class MyFunctionWrapper(wrapt.FunctionWrapper):
    def __reduce_ex__(self, protocol):
        ... do something custom here, original wrapped function is self.__wrapped__.

@wrapt.decorator(proxy=MyFunctionWrapper)
def mydecorator(wrapped, instance, args, kwargs):
    return wrapped(*args, **kwargs)

I really need someone who understands the in and outs of pickle to work out whether there might be a general solution that may work in most cases at least. May be harder to have something work for wrapt decorators when using signature adapter functions etc though.