python-betterproto icon indicating copy to clipboard operation
python-betterproto copied to clipboard

Pickling a Betterproto message

Open tijs2 opened this issue 3 years ago • 2 comments

Hi,

I have found an issue when I wanted to use this library in combination with multiprocessing/pickling. I am using version 2.0.0b5

When the resulting message is send back by one of my running processes with the map function it changes empty values and also when I only run pickle they are changed.

The message:

syntax = "proto3";

message TestMessage {
    string value = 1;
}

To code:

test_message = TestMessage()
print(test_message)
pickled = pickle.dumps(test_message)
unpickled = pickle.loads(pickled)
print(pickle.loads(pickled))
print(unpickled.value)
print(type(unpickled.value))

The output:

TestMessage()
TestMessage(value=<object object at 0x7febf0791fe0>)
<object object at 0x7fd9ef242ff0>
<class 'object'>

So it seems it adds an object instead of an empty value.

tijs2 avatar Sep 06 '22 12:09 tijs2

Thanks for the bug report, if this is critically important you can just pass the bytes for the message around until someone implements the required methods for pick to not do this

Gobot1234 avatar Sep 06 '22 12:09 Gobot1234

Yes that indeed works but it saves a lot of processing time when I could use pickle. Because when I test pickle vs TestMessage().parse() for a bit more complicated messages it seems to be almost 10 times faster.

tijs2 avatar Sep 06 '22 13:09 tijs2

I think to fix this we need to implement the dunder methods __getstate__ and __setstate__ on betterproto.Message?

See the following article section: https://docs.python.org/3/library/pickle.html#handling-stateful-objects

cetanu avatar Jan 25 '23 05:01 cetanu