Likely memory leak with Python protobuf implementation

[pfreixes]

What version of protobuf and what language are you using? Version: v3.6.1 Language: Python

What operating system (Linux, Windows, ...) and version? Linux What runtime / compiler are you using (e.g., python version or gcc version) Python 3.7

What did you do?

I've executed the following script [1] that shows how after deallocating the protobuf object but iterating first over all of the nested and repeated fields do not return all of the memory to the operating system.

[1] https://github.com/pfreixes/proto_leak/blob/master/leak.py

What did you expect to see

After deallocating the object [2] the memory footprint should get back to the initial state, as has been seen in the previous example [3] where deallocating the object without previously iterating over the nested repeated fields didn't leak any kind of memory.

The following snippet shows the output of the previous command which basically shows you which is the memory footprint - VMS in MB - at each step, take a look at the last step which must have a memory footprint close to the initial memory used by the program. A significant difference can be spotted.

$ python leak.py
Memory consumed at the beginning 2371.203125
Memory consumed after parsed 3100.8984375
Memory consumed after deallocating 2422.8984375
Memory consumed after parsed 3100.9296875
Memory consumed after iterating 3243.4296875
Memory consumed after deallocating 3100.890625

We have repeated this experiment with a production proto file which its data takes serialized almost 1GB. In the following snippet you can appreciate that the differences are even more clear:

Memory consumed at the beginning 2389.2109375
Memory consumed after parsed 15919.359375
Memory consumed after deallocating 3461.35546875
Memory consumed after parsed 15918.859375
Memory consumed after iterating 28317.22265625
Memory consumed after deallocating 15797.609375

The last step tells us that the memory consumed is almost 16GB, while in the beginning, the memory consumed by the process is rough 2GB. This such difference makes me think that this could be related to a likely memory leak.

[2] https://github.com/pfreixes/proto_leak/blob/master/leak.py#L34 [3] https://github.com/pfreixes/proto_leak/blob/master/leak.py#L19

What did you see instead?

An incrase of the memory usage by the program.

[amauryfa]

Hi, in your script del foo is not enough. You should also del bar and del x to ensure that there is no reference to the data. Or move the iterating steps to another function.

Freed memory is not always returned to the system: at best, only empty pages of 4k can be released. With Python the situation is even worse, because pymalloc will allocate its own large arenas and release them when they are completely empty. This is not a memory leak: if you retry the operation, memory should not grow again; it should rapidly reach its maximum after a few attempts.

[amauryfa]

To explain the two levels of memory usage: ParseFromString completely works in C++ land, the message itself takes memory, but there is still a single Python object. When iterating over the data though, each sub-item will be fetched as a Python object which reference that part of the data. These objects are cached, and not released until the toplevel itself message is freed. This is why the memory usage increases while the code loops over the message.

We could do it differently, and cache these sub-messages in a less aggressive way. This would reduce memory usage, at the cost of more cpu to access fields, because the Python objects need to be rebuilt the second time.

[pfreixes]

Thanks for your answer @amauryfa.

I was aware of the lazy behavior of the Python API for protobuf [1], what I didn't know is that the scope of the variable of for statement persists after the iteration. This might explain why the memory is not almost all released - without taking into account the internal fragmentation because of the arena or the default allocator.

Though, It's for me unclear why by having only a reference to one of the repeated elements - the last one - most of the memory is unreleased, something that makes me think that the whole repeated element is still referenced.

am I wrong? Is there any technical decision behind the Python protobuf implementation that proofs this?

[1] https://groups.google.com/forum/#!topic/protobuf/7uF1s3BjoH8

[panaetov]

So is any opportunities to release memory forcely? We are using protobuf in our project and memory consumption is very sad, because memory only grows.

Now we switched off some services and memory graphic is horizontal line, but when we switch them on again memory will grow again.

[tbonza]

Running into this as well. My use case is that I've written out a significant number of individual protobufs to batch out a larger task and make better use of multiple CPUs. When trying to aggregate these batches as a final step, I'm unable to deallocate memory as described above.

My aggregation code only requires about 3 functions so I may be inclined to just try doing this in golang so we can move things along.

[github-actions[bot]]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.

This issue is labeled inactive because the last activity was over 90 days ago.

[github-actions[bot]]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please reopen it.

This issue was closed and archived because there has been no new activity in the 14 days since the inactive label was added.