How to force a cleanup of memory allocated by the constant initialized variables

[ioancea]

Version: >= 3.15.0 Language: C++ Windows 10, MSVC 14.28.29910

Starting with 3.15.0 version, the global message instances are constant initialized (see https://github.com/protocolbuffers/protobuf/blob/v3.15.0/src/google/protobuf/compiler/cpp/cpp_message.h#L129) which seems to upset some memory detection tools (e.g. AppVerifier) when protobuf is statically linked in a DLL which gets unloaded at runtime (by FreeLibrary).

Backtraces are usually pointing to stuff like:

`operator new+0x0000000000000013 [vctools\crt\vcstartup\src\heap\new_scalar.cpp @ 35] std::_Default_allocate_traits::_Allocate+0x0000000000000028 std::_Allocate<16,std::_Default_allocate_traits,0>+0x0000000000000047 std::allocatorstd::_Container_proxy::allocate+0x0000000000000035 std::_Container_base12::_Alloc_proxy<std::allocatorstd::_Container_proxy std::basic_string<char,std::char_traits,std::allocator>::basic_string<char,std::char_traits,std::allocator>

google::protobuf::internal::ExplicitlyConstructed<std::basic_string<char,std::char_traits,std::allocator> >::DefaultConstruct google::protobuf::internal::SerializeInternalToArray google::protobuf::internal::InitProtobufDefaultsSlow rtps::"dynamic atexit destructor for 'MyMessage_default_instance'" `

In the past I've encountered similar issues with protobuf and global memory but it was possible to force a cleanup (right before unloading the DLL) by calling google::protobuf::ShutdownProtobufLibrary. However, for those constants this doesn't seem to be working.

Thus, the question is if there's a way to either cleanup the memory allocated by the global constants initialization OR to turn off the constant initialization of the global message instances?

[TeBoring]

Hi, could you provide a full instruction to reproduce this issue?

1. the OS
2. code
3. how to compile
4. how to run

[ioancea]

(Late) Update: I was wrong about the initial statement - the google::protobuf::ShutdownProtobufLibrary method is indeed freeing the memory hold by the constant initializers.

However, it seems there is a difference in behavior starting with protobuf 3.15: prior that version the protobuf globals were allocated at the 1st usage while with 3.15+ they're created at "load" time. In my case, I have protobuf statically linked into a shared library which means that a LoadLibrary followed by FreeLibrary leaks with 3.15+.

@TeBoring I have created a small POC for the above part, see the attached archive. The idea of the test is to have a shared library (that statically links against protobuf) which is then dynamically loaded, used and unloaded by an executable that is monitored by appverifier. The dependencies of the POC are: Windows 10 (it should work with other versions too), cmake >= 3.13, appverifier and a compatible MSVC compiler.

Compile and run the POC proto_load_unload_leak.zip

Unpack the zip archive. For every protobuf versions to test against (used 3.14.0 and 3.19.1 in my local tests), do the following:

• Generate the build tree for the version X. E.g. cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DPROTOBUF_VERSION=3.14.0 -B build-with-proto3.14.0 proto_load_unload_leak
• Build the POC. E.g. cmake --build build-with-proto3.14.0. This will create an executable (testapp.exe) and a shared library (testlib.dll)
• From an administrator console (because it's starting appverif) cd build-with-proto3.14.0 && ctest -V. In this step, the testapp.exe will be attached to appverif and then 2 runs will be performed - 1st will explicitly cleanup the protobuf resources from the testlib.dll before unloading it while the 2nd test will only load (aka LoadLibrary) and unload (aka FreeLibrary) the testlib.dll library.

Test results of the POC

The above tests pass if the PROTOBUF_VERSION is <= 3.14.0 and they fail at the run-testapp-load-only case with PROTOBUF_VERSION 3.15+

Possible Solutions

I'm not sure if there will be a lot of real-life scenarios to reach into this sort of setup. However, if that happens (like in my case) it would be nice to have a good (to be read as compliant) solution for it. In general, I considered 2 possible ideas:

• Export a symbol from the shared library which must be called by the user to explicitly free up the memory. That's a bit nasty because if one only does LoadLibrary and FreeLibrary (due to whatever reason), it will have to load and call an extra symbol in order not to leak.
• Automatically cleanup the protobuf resources when the library is unloaded. This can be implemented using yet another static/global C++ object (at dll "level") whose destructor calls protobuf' cleanup. I have implemented an example in the POC, which "activates" the code under the -DPROTO_AUTO_CLEANUP=ON cmake variable. E.g. cmake -DPROTO_AUTO_CLEANUP=ON <rest of params>. I know there might be other solutions, like special hooks which are called when the library is loaded/unloaded (e.g. defining/overwriting DllMain on Windows, the deprecated .init/.fini or the "new" __attribute__((constructor))/((destructor)) on unix) but the global object seemed the most straightforward fix.

Therefore as a conclusion, I'll shift the original question to asking for your suggestion(s) regarding what would be the most compliant solution to (preferably automatically) cleanup protobuf resources in the above described use-case? Could there be any pitfalls in using the global C++ object method? Thank you!

[github-actions[bot]]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.

This issue is labeled inactive because the last activity was over 90 days ago.

[github-actions[bot]]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please reopen it.

This issue was closed and archived because there has been no new activity in the 14 days since the inactive label was added.