Incorrect usage of DllMain leads to a crash

[TautvydasZilys]

On Windows, oneTBB tries to load tbbmalloc.dll with LoadLIbraryA from within DLL_PROCESS_ATTACH, which is documented as a big no-no by Microsoft: https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-best-practices

We noticed that this causes crashes on machines with certain versions of VC++ runtime installed. Since this is undefined behaviour, this happens to work in most cases but nevertheless is not a safe thing to do. This seems to predominantly affect Windows 7 machines, but there are no guarantees that it cannot happen on later Windows versions.

Callstack for the LoadLibraryA call:

KernelBase.dll!LoadLibraryExW()
KernelBase.dll!LoadLibraryExA()
kernel32.dll!LoadLibraryA()
tbb12.dll!tbb::detail::r1::dynamic_load(const char * library, const tbb::detail::r1::dynamic_link_descriptor * descriptors, unsigned __int64 required, bool local_binding) Line 441
tbb12.dll!tbb::detail::r1::dynamic_link(const char * library, const tbb::detail::r1::dynamic_link_descriptor * descriptors, unsigned __int64 required, HINSTANCE__ * * handle, int flags) Line 477
tbb12.dll!tbb::detail::r1::initialize_handler_pointers() Line 118
[Inline Frame] tbb12.dll!std::invoke(void(*)() &&) Line 1478
tbb12.dll!std::call_once<void (__cdecl*)(void)>(std::once_flag & _Fx, void(*)() &&) Line 555
tbb12.dll!tbb::detail::r1::initialize_cache_aligned_allocator() Line 139
tbb12.dll!tbb::detail::r1::initialize_allocate_handler(unsigned __int64 size) Line 145
tbb12.dll!tbb::detail::r1::allocate_memory(unsigned __int64 size) Line 215
[Inline Frame] tbb12.dll!tbb::detail::d1::tbb_allocator<std::_Tree_node<tbb::detail::d1::global_control *,void *>>::allocate(unsigned __int64) Line 60
[Inline Frame] tbb12.dll!std::_Tree_node<tbb::detail::d1::global_control *,void *>::_Buyheadnode(tbb::detail::d1::tbb_allocator<std::_Tree_node<tbb::detail::d1::global_control *,void *>> &) Line 344
[Inline Frame] tbb12.dll!std::_Tree<std::_Tset_traits<tbb::detail::d1::global_control *,tbb::detail::r1::control_storage_comparator,tbb::detail::d1::tbb_allocator<tbb::detail::d1::global_control *>,0>>::_Alloc_sentinel_and_proxy() Line 1917
[Inline Frame] tbb12.dll!std::_Tree<std::_Tset_traits<tbb::detail::d1::global_control *,tbb::detail::r1::control_storage_comparator,tbb::detail::d1::tbb_allocator<tbb::detail::d1::global_control *>,0>>::{ctor}(const tbb::detail::r1::control_storage_comparator &) Line 884
[Inline Frame] tbb12.dll!std::set<tbb::detail::d1::global_control *,tbb::detail::r1::control_storage_comparator,tbb::detail::d1::tbb_allocator<tbb::detail::d1::global_control *>>::{ctor}() Line 82
tbb12.dll!tbb::detail::r1::`dynamic initializer for 'allowed_parallelism_ctl''() Line 142
ucrtbase.dll!_initterm()
tbb12.dll!dllmain_crt_process_attach(HINSTANCE__ * const instance, void * const reserved) Line 66
tbb12.dll!dllmain_dispatch(HINSTANCE__ * const instance, const unsigned long reason, void * const reserved) Line 276
ntdll.dll!LdrpRunInitializeRoutines()
ntdll.dll!LdrpInitializeProcess()
ntdll.dll!string "Enabling heap debug options\n"()
ntdll.dll!LdrInitializeThunk()

We implemented a fix on our fork that makes tbb12.dll hard link to tbbmalloc.dll. This solution will likely not be acceptable for you, so I'm opening an issue here for discussion instead of a PR.

[anton-potapov]

@TautvydasZilys that is the versions of VC++ runtime the problem is experienced on ?

[TautvydasZilys]

It starts crashing when we install VC++ runtime version 12.0.40660.0 on our windows 7 VMs. It apparently doesn't come through Windows Update and can only be installed manually: https://support.microsoft.com/en-us/topic/update-for-visual-c-2013-and-visual-c-redistributable-package-5b2ac5ab-4139-8acc-08e2-9578ec9b2cf1

[anton-potapov]

Here is the short analysis of the problem:

• global_control related C++ static objects are being initialized.
• These objects have a std::set as a member.
• std::set default constructor being called as a result.
• Different implementation of std::set might allocate memory during in the default constructor
• this std::set is instantiated tbb_allocator , which internally uses tbbmalloc
• which leads to dynamic loading of the tbbmalloc library during load of tbb library which leads to a deadlock

As a workaround, for those versions of the C++ standard library, which do allocate memory in the default constructor of std::set use of tbb_allocator might be disabled.

However, more general solution is to not use C++ static objects (as the order of initialization/de-initialization for static objects between different translation units is not defined), but instead use explicit reference counting for lifetime management

@alexey-katranov, FYI