Global TfToken destructors causing hang on shutdown in standalone program.

[jjzanin-SMS]

Hello,

I have a standalone C++ program that uses the Maya library to export to USD. Frequently when the program exits, it hangs, and when I attach a debugger to it, it's attempting to acquire a mutex in the Tf_TokenRegistry when destructing global TfTokens. It seems like using plain global TfTokens are a no-no in general in USD as TF_DEFINE_PUBLIC_TOKENS and TF_DEFINE_PRIVATE_TOKENS use TfStaticData, which calls new on first-use and never calls delete so the destructor is never called. I can see which variable is being destructed and have fixed a few but I don't know how deep the rabbit hole goes. Some examples of global TfTokens that have caused a hang:

https://github.com/Autodesk/maya-usd/blob/f8c1d931d015fe070bad6ba0fceab0a180f4075e/lib/mayaUsd/fileio/shading/shadingModePxrRis.cpp#L83 https://github.com/Autodesk/maya-usd/blob/f8c1d931d015fe070bad6ba0fceab0a180f4075e/lib/usd/translators/particleWriter.cpp#L136 https://github.com/Autodesk/maya-usd/blob/f8c1d931d015fe070bad6ba0fceab0a180f4075e/lib/mayaUsd/ufe/UsdTransform3dMayaXformStack.cpp#L76

From what I have seen, there are multiple TBB threads destructing global TfTokens, grabbing the mutex and my guess is that a thread grabs a mutex and then is killed somehow without releasing the mutex, causing a hang when the next destructor is called. Is this a real problem or am I missing something?

Example callstack:

ntdll.dll!NtYieldExecution()	Unknown
KernelBase.dll!SwitchToThread()	Unknown
pxr_tf.dll!tbb::internal::atomic_backoff::pause() Line 369	C++
pxr_tf.dll!__TBB_LockByte(unsigned char & flag) Line 921	C++
pxr_tf.dll!pxrInternal_v0_23__pxrReserved__::Tf_TokenRegistry::_PossiblyDestroyRep(const pxrInternal_v0_23__pxrReserved__::TfToken::_Rep * rep) Line 149	C++
pxr_tf.dll!pxrInternal_v0_23__pxrReserved__::TfToken::_PossiblyDestroyRep() Line 279	C++
mayaUsd.dll!pxrInternal_v0_23__pxrReserved__::TfToken::_RemoveRef() Line 339	C++
mayaUsd.dll!std::_List_node<std::pair<pxrInternal_v0_23__pxrReserved__::TfToken const ,enum MayaUsd_v0::ufe::UsdTransform3dMayaXformStack::OpNdx>,void *>::_Freenode<std::allocator<std::_List_node<std::pair<pxrInternal_v0_23__pxrReserved__::TfToken const ,enum MayaUsd_v0::ufe::UsdTransform3dMayaXformStack::OpNdx>,void *>>>(std::allocator<std::_List_node<std::pair<pxrInternal_v0_23__pxrReserved__::TfToken const ,enum MayaUsd_v0::ufe::UsdTransform3dMayaXformStack::OpNdx>,void *>> & _Al, std::_List_node<std::pair<pxrInternal_v0_23__pxrReserved__::TfToken const ,enum MayaUsd_v0::ufe::UsdTransform3dMayaXformStack::OpNdx>,void *> * _Ptr) Line 317	C++
mayaUsd.dll!std::list<std::pair<pxrInternal_v0_23__pxrReserved__::TfToken const ,enum MayaUsd_v0::ufe::UsdTransform3dMayaXformStack::OpNdx>,std::allocator<std::pair<pxrInternal_v0_23__pxrReserved__::TfToken const ,enum MayaUsd_v0::ufe::UsdTransform3dMayaXformStack::OpNdx>>>::_Tidy() Line 1500	C++
mayaUsd.dll!`anonymous namespace'::`dynamic atexit destructor for 'gOpNameToNdx''()	C++
ucrtbase.dll!<lambda>(void)()	Unknown
ucrtbase.dll!__crt_seh_guarded_call<int>::operator()<<lambda_7777bce6b2f8c936911f934f8298dc43>,<lambda>(void) &,<lambda_3883c3dff614d5e0c5f61bb1ac94921c>>()	Unknown
ucrtbase.dll!_execute_onexit_table()	Unknown
mayaUsd.dll!dllmain_crt_process_detach(const bool is_terminating) Line 182	C++
mayaUsd.dll!dllmain_dispatch(HINSTANCE__ * const instance, const unsigned long reason, void * const reserved) Line 293	C++
ntdll.dll!LdrpCallInitRoutine()	Unknown
ntdll.dll!LdrShutdownProcess()	Unknown
ntdll.dll!RtlExitUserProcess()	Unknown
kernel32.dll!ExitProcessImplementation()	Unknown
ucrtbase.dll!exit_or_terminate_process()	Unknown
ucrtbase.dll!common_exit()	Unknown

Specs

• Windows 10
• Visual Studio 2022 17.8.4
• Maya 2023
• Maya USD 0.23.0 (built by us)
• Pixar USD 23.05 (built by us)