cmssw
cmssw copied to clipboard
cms-sw
ROOT write errors in prompt reconstruction
The thread https://cms-talk.web.cern.ch/t/segmentation-fault-in-promptreco-for-jetmet-in-run-362063/17785 reports several errors in writing output files in prompt reconstruction in T0.
New categories assigned: core
@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks
A new Issue was created by @makortel Matti Kortelainen.
@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.
cms-bot commands are listed here
In CMSSW_12_4_10 first an exception
----- Begin Fatal Exception 20-Nov-2022 15:33:55 CET-----------------------
An exception of category 'FatalRootError' occurred while
[0] Calling EventProcessor::runToCompletion (which does almost everything after beginJob and before endJob)
Additional Info:
[a] Fatal Root Error: @SUB=TBufferFile::AutoExpand
Request to expand to a negative size, likely due to an integer overflow: 0x8001588e for a max of 0x7ffffffe.
----- End Fatal Exception -------------------------------------------------
followed with a segfault in the job shutdown
#3 0x00002b82f2c71a0b in sig_dostack_then_abort () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/pluginFWCoreServicesPlugins.so
#4 <signal handler called>
#5 0x00002b82eb3308b3 in TDirectoryFile::WriteKeys() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/external/el8_amd64_gcc10/lib/libRIO.so
#6 0x00002b82eb333e0d in TDirectoryFile::SaveSelf(bool) () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/external/el8_amd64_gcc10/lib/libRIO.so
#7 0x00002b82eb332307 in TDirectoryFile::Save() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/external/el8_amd64_gcc10/lib/libRIO.so
#8 0x00002b82eb3321dc in TDirectoryFile::Close(char const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/external/el8_amd64_gcc10/lib/libRIO.so
#9 0x00002b82eb34f04c in TFile::Close(char const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/external/el8_amd64_gcc10/lib/libRIO.so
#10 0x00002b82f3455e9e in TStorageFactoryFile::~TStorageFactoryFile() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libIOPoolTFileAdaptor.so
#11 0x00002b82f3455ec9 in TStorageFactoryFile::~TStorageFactoryFile() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libIOPoolTFileAdaptor.so
#12 0x00002b8370f35ada in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libIOPoolOutput.so
#13 0x00002b8370f361e7 in edm::RootOutputFile::~RootOutputFile() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libIOPoolOutput.so
#14 0x00002b8370f2e7ab in edm::PoolOutputModule::~PoolOutputModule() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libIOPoolOutput.so
#15 0x00002b8370f2eb49 in virtual thunk to edm::PoolOutputModule::~PoolOutputModule() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libIOPoolOutput.so
#16 0x00002b833472f31d in std::_Sp_counted_deleter<edm::one::OutputModuleBase*, std::default_delete<edm::one::OutputModuleBase>, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/pluginFWCoreModules.so
#17 0x00002b833472f572 in std::_Sp_counted_ptr_inplace<edm::maker::ModuleHolderT<edm::one::OutputModuleBase>, std::allocator<edm::maker::ModuleHolderT<edm::one::OutputModuleBase> >, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/pluginFWCoreModules.so
#18 0x00002b82ea994d12 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libFWCoreFramework.so
#19 0x00002b82ea994ca2 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libFWCoreFramework.so
#20 0x00002b82ea994ca2 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libFWCoreFramework.so
#21 0x00002b82ea994ca2 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libFWCoreFramework.so
#22 0x00002b82ea994ca2 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libFWCoreFramework.so
#23 0x00002b82ea9a9d59 in std::_Sp_counted_ptr<edm::ModuleRegistry*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libFWCoreFramework.so
#24 0x00002b82ea89ccc7 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libFWCoreFramework.so
#25 0x00002b82ea8dd869 in edm::Schedule::~Schedule() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libFWCoreFramework.so
#26 0x00002b82ea8c1ed7 in edm::EventProcessor::~EventProcessor() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libFWCoreFramework.so
#27 0x0000000000409b11 in (anonymous namespace)::EventProcessorWithSentry::~EventProcessorWithSentry() ()
#28 0x0000000000409729 in main ()
In CMSSW_12_4_10 first an exception
----- Begin Fatal Exception 20-Nov-2022 09:28:01 CET-----------------------
An exception of category 'FatalRootError' occurred while
[0] Calling EventProcessor::runToCompletion (which does almost everything after beginJob and before endJob)
Additional Info:
[a] Fatal Root Error: @SUB=TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
----- End Fatal Exception -------------------------------------------------
followed by a segfault in the job shutdown
#3 0x00002b1d92fc9a0b in sig_dostack_then_abort () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/pluginFWCoreServicesPlugins.so
#4 <signal handler called>
#5 0x00002b1d8b6e78b3 in TDirectoryFile::WriteKeys() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/external/el8_amd64_gcc10/lib/libRIO.so
#6 0x00002b1d8b6eae0d in TDirectoryFile::SaveSelf(bool) () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/external/el8_amd64_gcc10/lib/libRIO.so
#7 0x00002b1d8b6e9307 in TDirectoryFile::Save() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/external/el8_amd64_gcc10/lib/libRIO.so
#8 0x00002b1d8b6e91dc in TDirectoryFile::Close(char const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/external/el8_amd64_gcc10/lib/libRIO.so
#9 0x00002b1d8b70604c in TFile::Close(char const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/external/el8_amd64_gcc10/lib/libRIO.so
#10 0x00002b1d937bce9e in TStorageFactoryFile::~TStorageFactoryFile() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libIOPoolTFileAdaptor.so
#11 0x00002b1d937bcec9 in TStorageFactoryFile::~TStorageFactoryFile() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libIOPoolTFileAdaptor.so
#12 0x00002b1e089a7ada in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libIOPoolOutput.so
#13 0x00002b1e089a81e7 in edm::RootOutputFile::~RootOutputFile() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libIOPoolOutput.so
#14 0x00002b1e089a07ab in edm::PoolOutputModule::~PoolOutputModule() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libIOPoolOutput.so
#15 0x00002b1e089a0b49 in virtual thunk to edm::PoolOutputModule::~PoolOutputModule() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libIOPoolOutput.so
#16 0x00002b1dd4aa431d in std::_Sp_counted_deleter<edm::one::OutputModuleBase*, std::default_delete<edm::one::OutputModuleBase>, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/pluginFWCoreModules.so
#17 0x00002b1dd4aa4572 in std::_Sp_counted_ptr_inplace<edm::maker::ModuleHolderT<edm::one::OutputModuleBase>, std::allocator<edm::maker::ModuleHolderT<edm::one::OutputModuleBase> >, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/pluginFWCoreModules.so
#18 0x00002b1d8ad4bd12 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libFWCoreFramework.so
#19 0x00002b1d8ad4bca2 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libFWCoreFramework.so
#20 0x00002b1d8ad4bca2 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libFWCoreFramework.so
#21 0x00002b1d8ad4bca2 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libFWCoreFramework.so
#22 0x00002b1d8ad4bca2 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libFWCoreFramework.so
#23 0x00002b1d8ad60d59 in std::_Sp_counted_ptr<edm::ModuleRegistry*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libFWCoreFramework.so
#24 0x00002b1d8ac53cc7 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libFWCoreFramework.so
#25 0x00002b1d8ac94869 in edm::Schedule::~Schedule() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libFWCoreFramework.so
#26 0x00002b1d8ac78ed7 in edm::EventProcessor::~EventProcessor() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_10/lib/el8_amd64_gcc10/libFWCoreFramework.so
#27 0x0000000000409b11 in (anonymous namespace)::EventProcessorWithSentry::~EventProcessorWithSentry() ()
#28 0x0000000000409729 in main ()
In CMSSW_12_4_11_patch1 first an exception
----- Begin Fatal Exception 21-Nov-2022 06:56:31 CET-----------------------
An exception of category 'FatalRootError' occurred while
[0] Calling EventProcessor::runToCompletion (which does almost everything after beginJob and before endJob)
Additional Info:
[a] Fatal Root Error: @SUB=TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
----- End Fatal Exception -------------------------------------------------
followed by a segfault in the job shutdown
#3 0x00002b0114812a0b in sig_dostack_then_abort () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_11/lib/el8_amd64_gcc10/pluginFWCoreServicesPlugins.so
#4 <signal handler called>
#5 0x00002b010ceb08b3 in TDirectoryFile::WriteKeys() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw-patch/CMSSW_12_4_11_patch1/external/el8_amd64_gcc10/lib/libRIO.so
#6 0x00002b010ceb3e0d in TDirectoryFile::SaveSelf(bool) () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw-patch/CMSSW_12_4_11_patch1/external/el8_amd64_gcc10/lib/libRIO.so
#7 0x00002b010ceb2307 in TDirectoryFile::Save() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw-patch/CMSSW_12_4_11_patch1/external/el8_amd64_gcc10/lib/libRIO.so
#8 0x00002b010ceb21dc in TDirectoryFile::Close(char const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw-patch/CMSSW_12_4_11_patch1/external/el8_amd64_gcc10/lib/libRIO.so
#9 0x00002b010cecf04c in TFile::Close(char const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw-patch/CMSSW_12_4_11_patch1/external/el8_amd64_gcc10/lib/libRIO.so
#10 0x00002b0114c11e9e in TStorageFactoryFile::~TStorageFactoryFile() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_11/lib/el8_amd64_gcc10/libIOPoolTFileAdaptor.so
#11 0x00002b0114c11ec9 in TStorageFactoryFile::~TStorageFactoryFile() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_11/lib/el8_amd64_gcc10/libIOPoolTFileAdaptor.so
#12 0x00002b0192f41ada in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_11/lib/el8_amd64_gcc10/libIOPoolOutput.so
#13 0x00002b0192f421e7 in edm::RootOutputFile::~RootOutputFile() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_11/lib/el8_amd64_gcc10/libIOPoolOutput.so
#14 0x00002b0192f3a7ab in edm::PoolOutputModule::~PoolOutputModule() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_11/lib/el8_amd64_gcc10/libIOPoolOutput.so
#15 0x00002b0192f3ab49 in virtual thunk to edm::PoolOutputModule::~PoolOutputModule() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_11/lib/el8_amd64_gcc10/libIOPoolOutput.so
#16 0x00002b015633531d in std::_Sp_counted_deleter<edm::one::OutputModuleBase*, std::default_delete<edm::one::OutputModuleBase>, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_11/lib/el8_amd64_gcc10/pluginFWCoreModules.so
#17 0x00002b0156335572 in std::_Sp_counted_ptr_inplace<edm::maker::ModuleHolderT<edm::one::OutputModuleBase>, std::allocator<edm::maker::ModuleHolderT<edm::one::OutputModuleBase> >, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_11/lib/el8_amd64_gcc10/pluginFWCoreModules.so
#18 0x00002b010c514d12 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_11/lib/el8_amd64_gcc10/libFWCoreFramework.so
#19 0x00002b010c514ca2 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_11/lib/el8_amd64_gcc10/libFWCoreFramework.so
#20 0x00002b010c514ca2 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_11/lib/el8_amd64_gcc10/libFWCoreFramework.so
#21 0x00002b010c514ca2 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_11/lib/el8_amd64_gcc10/libFWCoreFramework.so
#22 0x00002b010c514ca2 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_11/lib/el8_amd64_gcc10/libFWCoreFramework.so
#23 0x00002b010c529d59 in std::_Sp_counted_ptr<edm::ModuleRegistry*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_11/lib/el8_amd64_gcc10/libFWCoreFramework.so
#24 0x00002b010c41ccc7 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_11/lib/el8_amd64_gcc10/libFWCoreFramework.so
#25 0x00002b010c45d869 in edm::Schedule::~Schedule() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_11/lib/el8_amd64_gcc10/libFWCoreFramework.so
#26 0x00002b010c441ed7 in edm::EventProcessor::~EventProcessor() () from /cvmfs/cms.cern.ch/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_11/lib/el8_amd64_gcc10/libFWCoreFramework.so
#27 0x0000000000409b11 in (anonymous namespace)::EventProcessorWithSentry::~EventProcessorWithSentry() ()
#28 0x0000000000409729 in main ()
@pcanal Do you have any thoughts what kind of conditions trigger these kind of errors?
Not sure if this has any significance, but the "magic number" in
[a] Fatal Root Error: @SUB=TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
is $2^{30} - 2$.
@dan131riley commented in https://cms-talk.web.cern.ch/t/segmentation-fault-in-promptreco-for-jetmet-in-run-362063/17785/4
Unfortunately, running this under valgrind is going to take forever, and we don’t have UBSAN builds for older releases.
@smuzaffar Would it be feasible to build one (?) IB in 12_4_X with UBSAN? (UBSAN build of 12_4_10 or 12_4_11_patch1 would be even better)
@makortel , I have started 12.4.UBSAN.X IB. Once working IB is available then we can think of building 12.4.10.UBSAN release
> [a] Fatal Root Error: @SUB=TBufferFile::WriteByteCount
> bytecount too large (more than 1073741822)
This indicates that one attempted to write more than 1GB in a single TBuffer/TBasket, this is not yet supported. This usually indicates that a single non split object is too large (this could be a TTree with too many branches x baskets)
I guess we should instrument the writing side as well on what exactly is being written to when these exceptions get thrown. (the exact call stack is unclear to me as the exception looks like it originates from InitRootHandles with nothing else than EventProcessor::runToCompletion() catching it in between)
@pcanal it looks like it happens during the call to TDirectoryFile::WriteKeys
@pcanal it looks like it happens during the call to
TDirectoryFile::WriteKeys
The segfault is in TDirectoryFile::WriteKeys(), but that's during shutdown after throwing a fatal error. The fatal errors come from TBuffer::AutoExpand() and TBufferFile::SetByteCount()
The segfault is in TDirectoryFile::WriteKeys(), but that's during shutdown after throwing a fatal error.
I thought it was the first error ... what is the first error?
The segfault is in TDirectoryFile::WriteKeys(), but that's during shutdown after throwing a fatal error.
I thought it was the first error ... what is the first error?
The first error in each job is the one reported by the (CMSSW) exception message, i.e.
Request to expand to a negative size, likely due to an integer overflow: 0x8001588e for a max of 0x7ffffffe.bytecount too large (more than 1073741822)
Here's the traceback up to the point where TObject::Error() gets called:
#3 0x00007fc02925c39c in TObject::Error(char const*, char const*, ...) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_4_11_patch1/external/slc7_amd64_gcc10/lib/libCore.so
#4 0x00000000004309d5 in TBufferFile::SetByteCount (this=0x7fc01add10c0, cntpos=243, packInVersion=true) at /mnt/data1/dsr/tmp/CMSSW_12_4_11_patch1/src/FWCore/Framework/bin/TBufferFile.cxx:342
#5 0x00007fc0292db2b6 in TObjArray::Streamer(TBuffer&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_4_11_patch1/external/slc7_amd64_gcc10/lib/libCore.so
#6 0x0000000000439b11 in TClass::Streamer (this=0x7fc01fe54500, obj=0x7fbf72f14e58, b=..., onfile_class=0x0) at /cvmfs/cms.cern.ch/slc7_amd64_gcc10/lcg/root/6.24.07-bf41b0420bc269850b74e23486e2953a/include/TClass.h:609
#7 0x000000000043527b in TBufferFile::WriteFastArray (this=0x7fc01add10c0, start=0x7fbf72f14e58, cl=0x7fc01fe54500, n=1, streamer=0x0) at /mnt/data1/dsr/tmp/CMSSW_12_4_11_patch1/src/FWCore/Framework/bin/TBufferFile.cxx:2255
#8 0x00007fc0298c906a in int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_4_11_patch1/external/slc7_amd64_gcc10/lib/libRIO.so
#9 0x00007fc02972cb3d in TStreamerInfoActions::GenericWriteAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_4_11_patch1/external/slc7_amd64_gcc10/lib/libRIO.so
#10 0x000000000043b29f in TStreamerInfoActions::TConfiguredAction::operator() (this=0x7fc01da69e00, buffer=..., object=0x7fbf72f14d00) at /cvmfs/cms.cern.ch/slc7_amd64_gcc10/lcg/root/6.24.07-bf41b0420bc269850b74e23486e2953a/include/TStreamerInfoActions.h:123
#11 0x000000000043911b in TBufferFile::ApplySequence (this=0x7fc01add10c0, sequence=..., obj=0x7fbf72f14d00) at /mnt/data1/dsr/tmp/CMSSW_12_4_11_patch1/src/FWCore/Framework/bin/TBufferFile.cxx:3574
#12 0x0000000000438efa in TBufferFile::WriteClassBuffer (this=0x7fc01add10c0, cl=0x7fbfe2748b80, pointer=0x7fbf72f14d00) at /mnt/data1/dsr/tmp/CMSSW_12_4_11_patch1/src/FWCore/Framework/bin/TBufferFile.cxx:3543
#13 0x00007fc029706955 in TKey::TKey(TObject const*, char const*, int, TDirectory*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_4_11_patch1/external/slc7_amd64_gcc10/lib/libRIO.so
#14 0x00007fc0296c65a5 in TFile::CreateKey(TDirectory*, TObject const*, char const*, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_4_11_patch1/external/slc7_amd64_gcc10/lib/libRIO.so
#15 0x0000000000442f37 in TDirectoryFile::WriteTObject (this=0x7fbfa3915d00, obj=0x7fbf72f14d00, name=0x0, option=0x7fc029e12575 "", bufsize=0) at /mnt/data1/dsr/tmp/CMSSW_12_4_11_patch1/src/FWCore/Framework/bin/TDirectoryFile.cxx:1918
#16 0x00007fc029dfa85f in TTree::AutoSave(char const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_4_11_patch1/external/slc7_amd64_gcc10/lib/libTree.so
#17 0x00007fbf9f29b762 in edm::RootOutputTree::writeTTree (tree=0x7fbf72f14d00) at /mnt/data1/dsr/tmp/CMSSW_12_4_11_patch1/src/IOPool/Output/src/RootOutputTree.cc:328
#18 0x00007fbf9f29b7da in edm::RootOutputTree::writeTree (this=0x7fbf7c345e18) at /mnt/data1/dsr/tmp/CMSSW_12_4_11_patch1/src/IOPool/Output/src/RootOutputTree.cc:335
#19 0x00007fbf9f286d77 in edm::RootOutputFile::finishEndFile (this=0x7fbf7c345b00) at /mnt/data1/dsr/tmp/CMSSW_12_4_11_patch1/src/IOPool/Output/src/RootOutputFile.cc:782
#20 0x00007fbf9f23e19e in edm::PoolOutputModule::finishEndFile (this=0x7fbfa2b8f800) at /mnt/data1/dsr/tmp/CMSSW_12_4_11_patch1/src/IOPool/Output/src/PoolOutputModule.cc:354
#21 0x00007fbf9f23df24 in edm::PoolOutputModule::reallyCloseFile (this=0x7fbfa2b8f800) at /mnt/data1/dsr/tmp/CMSSW_12_4_11_patch1/src/IOPool/Output/src/PoolOutputModule.cc:330
so it starts from https://github.com/cms-sw/cmssw/blob/e5183acce9265b4f9066810496bd529269b31caf/IOPool/Output/src/RootOutputTree.cc#L328
In TBufferFile::SetByteCount(), cnt is 1412876593. I haven't figured out yet what class is being written, keep getting <incomplete type>, so I need some more debugging symbols.
Thanks @dan131riley. In RootOutputFile::finishEndFile(), would you be able to tell which of the treePointers_[i] was called?
https://github.com/cms-sw/cmssw/blob/e5183acce9265b4f9066810496bd529269b31caf/IOPool/Output/src/RootOutputFile.cc#L782
Thanks @dan131riley. In
RootOutputFile::finishEndFile(), would you be able to tell which of thetreePointers_[i]was called?
Appears to be the "Events" tree.
Do we know the number of branches and entries (or better yet, can we call Events->Print("") before the crash)? The maximum number of baskets per TTree (to stay under the 1GB) is 50 millions.
I don't yet have a clean Events->Print(""), but I can say that the number of baskets is a few hundred, definitely not 50 million. Two pieces of evidence also suggest that it isn't so simple: (1) crash goes away if some number of events are skipped, and (2) I get different behavior reading a local file.
Specifically, reading a local file instead of via xrootd, the process does not segfault, but it hangs on exit waiting on a lock:
Thread 1 (Thread 0x7f28da90b740 (LWP 135820) "cmsRun"):
#0 0x00007f28dca9554d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f28dca9290e in _L_lock_39 () from /lib64/libpthread.so.0
#2 0x00007f28dca92848 in pthread_cond_destroy@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#3 0x00007f28dead0138 in ROOT::Internal::RConcurrentHashColl::~RConcurrentHashColl() () from /cvmfs/cms.cern.ch/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_4_11_patch1/external/slc7_amd64_gcc10/lib/libThread.so
#4 0x00007f28dc6f30aa in __cxa_finalize () from /lib64/libc.so.6
#5 0x00007f28de55dff3 in __do_global_dtors_aux () from /cvmfs/cms.cern.ch/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_4_11_patch1/external/slc7_amd64_gcc10/lib/ibRIO.so
#6 0x00007fffbe662d10 in ?? ()
#7 0x00007f28df63608a in _dl_fini () from /lib64/ld-linux-x86-64.so.2
All the other threads are in TBB idle-wait states, so it appears that a lock is getting lost?
The combination of the two pretty strongly suggests a memory overwrite or logic error. I'm running a few more tests. If they don't reveal anything I'll try valgrind (which may take several days).
@dan131riley on the local file usage, could you try running in GDB and see if an exception happens? Maybe there is a lock taken explicitly and not being released?
@dan131riley purely as a guess, I think pthread_cond_destroy is probably coming from the destruction of the std::condition_variable_any member variable in TRWSpinLock which is being deleted because of the std::unique_ptr<TRWSpinLock> within RConcurrentHashColl.
If we aren't dealing with an exception safety problem (where some code explicitly called Lock without using the helper classes) then maybe we are having a double delete? If the memory was already freed and something wrote over the address the second call to delete could think the condition is still being held somewhere even though it actually isn't .
Looking at a full log with the local input file, there is an exception
----- Begin Fatal Exception 28-Nov-2022 16:11:41 EST-----------------------
An exception of category 'FatalRootError' occurred while
[0] Calling EventProcessor::runToCompletion (which does almost everything after beginJob and before endJob)
Additional Info:
[a] Fatal Root Error: @SUB=TBufferFile::WriteByteCount
bytecount too large (more than 1073741822) 0x7ef9ed05c491 0x7ef998800640 195
----- End Fatal Exception -------------------------------------------------
That's after the input file is closed, and probably at the point where tree->Print("") is called, since that seems to trigger some kind of cleanup. However, with the local input file the exception isn't followed by a segfault, instead it just deadlocks in the global destructors.
So it seems there is 2 slightly distinct problem. The second one is a problem in the error recovery that lead to either the dead lock or the crash. The first is one is why it tries to write such a large object.
Looking at a full log with the local input file
Since it uses a local input, I should be able to reproduce. Can you give me the instruction on how to reproduce it (preferably with a ROOT build in debug mode)? Thanks.
To reproduce , set up a CMSSW_12_4_11_patch1 area and use cmsRun /uscms_data/d2/dsr/issue-40132/crash1.py. However, this will run up against the LPC node virtual memory limits, so you'll need a system with higher memory limits.
With an added call to TTree::Print(""), The traceback for the error looks like:
#4 0x00007fd798c7439c in TObject::Error(char const*, char const*, ...) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_4_11_patch1/external/slc7_amd64_gcc10/lib/libCore.so
#5 0x000000000043ec73 in TBufferFile::SetByteCount (this=0x7fff86514fa0, cntpos=195, packInVersion=true) at /mnt/data1/dsr/tmp/CMSSW_12_4_11_patch1/src/FWCore/Framework/bin/TBufferFile.cxx:342
#6 0x0000000000476a60 in TObjArray::Streamer (this=0x7fd6c502ad58, b=...) at /mnt/data1/dsr/tmp/CMSSW_12_4_11_patch1/src/FWCore/Framework/bin/TObjArray.cxx:489
#7 0x0000000000447de1 in TClass::Streamer (this=0x7fd7911d6880, obj=0x7fd6c502ad58, b=..., onfile_class=0x0) at /cvmfs/cms.cern.ch/slc7_amd64_gcc10/lcg/root/6.24.07-bf41b0420bc269850b74e23486e2953a/include/TClass.h:609
#8 0x000000000044351d in TBufferFile::WriteFastArray (this=0x7fff86514fa0, start=0x7fd6c502ad58, cl=0x7fd7911d6880, n=1, streamer=0x0) at /mnt/data1/dsr/tmp/CMSSW_12_4_11_patch1/src/FWCore/Framework/bin/TBufferFile.cxx:2255
#9 0x00007fd7992e106a in int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_4_11_patch1/external/slc7_amd64_gcc10/lib/libRIO.so
#10 0x00007fd799144b3d in TStreamerInfoActions::GenericWriteAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_4_11_patch1/external/slc7_amd64_gcc10/lib/libRIO.so
#11 0x000000000044956f in TStreamerInfoActions::TConfiguredAction::operator() (this=0x7fd7880d4800, buffer=..., object=0x7fd6c502ac00) at /cvmfs/cms.cern.ch/slc7_amd64_gcc10/lcg/root/6.24.07-bf41b0420bc269850b74e23486e2953a/include/TStreamerInfoActions.h:123
#12 0x000000000044737c in TBufferFile::ApplySequence (this=0x7fff86514fa0, sequence=..., obj=0x7fd6c502ac00) at /mnt/data1/dsr/tmp/CMSSW_12_4_11_patch1/src/FWCore/Framework/bin/TBufferFile.cxx:3568
#13 0x00000000004471c9 in TBufferFile::WriteClassBuffer (this=0x7fff86514fa0, cl=0x7fd786dd1880, pointer=0x7fd6c502ac00) at /mnt/data1/dsr/tmp/CMSSW_12_4_11_patch1/src/FWCore/Framework/bin/TBufferFile.cxx:3546
#14 0x00007fd798d2d1c6 in TClass::WriteBuffer(TBuffer&, void*, char const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_4_11_patch1/external/slc7_amd64_gcc10/lib/libCore.so
#15 0x00007fd79981cbb7 in TTree::Print(char const*) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_4_11_patch1/external/slc7_amd64_gcc10/lib/libTree.so
#16 0x00007fd6d9399e23 in edm::RootOutputFile::finishEndFile() () from /mnt/data1/dsr/tmp/CMSSW_12_4_11_patch1/lib/slc7_amd64_gcc10/libIOPoolOutput.so
#17 0x00007fd6d935199e in edm::PoolOutputModule::finishEndFile (this=0x7fd740101000) at /mnt/data1/dsr/tmp/CMSSW_12_4_11_patch1/src/IOPool/Output/src/PoolOutputModule.cc:354
#18 0x00007fd6d9351724 in edm::PoolOutputModule::reallyCloseFile (this=0x7fd740101000) at /mnt/data1/dsr/tmp/CMSSW_12_4_11_patch1/src/IOPool/Output/src/PoolOutputModule.cc:330
Looking at the underlying TBuffer I get
(gdb) p *(TBuffer*)this
$13 = {
<TObject> = {<No data fields>},
members of TBuffer:
fMode = true,
fVersion = 0,
fBufSize = 2147483638,
fBuffer = 0x7fcfc1000940 "",
fBufCur = 0x7fd015880146 "",
fBufMax = 0x7fd041000936 "",
fParent = 0x0,
fReAllocFunc = 0x7fd798c8cb80 <TStorage::ReAllocChar(char*, unsigned long, unsigned long)>,
fCacheStack = std::vector of length 0, capacity 0,
static fgIsA = {
_M_b = {
_M_p = 0x7fd738baa980
},
static is_always_lock_free = <optimized out>
}
}
In a UBSAN build, I do get a signed integer overflow runtime error:
/mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:2075:41: runtime error: signed integer overflow: 88208 + 2147483638 cannot be represented in type 'int'
#0 0x49d624 in TBufferFile::WriteFastArray(long long const*, int) /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:2075
#1 0x7fe5513dd1ab in int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) [clone .localalias] (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libRIO.so+0x3481ab)
#2 0x7fe55123fb3c in TStreamerInfoActions::GenericWriteAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libRIO.so+0x1aab3c)
#3 0x4d52ca in TStreamerInfoActions::TConfiguredAction::operator()(TBuffer&, void*) const (/mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/bin/el8_amd64_gcc10/cmsRun+0x4d52ca)
#4 0x4c5479 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:3574
#5 0x4c44e9 in TBufferFile::WriteClassBuffer(TClass const*, void*) /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:3543
#6 0x7fe55189de9d in TBranch::Streamer(TBuffer&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libTree.so+0xb9e9d)
#7 0x7fe550e6671a in TStreamerBase::WriteBuffer(TBuffer&, char*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libCore.so+0x2d171a)
#8 0x7fe5513de90c in int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) [clone .localalias] (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libRIO.so+0x34990c)
#9 0x7fe55123fb3c in TStreamerInfoActions::GenericWriteAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libRIO.so+0x1aab3c)
#10 0x4d52ca in TStreamerInfoActions::TConfiguredAction::operator()(TBuffer&, void*) const (/mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/bin/el8_amd64_gcc10/cmsRun+0x4d52ca)
#11 0x4c5479 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:3574
#12 0x4c44e9 in TBufferFile::WriteClassBuffer(TClass const*, void*) /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:3543
#13 0x7fe5518b7f76 in TBranchElement::Streamer(TBuffer&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libTree.so+0xd3f76)
#14 0x4c9ff7 in TClass::Streamer(void*, TBuffer&, TClass const*) const (/mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/bin/el8_amd64_gcc10/cmsRun+0x4c9ff7)
#15 0x4a816c in TBufferFile::WriteObjectClass(void const*, TClass const*, bool) /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:2551
#16 0x7fe55117d393 in TBufferIO::WriteObjectAny(void const*, TClass const*,bool) [clone .localalias] (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libRIO.so+0xe8393)
#17 0x7fe550dee1fc in TObjArray::Streamer(TBuffer&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libCore.so+0x2591fc)
#18 0x4c9ff7 in TClass::Streamer(void*, TBuffer&, TClass const*) const (/mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/bin/el8_amd64_gcc10/cmsRun+0x4c9ff7)
#19 0x4a0581 in TBufferFile::WriteFastArray(void*, TClass const*, int, TMemberStreamer*) /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:2255
#20 0x7fe5513dc069 in int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) [clone .localalias] (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libRIO.so+0x347069)
#21 0x7fe55123fb3c in TStreamerInfoActions::GenericWriteAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libRIO.so+0x1aab3c)
#22 0x4d52ca in TStreamerInfoActions::TConfiguredAction::operator()(TBuffer&, void*) const (/mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/bin/el8_amd64_gcc10/cmsRun+0x4d52ca)
#23 0x4c5479 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:3574
#24 0x4c44e9 in TBufferFile::WriteClassBuffer(TClass const*, void*) /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:3543
#25 0x7fe55189de9d in TBranch::Streamer(TBuffer&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libTree.so+0xb9e9d)
#26 0x7fe550e6671a in TStreamerBase::WriteBuffer(TBuffer&, char*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libCore.so+0x2d171a)
#27 0x7fe5513de90c in int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) [clone .localalias] (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libRIO.so+0x34990c)
#28 0x7fe55123fb3c in TStreamerInfoActions::GenericWriteAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libRIO.so+0x1aab3c)
#29 0x4d52ca in TStreamerInfoActions::TConfiguredAction::operator()(TBuffer&, void*) const (/mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/bin/el8_amd64_gcc10/cmsRun+0x4d52ca)
#30 0x4c5479 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:3574
#31 0x4c44e9 in TBufferFile::WriteClassBuffer(TClass const*, void*) /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:3543
#32 0x7fe5518b7f76 in TBranchElement::Streamer(TBuffer&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libTree.so+0xd3f76)
#33 0x4c9ff7 in TClass::Streamer(void*, TBuffer&, TClass const*) const (/mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/bin/el8_amd64_gcc10/cmsRun+0x4c9ff7)
#34 0x4a816c in TBufferFile::WriteObjectClass(void const*, TClass const*, bool) /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:2551
#35 0x7fe55117d393 in TBufferIO::WriteObjectAny(void const*, TClass const*,bool) [clone .localalias] (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libRIO.so+0xe8393)
#36 0x7fe550dee1fc in TObjArray::Streamer(TBuffer&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libCore.so+0x2591fc)
#37 0x4c9ff7 in TClass::Streamer(void*, TBuffer&, TClass const*) const (/mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/bin/el8_amd64_gcc10/cmsRun+0x4c9ff7)
#38 0x4a0581 in TBufferFile::WriteFastArray(void*, TClass const*, int, TMemberStreamer*) /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:2255
#39 0x7fe5513dc069 in int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) [clone .localalias] (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libRIO.so+0x347069)
#40 0x7fe55123fb3c in TStreamerInfoActions::GenericWriteAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libRIO.so+0x1aab3c)
#41 0x4d52ca in TStreamerInfoActions::TConfiguredAction::operator()(TBuffer&, void*) const (/mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/bin/el8_amd64_gcc10/cmsRun+0x4d52ca)
#42 0x4c5479 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:3574
#43 0x4c44e9 in TBufferFile::WriteClassBuffer(TClass const*, void*) /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:3543
#44 0x7fe55189de9d in TBranch::Streamer(TBuffer&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libTree.so+0xb9e9d)
#45 0x7fe550e6671a in TStreamerBase::WriteBuffer(TBuffer&, char*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libCore.so+0x2d171a)
#46 0x7fe5513de90c in int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) [clone .localalias] (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libRIO.so+0x34990c)
#47 0x7fe55123fb3c in TStreamerInfoActions::GenericWriteAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libRIO.so+0x1aab3c)
#48 0x4d52ca in TStreamerInfoActions::TConfiguredAction::operator()(TBuffer&, void*) const (/mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/bin/el8_amd64_gcc10/cmsRun+0x4d52ca)
#49 0x4c5479 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:3574
#50 0x4c44e9 in TBufferFile::WriteClassBuffer(TClass const*, void*) /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:3543
#51 0x7fe5518b7f76 in TBranchElement::Streamer(TBuffer&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libTree.so+0xd3f76)
#52 0x4c9ff7 in TClass::Streamer(void*, TBuffer&, TClass const*) const (/mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/bin/el8_amd64_gcc10/cmsRun+0x4c9ff7)
#53 0x4a816c in TBufferFile::WriteObjectClass(void const*, TClass const*, bool) /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:2551
#54 0x7fe55117d393 in TBufferIO::WriteObjectAny(void const*, TClass const*,bool) [clone .localalias] (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libRIO.so+0xe8393)
#55 0x7fe550dee1fc in TObjArray::Streamer(TBuffer&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libCore.so+0x2591fc)
#56 0x4c9ff7 in TClass::Streamer(void*, TBuffer&, TClass const*) const (/mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/bin/el8_amd64_gcc10/cmsRun+0x4c9ff7)
#57 0x4a0581 in TBufferFile::WriteFastArray(void*, TClass const*, int, TMemberStreamer*) /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:2255
#58 0x7fe5513dc069 in int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) [clone .localalias] (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libRIO.so+0x347069)
#59 0x7fe55123fb3c in TStreamerInfoActions::GenericWriteAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libRIO.so+0x1aab3c)
#60 0x4d52ca in TStreamerInfoActions::TConfiguredAction::operator()(TBuffer&, void*) const (/mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/bin/el8_amd64_gcc10/cmsRun+0x4d52ca)
#61 0x4c5479 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:3574
#62 0x4c44e9 in TBufferFile::WriteClassBuffer(TClass const*, void*) /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/bin/TBufferFile.cxx:3543
#63 0x7fe550e28185 in TClass::WriteBuffer(TBuffer&, void*, char const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libCore.so+0x293185)
#64 0x7fe551917b86 in TTree::Print(char const*) const (/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_UBSAN_X_2022-11-22-2300/external/el8_amd64_gcc10/lib/libTree.so+0x133b86)
#65 0x7fe3c422c41b (/mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/lib/el8_amd64_gcc10/libIOPoolOutput.so+0x27e41b)
#66 0x7fe3c412ec2d (/mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/lib/el8_amd64_gcc10/libIOPoolOutput.so+0x180c2d)
#67 0x7fe3c412e02a (/mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/lib/el8_amd64_gcc10/libIOPoolOutput.so+0x18002a)
#68 0x7fe5565e160c in edm::core::OutputModuleCore::doCloseFile() /mnt/data1/dsr/tmp/CMSSW_12_4_UBSAN_X_2022-11-22-2300/src/FWCore/Framework/src/OutputModuleCore.cc:361