cmssw icon indicating copy to clipboard operation
cmssw copied to clipboard
verified publisher icon cms-sw

RelVal 11846.19: SIGSEGV in `TIsAProxy::operator()`

Open iarspider opened this issue 4 weeks ago • 8 comments

In CMSSW_16_0_X_2025-12-08-2300/el9_amd64_gcc13, RelVal 11846.19 crashed with SIGSEGV:

Thread 21 (Thread 0x14bb7aaa1640 (LWP 2177910) "cmsRun"):
#0  0x000014bbcdefa0bf in poll () from /lib64/libc.so.6
#1  0x000014bbc9f86b5e in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/pluginFWCoreServicesPlugins.so
#2  0x000014bbc9f86d63 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014bbce737e18 in TIsAProxy::operator()(void const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/external/el9_amd64_gcc13/lib/libCore.so
#5  0x000014bbc8328d23 in TBufferIO::WriteObjectAny(void const*, TClass const*, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/external/el9_amd64_gcc13/lib/libRIO.so
#6  0x000014bbc83a756b in TGenCollectionStreamer::WriteObjects(int, TBuffer&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/external/el9_amd64_gcc13/lib/libRIO.so
#7  0x000014bbc836c574 in TCollectionStreamer::Streamer(TBuffer&, void*, int, TClass*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/external/el9_amd64_gcc13/lib/libRIO.so
#8  0x000014bbc831c5aa in TBufferFile::WriteFastArray(void*, TClass const*, long long, TMemberStreamer*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/external/el9_amd64_gcc13/lib/libRIO.so
#9  0x000014bbc85259a0 in int TStreamerInfoActions::WriteSTL<&TStreamerInfoActions::WriteSTLMemberWise, &TStreamerInfoActions::WriteSTLObjectWiseFastArray>(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/external/el9_amd64_gcc13/lib/libRIO.so
#10 0x000014bbc83226de in TBufferFile::WriteClassBuffer(TClass const*, void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/external/el9_amd64_gcc13/lib/libRIO.so
#11 0x000014bbc842dc54 in TStreamerInfoActions::WriteViaClassBuffer(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/external/el9_amd64_gcc13/lib/libRIO.so
#12 0x000014bbc831b3c5 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/external/el9_amd64_gcc13/lib/libRIO.so
#13 0x000014bbc8f1bf7e in TBranch::FillImpl(ROOT::Internal::TBranchIMTHelper*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/external/el9_amd64_gcc13/lib/libTree.so
#14 0x000014bbc8f26dfc in TBranchElement::FillImpl(ROOT::Internal::TBranchIMTHelper*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/external/el9_amd64_gcc13/lib/libTree.so
#15 0x000014bbc8f26b61 in TBranchElement::FillImpl(ROOT::Internal::TBranchIMTHelper*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/external/el9_amd64_gcc13/lib/libTree.so
#16 0x000014bbc8f9d9b9 in TTree::Fill() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/external/el9_amd64_gcc13/lib/libTree.so
#17 0x000014bb26c82b38 in tbb::detail::d1::task_arena_function<edm::RootOutputTree::fillTree()::{lambda()#1}, void>::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libIOPoolOutput.so
#18 0x000014bbce2c91c1 in operator() (__closure=<optimized out>) at src/tbb/arena.cpp:944
#19 tbb::detail::d0::try_call_proxy<tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, intptr_t)::<lambda()> >::on_completion<tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, intptr_t)::<lambda()> > (on_completion_body=..., this=<optimized out>) at src/tbb/../../include/oneapi/tbb/detail/_template_helpers.h:234
#20 tbb::detail::r1::isolate_within_arena (d=..., isolation=<optimized out>) at src/tbb/arena.cpp:945
#21 0x000014bb26c8570a in edm::RootOutputTree::fillTree() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libIOPoolOutput.so
#22 0x000014bb26c85df2 in edm::RootOutputFile::fillBranches(edm::BranchType const&, edm::OccurrenceForOutput const&, unsigned int, std::vector<edm::StoredProductProvenance, std::allocator<edm::StoredProductProvenance> >*, edm::ProductProvenanceRetriever const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libIOPoolOutput.so
#23 0x000014bb26c8643f in edm::RootOutputFile::writeOne(edm::EventForOutput const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libIOPoolOutput.so
#24 0x000014bb26c6f853 in edm::PoolOutputModule::write(edm::EventForOutput const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libIOPoolOutput.so
#25 0x000014bbcebcfa23 in edm::core::OutputModuleCore::doEvent_(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so
#26 0x000014bbcec51ac1 in edm::one::OutputModuleBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so
#27 0x000014bbcec43137 in edm::WorkerT<edm::one::OutputModuleBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so
#28 0x000014bbcebcfe82 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::TransitionActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::TransitionActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::TransitionActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so
#29 0x000014bbcebd1e34 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::TransitionActionType)1> >::execute()::{lambda()#1}&>(tbb::detail::d2::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::TransitionActionType)1> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so
#30 0x000014bbce9cd427 in tbb::detail::d2::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreConcurrency.so
#31 0x000014bbce2e119b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=<optimized out>, waiter=..., this=0x14bbccfcf980) at src/tbb/task_dispatcher.h:344
#32 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14bbccfcf980) at src/tbb/task_dispatcher.h:487
#33 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at src/tbb/arena.cpp:216
#34 tbb::detail::r1::thread_dispatcher_client::process (td=..., this=<optimized out>) at src/tbb/thread_dispatcher_client.h:41
#35 tbb::detail::r1::thread_dispatcher::process (this=<optimized out>, j=...) at src/tbb/thread_dispatcher.cpp:195
#36 0x000014bbce2d4c75 in tbb::detail::r1::rml::private_worker::run (this=0x14bbca4c7080) at src/tbb/private_server.cpp:271
#37 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14bbca4c7080) at src/tbb/private_server.cpp:221
#38 0x000014bbcde822ea in start_thread () from /lib64/libc.so.6
#39 0x000014bbcdf073c0 in clone3 () from /lib64/libc.so.6

iarspider avatar Dec 09 '25 09:12 iarspider

cms-bot internal usage

cmsbuild avatar Dec 09 '25 09:12 cmsbuild

A new Issue was created by @iarspider.

@Dr15Jones, @ftenchini, @makortel, @mandrenguyen, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

cmsbuild avatar Dec 09 '25 09:12 cmsbuild

Other non-idle threads:

Thread 1 (Thread 0x14bbcda3b540 (LWP 2177882) "cmsRun"):
#0  0x000014bbcdecc795 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1  0x000014bbcded1247 in nanosleep () from /lib64/libc.so.6
#2  0x000014bbcded117e in sleep () from /lib64/libc.so.6
#3  0x000014bbc9f84a5c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000014bbce39d863 in sizedDeleteImpl (size=32, ptr=0x14b8c80c4080) at src/jemalloc_cpp.cpp:192
#6  operator delete (ptr=0x14b8c80c4080, size=32) at src/jemalloc_cpp.cpp:200
#7  0x000014bab59ff317 in ClusterTPAssociationProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/pluginSimTrackerTrackerHitAssociationPlugins.so
#8  0x000014bbcec4ae71 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so
#9  0x000014bbcec43357 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so
#10 0x000014bbcebcfe82 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::TransitionActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::TransitionActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::TransitionActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so
#11 0x000014bbcebd0394 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::TransitionActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so
#12 0x000014bbce9cae64 in tbb::detail::d2::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreConcurrency.so
#13 0x000014bbce2d9243 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x14baaaabb400, this=<optimized out>) at src/tbb/task_dispatcher.h:344
#14 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=<optimized out>) at src/tbb/task_dispatcher.h:487
#15 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at src/tbb/task_dispatcher.cpp:169
#16 0x000014bbceb4a02f in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so
#17 0x000014bbceb5f27b in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so
#18 0x000014bbceb591e1 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so
#19 0x00000000004091b3 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#20 0x000014bbce2c8832 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at src/tbb/arena.cpp:860
#21 0x000000000040ac27 in main::{lambda()#1}::operator()() const ()
#22 0x0000000000405288 in main ()

Thread 20 (Thread 0x14bb7a0a0640 (LWP 2177911) "cmsRun"):
#0  0x000014bbcdecc795 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1  0x000014bbcded1247 in nanosleep () from /lib64/libc.so.6
#2  0x000014bbcded117e in sleep () from /lib64/libc.so.6
#3  0x000014bbc9f84a5c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000014bbc92e2f04 in TH1::Fill(double, double) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/external/el9_amd64_gcc13/lib/libHist.so
#6  0x000014bbca64ea70 in dqm::impl::MonitorElement::Fill(double) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libDQMServicesCore.so
#7  0x000014bb4f4751f4 in HistogramManager::fill(double, double, DetId, edm::Event const*, int, int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libDQMSiPixelPhase1Common.so
#8  0x000014bb313ea9d7 in (anonymous namespace)::SiPixelPhase1Clusters::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/pluginDQMSiPixelPhase1CommonAuto.so
#9  0x000014bbcec5cbf2 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so
#10 0x000014bbcec43af7 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so
#11 0x000014bbcebcfe82 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::TransitionActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::TransitionActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::TransitionActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so
#12 0x000014bbcebd0394 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::TransitionActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so
#13 0x000014bbce9cae64 in tbb::detail::d2::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreConcurrency.so
#14 0x000014bbce2e119b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=<optimized out>, waiter=..., this=0x14bbccfcfa80) at src/tbb/task_dispatcher.h:344
#15 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14bbccfcfa80) at src/tbb/task_dispatcher.h:487
#16 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at src/tbb/arena.cpp:216
#17 tbb::detail::r1::thread_dispatcher_client::process (td=..., this=<optimized out>) at src/tbb/thread_dispatcher_client.h:41
#18 tbb::detail::r1::thread_dispatcher::process (this=<optimized out>, j=...) at src/tbb/thread_dispatcher.cpp:195
#19 0x000014bbce2d4c75 in tbb::detail::r1::rml::private_worker::run (this=0x14bbca4c7100) at src/tbb/private_server.cpp:271
#20 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14bbca4c7100) at src/tbb/private_server.cpp:221
#21 0x000014bbcde822ea in start_thread () from /lib64/libc.so.6
#22 0x000014bbcdf073c0 in clone3 () from /lib64/libc.so.6

Thread 19 (Thread 0x14bb78fff640 (LWP 2177912) "cmsRun"):
#0  0x000014bbcdecc795 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1  0x000014bbcded1247 in nanosleep () from /lib64/libc.so.6
#2  0x000014bbcded117e in sleep () from /lib64/libc.so.6
#3  0x000014bbc9f84a5c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000014bbcde35f13 in modff () from /lib64/libc.so.6
#6  0x000014bb9ea8655e in LinearGridInterpolator3D::interpolate(float, float, float) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libMagneticFieldInterpolation.so
#7  0x000014bb9ea87a99 in RectangularCylindricalMFGrid::uncheckedValueInTesla(Point3DBase<float, LocalTag> const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libMagneticFieldInterpolation.so
#8  0x000014bb9ea86c47 in MFGrid3D::valueInTesla(Point3DBase<float, LocalTag> const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libMagneticFieldInterpolation.so
#9  0x000014bba5d3afe0 in MagVolume::fieldInTesla(Point3DBase<float, GlobalTag> const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libMagneticFieldVolumeGeometry.so
#10 0x000014bba8c18e39 in MagGeometry::fieldInTesla(Point3DBase<float, GlobalTag> const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libMagneticFieldVolumeBasedEngine.so
#11 0x000014bba8c19510 in VolumeBasedMagneticField::inTesla(Point3DBase<float, GlobalTag> const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libMagneticFieldVolumeBasedEngine.so
#12 0x000014bbc5c881b9 in TwoTrackMinimumDistanceHelixLine::updateCoeffs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libTrackingToolsPatternTools.so
#13 0x000014bbc5c88d3d in TwoTrackMinimumDistanceHelixLine::calculate(GlobalTrajectoryParameters const&, GlobalTrajectoryParameters const&, float) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libTrackingToolsPatternTools.so
#14 0x000014bbc5c8a004 in TwoTrackMinimumDistance::calculate(GlobalTrajectoryParameters const&, GlobalTrajectoryParameters const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libTrackingToolsPatternTools.so
#15 0x000014bbc5c8a2ef in TSCBLBuilderNoMaterial::operator()(FreeTrajectoryState const&, reco::BeamSpot const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libTrackingToolsPatternTools.so
#16 0x000014bbc5c8b1fc in reco::trackingParametersAtClosestApproachToBeamSpot(Basic3DVector<double> const&, Basic3DVector<double> const&, float, MagneticField const&, reco::BeamSpot const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libTrackingToolsPatternTools.so
#17 0x000014bb4ff105e3 in TrackAssociatorByChi2Impl::associateRecoToSim(edm::RefToBaseVector<reco::Track> const&, edm::RefVector<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle, edm::refhelper::FindUsingAdvance<std::vector<TrackingParticle, std::allocator<TrackingParticle> >, TrackingParticle> > const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/pluginSimTrackerTrackAssociatorProducersAuto.so
#18 0x000014bb5ac38f35 in MultiTrackValidator::dqmAnalyze(edm::Event const&, edm::EventSetup const&, MultiTrackValidatorHistograms const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/pluginValidationRecoTrackPlugins.so
#19 0x000014bbcec4ae71 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so
#20 0x000014bbcec43357 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so
#21 0x000014bbcebcfe82 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::TransitionActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::TransitionActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::TransitionActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so
#22 0x000014bbcebd0394 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::TransitionActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so
#23 0x000014bbce9cae64 in tbb::detail::d2::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02919/el9_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-12-08-2300/lib/el9_amd64_gcc13/libFWCoreConcurrency.so
#24 0x000014bbce2e119b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=<optimized out>, waiter=..., this=0x14bbccfcfb80) at src/tbb/task_dispatcher.h:344
#25 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14bbccfcfb80) at src/tbb/task_dispatcher.h:487
#26 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at src/tbb/arena.cpp:216
#27 tbb::detail::r1::thread_dispatcher_client::process (td=..., this=<optimized out>) at src/tbb/thread_dispatcher_client.h:41
#28 tbb::detail::r1::thread_dispatcher::process (this=<optimized out>, j=...) at src/tbb/thread_dispatcher.cpp:195
#29 0x000014bbce2d4c75 in tbb::detail::r1::rml::private_worker::run (this=0x14bbca4c7000) at src/tbb/private_server.cpp:271
#30 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14bbca4c7000) at src/tbb/private_server.cpp:221
#31 0x000014bbcde822ea in start_thread () from /lib64/libc.so.6
#32 0x000014bbcdf073c0 in clone3 () from /lib64/libc.so.6

iarspider avatar Dec 09 '25 09:12 iarspider

assign core

makortel avatar Dec 09 '25 14:12 makortel

type root

makortel avatar Dec 09 '25 14:12 makortel

New categories assigned: core

@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild avatar Dec 09 '25 15:12 cmsbuild

@pcanal @vepadulano

Could this be a data race between TH1::Fill() and something between TTree::Fill() and TIsAProxy::operator()?

makortel avatar Dec 09 '25 15:12 makortel

There are a couple of exception messages just before the segfault, but it isn't clear if they're associated with the crash:

%MSG-e TrackProducer:  TrackRefitter:refittedForPixelDQM  09-Dec-2025 06:37:10 CET Run: 1 Event: 201
Genexception1: An exception of category 'WrongTrackerSubDet' occurred.
Exception Message:
Invalid DetID: no GeomDetUnit associated with raw ID 1065702741 of subdet ID 7


%MSG
%MSG-e TrackProducer:  TrackRefitter:refittedForPixelDQM  09-Dec-2025 06:37:10 CET Run: 1 Event: 201
cms::Exception caught during theAlgo.runWithTrack.
An exception of category 'WrongTrackerSubDet' occurred.
Exception Message:
Invalid DetID: no GeomDetUnit associated with raw ID 1065702741 of subdet ID 7

dan131riley avatar Dec 09 '25 15:12 dan131riley