cmssw icon indicating copy to clipboard operation
cmssw copied to clipboard

Published 20 hours ago • verified publisher icon cms-sw

Segfault in GeometricDet::deleteComponents()

Open makortel opened this issue 2 years ago • 3 comments

Workflow 138.5 step 3 crashed in slc7_amd64_gcc10/CMSSW_12_6_X_2022-10-06-2300 with an interesting segfault during the job shutdown

MemoryReport> Peak virtual size 3154.93 Mbytes
...
Thread 5 (Thread 0x2b1359000700 (LWP 101844) "cmsRun"):
#2  0x00002b131a78e420 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b13121aa117 in madvise () from /lib64/libc.so.6
#5  0x00002b1310c8a208 in je_pages_purge_forced (addr=<optimized out>, size=<optimized out>) at src/pages.c:476
#6  0x00002b1310c776de in je_ehooks_default_purge_forced_impl (addr=<optimized out>, offset=offset@entry=0, length=<optimized out>) at src/ehooks.c:146
#7  0x00002b1310c7cfdb in ehooks_purge_forced (length=<optimized out>, offset=0, size=<optimized out>, addr=<optimized out>, ehooks=0x2b132d8000c0, tsdn=0x2b1358ffa828) at include/jemalloc/internal/ehooks.h:318
#8  je_extent_dalloc_wrapper (tsdn=0x2b1358ffa828, pac=0x2b132d8039f0, ehooks=0x2b132d8000c0, edata=0x2b132d817280) at src/extent.c:1060
#9  0x00002b1310c7d2f7 in extent_maximally_purge (edata=0x2b132d817280, ehooks=0x2b132d8000c0, pac=0x2b132d8039f0, tsdn=0x2b1358ffa828) at src/extent.c:893
#10 je_extent_record (tsdn=0x2b1358ffa828, pac=0x2b132d8039f0, ehooks=0x2b132d8000c0, ecache=0x2b132d803a28, edata=0x2b132d817280) at src/extent.c:945
#11 0x00002b1310c7d354 in je_ecache_dalloc (tsdn=<optimized out>, pac=<optimized out>, ehooks=<optimized out>, ecache=<optimized out>, edata=<optimized out>) at src/extent.c:147
#12 0x00002b1310c88cd7 in pac_dalloc_impl (tsdn=0x2b1358ffa828, self=0x2b132d8039f0, edata=0x2b132d817280, deferred_work_generated=0x2b1358ff9c1f) at src/pac.c:277
#13 0x00002b1310c86882 in large_dalloc_finish_impl (edata=0x2b132d817280, arena=0x2b132d801040, tsdn=0x2b1358ffa828) at src/large.c:253
#14 je_large_dalloc (tsdn=tsdn@entry=0x2b1358ffa828, edata=0x2b132d817280) at src/large.c:273
#15 0x00002b1310c294ba in arena_dalloc_large (slow_path=<optimized out>, szind=<optimized out>, tcache=<optimized out>, ptr=<optimized out>, tsdn=<optimized out>) at include/jemalloc/internal/arena_inlines_b.h:297
#16 arena_sdalloc (slow_path=<optimized out>, caller_alloc_ctx=<optimized out>, tcache=<optimized out>, size=<optimized out>, ptr=<optimized out>, tsdn=<optimized out>) at include/jemalloc/internal/arena_inlines_b.h:421
#17 isdalloct (slow_path=<optimized out>, alloc_ctx=<optimized out>, tcache=<optimized out>, size=<optimized out>, ptr=<optimized out>, tsdn=<optimized out>) at include/jemalloc/internal/jemalloc_internal_inlines_c.h:133
#18 isfree (slow_path=<optimized out>, tcache=<optimized out>, usize=<optimized out>, ptr=<optimized out>, tsd=<optimized out>) at src/jemalloc.c:2982
#19 je_sdallocx_default (ptr=<optimized out>, size=<optimized out>, flags=flags@entry=0) at src/jemalloc.c:3924
#20 0x00002b1310c29d8a in je_je_sdallocx_noflags (ptr=<optimized out>, size=<optimized out>) at src/jemalloc.c:3951
#21 0x00002b1310c9a93e in sizedDeleteImpl (size=<optimized out>, ptr=<optimized out>) at src/jemalloc_cpp.cpp:195
#22 operator delete (ptr=<optimized out>, size=<optimized out>) at src/jemalloc_cpp.cpp:200
#23 0x00002b13404622db in PixelCPEClusterRepair::~PixelCPEClusterRepair() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libRecoLocalTrackerSiPixelRecHits.so
#24 0x00002b1340462319 in PixelCPEClusterRepair::~PixelCPEClusterRepair() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libRecoLocalTrackerSiPixelRecHits.so
#25 0x00002b134031b38b in edm::eventsetup::CallbackProxy<edm::eventsetup::Callback<edm::ESProducer, edm::ESProducer::setWhatProduced<PixelCPEClusterRepairESProducer, std::unique_ptr<PixelClusterParameterEstimator, std::default_delete<PixelClusterParameterEstimator> >, TkPixelCPERecord, edm::eventsetup::CallbackSimpleDecorator<TkPixelCPERecord> >(PixelCPEClusterRepairESProducer*, std::unique_ptr<PixelClusterParameterEstimator, std::default_delete<PixelClusterParameterEstimator> > (PixelCPEClusterRepairESProducer::*)(TkPixelCPERecord const&), edm::eventsetup::CallbackSimpleDecorator<TkPixelCPERecord> const&, edm::es::Label const&)::{lambda(TkPixelCPERecord const&)#1}, std::unique_ptr<PixelClusterParameterEstimator, std::default_delete<PixelClusterParameterEstimator> >, TkPixelCPERecord, edm::eventsetup::CallbackSimpleDecorator<TkPixelCPERecord> >, TkPixelCPERecord, std::unique_ptr<PixelClusterParameterEstimator, std::default_delete<PixelClusterParameterEstimator> > >::invalidateCache() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/pluginRecoLocalTrackerSiPixelRecHitsPlugins.so
#26 0x00002b130f9e1e51 in edm::eventsetup::EventSetupRecordImpl::invalidateProxies() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#27 0x00002b130f9dfd6f in edm::FunctorWaitingTask<edm::eventsetup::EventSetupRecordIOVQueue::startNewIOVAsync(edm::WaitingTaskHolder const&, edm::WaitingTaskList&)::{lambda(edm::LimitedTaskQueue::Resumer)#1}::operator()(edm::LimitedTaskQueue::Resumer)::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#28 0x00002b130f95ac9f in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#29 0x00002b1311293ffc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x2b1314fd1f00, waiter=..., this=0x2b1314feea00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/task_dispatcher.h:322
#30 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x2b1314feea00) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/task_dispatcher.h:463
#31 tbb::detail::r1::arena::process (this=<optimized out>, tls=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/arena.cpp:138
#32 0x00002b13112a0593 in tbb::detail::r1::market::process (j=..., this=0x2b1314fef580) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/market.cpp:597
#33 tbb::detail::r1::rml::private_worker::run (this=0x2b13192db000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/private_server.cpp:267
#34 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x2b13192db000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/private_server.cpp:221
#35 0x00002b1311e9cea5 in start_thread () from /lib64/libpthread.so.0
#36 0x00002b13121afb0d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x2b1358000700 (LWP 101843) "cmsRun"):
#3  0x00002b131a791d4b in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00002b133ae4d960 in GeometricDet::deleteComponents() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libGeometryTrackerNumberingBuilder.so
#6  0x00002b133ae4d9b9 in GeometricDet::~GeometricDet() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libGeometryTrackerNumberingBuilder.so
#7  0x00002b133ae4d970 in GeometricDet::deleteComponents() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libGeometryTrackerNumberingBuilder.so
#8  0x00002b133ae4d9b9 in GeometricDet::~GeometricDet() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libGeometryTrackerNumberingBuilder.so
#9  0x00002b133ae4d970 in GeometricDet::deleteComponents() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libGeometryTrackerNumberingBuilder.so
#10 0x00002b133ae4d9b9 in GeometricDet::~GeometricDet() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libGeometryTrackerNumberingBuilder.so
#11 0x00002b133ae4d970 in GeometricDet::deleteComponents() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libGeometryTrackerNumberingBuilder.so
#12 0x00002b133ae4d9b9 in GeometricDet::~GeometricDet() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libGeometryTrackerNumberingBuilder.so
#13 0x00002b133ae4d970 in GeometricDet::deleteComponents() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libGeometryTrackerNumberingBuilder.so
#14 0x00002b133ae4d9b9 in GeometricDet::~GeometricDet() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libGeometryTrackerNumberingBuilder.so
#15 0x00002b133ae4d970 in GeometricDet::deleteComponents() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libGeometryTrackerNumberingBuilder.so
#16 0x00002b133ae4d9b9 in GeometricDet::~GeometricDet() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libGeometryTrackerNumberingBuilder.so
#17 0x00002b1350fd2c12 in edm::eventsetup::CallbackProxy<edm::eventsetup::Callback<edm::ESProducer, edm::ESProducer::setWhatProduced<TrackerGeometricDetESModule, std::unique_ptr<GeometricDet, std::default_delete<GeometricDet> >, IdealGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<IdealGeometryRecord> >(TrackerGeometricDetESModule*, std::unique_ptr<GeometricDet, std::default_delete<GeometricDet> > (TrackerGeometricDetESModule::*)(IdealGeometryRecord const&), edm::eventsetup::CallbackSimpleDecorator<IdealGeometryRecord> const&, edm::es::Label const&)::{lambda(IdealGeometryRecord const&)#1}, std::unique_ptr<GeometricDet, std::default_delete<GeometricDet> >, IdealGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<IdealGeometryRecord> >, IdealGeometryRecord, std::unique_ptr<GeometricDet, std::default_delete<GeometricDet> > >::invalidateCache() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/pluginGeometryTrackerNumberingBuilderPlugins.so
#18 0x00002b130f9e1e51 in edm::eventsetup::EventSetupRecordImpl::invalidateProxies() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#19 0x00002b130f9dfd6f in edm::FunctorWaitingTask<edm::eventsetup::EventSetupRecordIOVQueue::startNewIOVAsync(edm::WaitingTaskHolder const&, edm::WaitingTaskList&)::{lambda(edm::LimitedTaskQueue::Resumer)#1}::operator()(edm::LimitedTaskQueue::Resumer)::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#20 0x00002b130f95ac9f in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#21 0x00002b1311293ffc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x2b136d34e200, waiter=..., this=0x2b1314fee900) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/task_dispatcher.h:322
#22 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x2b1314fee900) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/task_dispatcher.h:463
#23 tbb::detail::r1::arena::process (this=<optimized out>, tls=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/arena.cpp:138
#24 0x00002b13112a0593 in tbb::detail::r1::market::process (j=..., this=0x2b1314fef580) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/market.cpp:597
#25 tbb::detail::r1::rml::private_worker::run (this=0x2b13192db100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/private_server.cpp:267
#26 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x2b13192db100) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/private_server.cpp:221
#27 0x00002b1311e9cea5 in start_thread () from /lib64/libpthread.so.0
#28 0x00002b13121afb0d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x2b1357062700 (LWP 101812) "cmsRun"):
#2  0x00002b131a78e420 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b133ac5c2a7 in std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::vector<int, std::allocator<int> > >, std::_Select1st<std::pair<unsigned int const, std::vector<int, std::allocator<int> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::vector<int, std::allocator<int> > > > >::_M_erase(std::_Rb_tree_node<std::pair<unsigned int const, std::vector<int, std::allocator<int> > > >*) [clone .isra.0] () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libCalibFormatsSiStripObjects.so
#5  0x00002b133ac5c5f7 in SiStripDetCabling::~SiStripDetCabling() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libCalibFormatsSiStripObjects.so
#6  0x00002b133ac5c679 in SiStripDetCabling::~SiStripDetCabling() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libCalibFormatsSiStripObjects.so
#7  0x00002b1340a4bceb in edm::eventsetup::CallbackProxy<edm::eventsetup::Callback<edm::ESProducer, edm::ESProducer::setWhatProduced<SiStripConnectivity, std::unique_ptr<SiStripDetCabling, std::default_delete<SiStripDetCabling> >, SiStripDetCablingRcd, edm::eventsetup::CallbackSimpleDecorator<SiStripDetCablingRcd> >(SiStripConnectivity*, std::unique_ptr<SiStripDetCabling, std::default_delete<SiStripDetCabling> > (SiStripConnectivity::*)(SiStripDetCablingRcd const&), edm::eventsetup::CallbackSimpleDecorator<SiStripDetCablingRcd> const&, edm::es::Label const&)::{lambda(SiStripDetCablingRcd const&)#1}, std::unique_ptr<SiStripDetCabling, std::default_delete<SiStripDetCabling> >, SiStripDetCablingRcd, edm::eventsetup::CallbackSimpleDecorator<SiStripDetCablingRcd> >, SiStripDetCablingRcd, std::unique_ptr<SiStripDetCabling, std::default_delete<SiStripDetCabling> > >::invalidateCache() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/pluginCalibTrackerSiStripGeomESProducersPlugins.so
#8  0x00002b130f9e1e51 in edm::eventsetup::EventSetupRecordImpl::invalidateProxies() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#9  0x00002b130f9dfd6f in edm::FunctorWaitingTask<edm::eventsetup::EventSetupRecordIOVQueue::startNewIOVAsync(edm::WaitingTaskHolder const&, edm::WaitingTaskList&)::{lambda(edm::LimitedTaskQueue::Resumer)#1}::operator()(edm::LimitedTaskQueue::Resumer)::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#10 0x00002b130f95ac9f in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#11 0x00002b1311293ffc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x2b136d34c900, waiter=..., this=0x2b1314fee980) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/task_dispatcher.h:322
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x2b1314fee980) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/task_dispatcher.h:463
#13 tbb::detail::r1::arena::process (this=<optimized out>, tls=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/arena.cpp:138
#14 0x00002b13112a0593 in tbb::detail::r1::market::process (j=..., this=0x2b1314fef580) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/market.cpp:597
#15 tbb::detail::r1::rml::private_worker::run (this=0x2b13192db080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/private_server.cpp:267
#16 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x2b13192db080) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/private_server.cpp:221
#17 0x00002b1311e9cea5 in start_thread () from /lib64/libpthread.so.0
#18 0x00002b13121afb0d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x2b1336ad9700 (LWP 87187) "cmsRun"):
#0  0x00002b1311ea41d9 in waitpid () from /lib64/libpthread.so.0
#1  0x00002b131a78e5d7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#2  0x00002b131a78f1ca in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#3  0x00002b1311a78f90 in std::execute_native_thread_routine (__p=0x2b131797dfc0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#4  0x00002b1311e9cea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00002b13121afb0d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x2b1314232f80 (LWP 71316) "cmsRun"):
#2  0x00002b131a78e420 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b133aca01a4 in TrackerGeometry::~TrackerGeometry() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libGeometryTrackerGeometryBuilder.so
#5  0x00002b133aca0429 in TrackerGeometry::~TrackerGeometry() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libGeometryTrackerGeometryBuilder.so
#6  0x00002b1350f5e1fa in edm::eventsetup::CallbackProxy<edm::eventsetup::Callback<edm::ESProducer, edm::ESProducer::setWhatProduced<TrackerDigiGeometryESModule, std::unique_ptr<TrackerGeometry, std::default_delete<TrackerGeometry> >, TrackerDigiGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<TrackerDigiGeometryRecord> >(TrackerDigiGeometryESModule*, std::unique_ptr<TrackerGeometry, std::default_delete<TrackerGeometry> > (TrackerDigiGeometryESModule::*)(TrackerDigiGeometryRecord const&), edm::eventsetup::CallbackSimpleDecorator<TrackerDigiGeometryRecord> const&, edm::es::Label const&)::{lambda(TrackerDigiGeometryRecord const&)#1}, std::unique_ptr<TrackerGeometry, std::default_delete<TrackerGeometry> >, TrackerDigiGeometryRecord, edm::eventsetup::CallbackSimpleDecorator<TrackerDigiGeometryRecord> >, TrackerDigiGeometryRecord, std::unique_ptr<TrackerGeometry, std::default_delete<TrackerGeometry> > >::invalidateCache() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/pluginGeometryTrackerGeometryBuilderPlugins.so
#7  0x00002b130f9e1e51 in edm::eventsetup::EventSetupRecordImpl::invalidateProxies() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#8  0x00002b130f9dfd6f in edm::FunctorWaitingTask<edm::eventsetup::EventSetupRecordIOVQueue::startNewIOVAsync(edm::WaitingTaskHolder const&, edm::WaitingTaskList&)::{lambda(edm::LimitedTaskQueue::Resumer)#1}::operator()(edm::LimitedTaskQueue::Resumer)::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#9  0x00002b130f95ac9f in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#10 0x00002b13112a6bec in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x2b136d342500, this=0x2b1314fee880) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/task_dispatcher.h:322
#11 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x2b1314fee880) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/task_dispatcher.h:463
#12 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/task_dispatcher.cpp:168
#13 0x00002b130f97d099 in edm::EventProcessor::taskCleanup() () from /cvmfs/cms-ib.cern.ch/nweek-02753/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_6_X_2022-10-06-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#14 0x000000000040a2ac in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#15 0x00002b13112950eb in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.5.0-36aff7df349e0716374b1668ccd18e17/tbb-v2021.5.0/src/tbb/arena.cpp:698
#16 0x000000000040b094 in main::{lambda()#1}::operator()() const ()
#17 0x000000000040971c in main ()

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc10/CMSSW_12_6_X_2022-10-06-2300/pyRelValMatrixLogs/run/138.5_ExpressCollisions+RunMinimumBias2021+TIER0EXPRUN3+ALCARECOEXPR3+HARVESTDEXPR3/step3_ExpressCollisions+RunMinimumBias2021+TIER0EXPRUN3+ALCARECOEXPR3+HARVESTDEXPR3.log#/

makortel avatar Oct 07 '22 13:10 makortel

assign core,geometry

makortel avatar Oct 07 '22 13:10 makortel

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

cmsbuild avatar Oct 07 '22 13:10 cmsbuild

New categories assigned: geometry,core

@mdhildreth,@ianna,@Dr15Jones,@Dr15Jones,@smuzaffar,@makortel,@makortel,@bsunanda,@civanch you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild avatar Oct 07 '22 13:10 cmsbuild