cmssw
cmssw copied to clipboard
cms-sw
Random failures in `SagittaBiasNtuplizer` unit test
I noticed seemingly random failures in the test SagittaBiasNtuplizer in multiple IBs of different flavours.
The crash contains this message:
----- Begin Fatal Exception 26-May-2025 05:12:25 CEST-----------------------
An exception of category 'FileReadError' occurred while
[0] Processing Event run: 1 lumi: 132 event: 13102 stream: 1
[1] Running path 'p1'
[2] Prefetching for module TrackRefitter/'refittedVtxTracks'
[3] While reading from source std::vector<reco::Track> ALCARECOTkAlDiMuonVertexTracks '' ALCA
[4] Rethrowing an exception that happened on a different read request.
[5] Processing Event run: 1 lumi: 131 event: 13097 stream: 5
[6] Running path 'p1'
[7] Calling method for module TrackRefitter/'refittedVtxTracks'
[8] Processing Event run: 1 lumi: 131 event: 13097 stream: 5
[9] Running path 'p1'
[10] Calling method for module TrackRefitter/'refittedVtxTracks'
[11] While reading from source edmNew::DetSetVector<SiStripCluster> ALCARECOTkAlDiMuonVertexTracks '' ALCA
[12] Reading branch SiStripClusteredmNewDetSetVector_ALCARECOTkAlDiMuonVertexTracks__ALCA.
Additional Info:
[a] Fatal Root Error: @SUB=TMap::GetValue
argument is a null pointer
----- End Fatal Exception -------------------------------------------------
As far as I can tell, it started from IB CMSSW_15_1_X_2025-05-22-1100 (possibly pointing to some framework changes such as https://github.com/cms-sw/cmssw/pull/48138 or https://github.com/cms-sw/cmssw/pull/48088).
A new Issue was created by @mmusich.
@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.
cms-bot commands are listed here
assign core
To begin with (based on
(possibly pointing to some framework changes such as https://github.com/cms-sw/cmssw/pull/48138 or https://github.com/cms-sw/cmssw/pull/48088).
)
New categories assigned: core
@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks
When the test fails, is it always while reading the ALCA file or does it also happen when reading the RECO file?
It seems unlikely that https://github.com/cms-sw/cmssw/pull/48088 as the tests are using the delayed reading ability, while the PR in question should only be affecting the behavior when delayed reading is off.
When the test fails, is it always while reading the ALCA file or does it also happen when reading the RECO file?
in all instances I have observed it was always reading the ALCA file (but I do not claim to have checked all of them).
In CMSSW_15_1_X_2025-05-22-1100 the test failure occurred in el9_amd64_gcc14 build, and for that build that IB included a ROOT update https://github.com/cms-sw/cmsdist/pull/9877
So one piece of information, the class SiStripCluster has IO rules which may or may not be contributing to the issue being seen
https://github.com/cms-sw/cmssw/blob/ea8a8a7984665c7013c922b8a0ee0b30eb1b1ee9/DataFormats/SiStripCluster/src/classes_def.xml#L10-L19
I ran the test repeatedly in gdb with ROOT debug build overnight, and got a hit on where the exception is thrown. Here is the stack trace for all threads
Thread 9 (Thread 0x7fff9a5fe700 (LWP 65178) "cmsRun"):
#0 0x00007ffff62f34cd in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007ffff62ecb94 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007ffff7c794cf in edm::DelayedReaderInputProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}::operator()() const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#3 0x00007ffff7c795ec in edm::DelayedReaderInputProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#4 0x00007ffff7bddfc6 in edm::EventPrincipal::getByProductID(edm::ProductID const&) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5 0x00007ffff7bde246 in edm::EventPrincipal::getIt(edm::ProductID const&) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6 0x00007ffff6a541f9 in edm::RefCore::tryToGetProductPtr(std::type_info const&, edm::EDProductGetter const*) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libDataFormatsCommon.so
#7 0x00007fffc21206b4 in TkClonerImpl::makeShared(SiStripRecHit1D const&, TrajectoryStateOnSurface const&) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libRecoTrackerTransientTrackingRecHit.so
#8 0x00007fffc66ecdce in SiStripRecHit1D::cloneSH_(TkCloner const&, TrajectoryStateOnSurface const&) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libDataFormatsTrackerRecHit2D.so
#9 0x00007fffc143a98a in KFTrajectoryFitter::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libTrackingToolsTrackFitters.so
#10 0x00007fffc1461b8f in (anonymous namespace)::KFFittingSmoother::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginTrackingToolsTrackFittersPlugins.so
#11 0x00007fff9de37fe2 in TrackProducerAlgorithm<reco::Track>::buildTrack(TrajectoryFitter const*, Propagator const*, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > >&, TrajectoryStateOnSurface&, TrajectorySeed const&, float, reco::BeamSpot const&, edm::RefToBase<TrajectorySeed>, int, signed char) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libRecoTrackerTrackProducer.so
#12 0x00007fff9cfbeff3 in TrackProducerAlgorithm<reco::Track>::runWithTrack(TrackingGeometry const*, MagneticField const*, edm::View<reco::Track> const&, TrajectoryFitter const*, Propagator const*, TransientTrackingRecHitBuilder const*, reco::BeamSpot const&, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#13 0x00007fff9cfc3e1b in TrackRefitter::produce(edm::Event&, edm::EventSetup const&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#14 0x00007ffff7cef1e5 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#15 0x00007ffff7cd5acc in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#16 0x00007ffff7c5fb59 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#17 0x00007ffff7c61251 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#18 0x00007ffff7ec9828 in tbb::detail::d2::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#19 0x00007ffff70aa60a in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=<optimized out>, waiter=..., this=0x7ffff4bcfc00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/task_dispatcher.h:334
#20 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x7ffff4bcfc00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/task_dispatcher.h:470
#21 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/arena.cpp:215
#22 tbb::detail::r1::thread_dispatcher_client::process (td=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/thread_dispatcher_client.h:41
#23 tbb::detail::r1::thread_dispatcher::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/thread_dispatcher.cpp:195
#24 0x00007ffff70a26b8 in tbb::detail::r1::rml::private_worker::run (this=0x7ffff1cecf00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/private_server.cpp:271
#25 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7ffff1cecf00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/private_server.cpp:221
#26 0x00007ffff62ea1ca in start_thread () from /lib64/libpthread.so.0
#27 0x00007ffff6ac08d3 in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fff9afff700 (LWP 65177) "cmsRun"):
#0 0x00007fffc2e520af in KFUpdator::update(TrajectoryStateOnSurface const&, TrackingRecHit const&) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libTrackingToolsKalmanUpdators.so
#1 0x00007fffc143aa04 in KFTrajectoryFitter::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libTrackingToolsTrackFitters.so
#2 0x00007fffc1461b8f in (anonymous namespace)::KFFittingSmoother::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginTrackingToolsTrackFittersPlugins.so
#3 0x00007fff9de37fe2 in TrackProducerAlgorithm<reco::Track>::buildTrack(TrajectoryFitter const*, Propagator const*, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > >&, TrajectoryStateOnSurface&, TrajectorySeed const&, float, reco::BeamSpot const&, edm::RefToBase<TrajectorySeed>, int, signed char) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libRecoTrackerTrackProducer.so
#4 0x00007fff9cfbeff3 in TrackProducerAlgorithm<reco::Track>::runWithTrack(TrackingGeometry const*, MagneticField const*, edm::View<reco::Track> const&, TrajectoryFitter const*, Propagator const*, TransientTrackingRecHitBuilder const*, reco::BeamSpot const&, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#5 0x00007fff9cfc3e1b in TrackRefitter::produce(edm::Event&, edm::EventSetup const&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#6 0x00007ffff7cef1e5 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7 0x00007ffff7cd5acc in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#8 0x00007ffff7c5fb59 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#9 0x00007ffff7c61251 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#10 0x00007ffff7ec9828 in tbb::detail::d2::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#11 0x00007ffff70aa60a in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=<optimized out>, waiter=..., this=0x7ffff4bcfb00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/task_dispatcher.h:334
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x7ffff4bcfb00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/task_dispatcher.h:470
#13 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/arena.cpp:215
#14 tbb::detail::r1::thread_dispatcher_client::process (td=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/thread_dispatcher_client.h:41
#15 tbb::detail::r1::thread_dispatcher::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/thread_dispatcher.cpp:195
#16 0x00007ffff70a26b8 in tbb::detail::r1::rml::private_worker::run (this=0x7ffff1cece00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/private_server.cpp:271
#17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7ffff1cece00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/private_server.cpp:221
#18 0x00007ffff62ea1ca in start_thread () from /lib64/libpthread.so.0
#19 0x00007ffff6ac08d3 in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fff9bffe700 (LWP 65176) "cmsRun"):
#0 0x00007ffff62f34cd in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007ffff62ecb94 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007ffff7c794cf in edm::DelayedReaderInputProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}::operator()() const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#3 0x00007ffff7c795ec in edm::DelayedReaderInputProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#4 0x00007ffff7bddfc6 in edm::EventPrincipal::getByProductID(edm::ProductID const&) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5 0x00007ffff7bde246 in edm::EventPrincipal::getIt(edm::ProductID const&) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6 0x00007ffff6a541f9 in edm::RefCore::tryToGetProductPtr(std::type_info const&, edm::EDProductGetter const*) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libDataFormatsCommon.so
#7 0x00007fffc2120a46 in TkClonerImpl::makeShared(SiPixelRecHit const&, TrajectoryStateOnSurface const&) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libRecoTrackerTransientTrackingRecHit.so
#8 0x00007fffc66ecdec in SiPixelRecHit::cloneSH_(TkCloner const&, TrajectoryStateOnSurface const&) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libDataFormatsTrackerRecHit2D.so
#9 0x00007fffc143a98a in KFTrajectoryFitter::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libTrackingToolsTrackFitters.so
#10 0x00007fffc1461b8f in (anonymous namespace)::KFFittingSmoother::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginTrackingToolsTrackFittersPlugins.so
#11 0x00007fff9de37fe2 in TrackProducerAlgorithm<reco::Track>::buildTrack(TrajectoryFitter const*, Propagator const*, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > >&, TrajectoryStateOnSurface&, TrajectorySeed const&, float, reco::BeamSpot const&, edm::RefToBase<TrajectorySeed>, int, signed char) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libRecoTrackerTrackProducer.so
#12 0x00007fff9cfbeff3 in TrackProducerAlgorithm<reco::Track>::runWithTrack(TrackingGeometry const*, MagneticField const*, edm::View<reco::Track> const&, TrajectoryFitter const*, Propagator const*, TransientTrackingRecHitBuilder const*, reco::BeamSpot const&, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#13 0x00007fff9cfc3e1b in TrackRefitter::produce(edm::Event&, edm::EventSetup const&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#14 0x00007ffff7cef1e5 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#15 0x00007ffff7cd5acc in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#16 0x00007ffff7c5fb59 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#17 0x00007ffff7c61251 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#18 0x00007ffff7ec9828 in tbb::detail::d2::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#19 0x00007ffff70aa60a in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=<optimized out>, waiter=..., this=0x7ffff4bcfa00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/task_dispatcher.h:334
#20 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x7ffff4bcfa00) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/task_dispatcher.h:470
#21 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/arena.cpp:215
#22 tbb::detail::r1::thread_dispatcher_client::process (td=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/thread_dispatcher_client.h:41
#23 tbb::detail::r1::thread_dispatcher::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/thread_dispatcher.cpp:195
#24 0x00007ffff70a26b8 in tbb::detail::r1::rml::private_worker::run (this=0x7ffff1ced000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/private_server.cpp:271
#25 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7ffff1ced000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/private_server.cpp:221
#26 0x00007ffff62ea1ca in start_thread () from /lib64/libpthread.so.0
#27 0x00007ffff6ac08d3 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fff9c9ff700 (LWP 65175) "cmsRun"):
#0 0x00007fff9ce53161 in VertexTrack<5u>::VertexTrack(ReferenceCountingPointer<LinearizedTrackState<5u> >, VertexState, float, ReferenceCountingPointer<RefittedTrackState<5u> > const&, float, ROOT::Math::SMatrix<double, 6u, 6u, ROOT::Math::MatRepSym<double, 6u> > const&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libRecoVertexVertexPrimitives.so
#1 0x00007fff9b4f482b in KalmanVertexTrackUpdator<5u>::update(CachingVertex<5u> const&, ReferenceCountingPointer<VertexTrack<5u> >) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libRecoVertexKalmanVertexFit.so
#2 0x00007fff9b4f49b6 in KalmanVertexTrackCompatibilityEstimator<5u>::estimateFittedTrack(CachingVertex<5u> const&, ReferenceCountingPointer<VertexTrack<5u> >) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libRecoVertexKalmanVertexFit.so
#3 0x00007fff9b4f4b88 in KalmanVertexTrackCompatibilityEstimator<5u>::estimate(CachingVertex<5u> const&, ReferenceCountingPointer<VertexTrack<5u> >, unsigned int) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libRecoVertexKalmanVertexFit.so
#4 0x00007fff9b4eca28 in KalmanVertexTrackCompatibilityEstimator<5u>::estimate(CachingVertex<5u> const&, ReferenceCountingPointer<LinearizedTrackState<5u> >, unsigned int) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libRecoVertexKalmanVertexFit.so
#5 0x00007fff9b514cec in AdaptiveVertexFitter::reWeightTracks(std::vector<ReferenceCountingPointer<LinearizedTrackState<5u> >, std::allocator<ReferenceCountingPointer<LinearizedTrackState<5u> > > > const&, CachingVertex<5u> const&) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libRecoVertexAdaptiveVertexFit.so
#6 0x00007fff9b51667c in AdaptiveVertexFitter::reWeightTracks(std::vector<ReferenceCountingPointer<VertexTrack<5u> >, std::allocator<ReferenceCountingPointer<VertexTrack<5u> > > > const&, CachingVertex<5u> const&) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libRecoVertexAdaptiveVertexFit.so
#7 0x00007fff9b518a5b in AdaptiveVertexFitter::fit(std::vector<ReferenceCountingPointer<VertexTrack<5u> >, std::allocator<ReferenceCountingPointer<VertexTrack<5u> > > > const&, VertexState const&, bool) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libRecoVertexAdaptiveVertexFit.so
#8 0x00007fff9b519891 in AdaptiveVertexFitter::vertex(std::vector<reco::TransientTrack, std::allocator<reco::TransientTrack> > const&) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libRecoVertexAdaptiveVertexFit.so
#9 0x00007fff9b590d43 in SequentialPrimaryVertexFitterAdapter::fit(std::vector<reco::TransientTrack, std::allocator<reco::TransientTrack> > const&, std::vector<TransientVertex, std::allocator<TransientVertex> > const&, reco::BeamSpot const&, bool) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginRecoVertexPrimaryVertexProducerPlugins.so
#10 0x00007fff9b5938e2 in PrimaryVertexProducer::produce(edm::Event&, edm::EventSetup const&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginRecoVertexPrimaryVertexProducerPlugins.so
#11 0x00007ffff7cef1e5 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#12 0x00007ffff7cd5acc in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#13 0x00007ffff7c5fb59 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#14 0x00007ffff7c61251 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#15 0x00007ffff7ec9828 in tbb::detail::d2::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#16 0x00007ffff70aa60a in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=<optimized out>, waiter=..., this=0x7ffff4bcf980) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/task_dispatcher.h:334
#17 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x7ffff4bcf980) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/task_dispatcher.h:470
#18 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/arena.cpp:215
#19 tbb::detail::r1::thread_dispatcher_client::process (td=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/thread_dispatcher_client.h:41
#20 tbb::detail::r1::thread_dispatcher::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/thread_dispatcher.cpp:195
#21 0x00007ffff70a26b8 in tbb::detail::r1::rml::private_worker::run (this=0x7ffff1cece80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/private_server.cpp:271
#22 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7ffff1cece80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/private_server.cpp:221
#23 0x00007ffff62ea1ca in start_thread () from /lib64/libpthread.so.0
#24 0x00007ffff6ac08d3 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fff9d9ff700 (LWP 65174) "cmsRun"):
#0 0x00007ffff62f34cd in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007ffff62ecb94 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007ffff7c7ae83 in edm::DelayedReaderInputProductResolver::retrieveAndMerge_(edm::Principal const&, edm::MergeableRunProductMetadata const*) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#3 0x00007ffff7c6e70c in edm::Principal::readAllFromSourceAndMergeImmediately(edm::MergeableRunProductMetadata const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#4 0x00007fff9dfaa75a in edm::RootFile::readLuminosityBlock_(edm::LuminosityBlockPrincipal&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#5 0x00007fff9df8e283 in edm::PoolSource::readLuminosityBlock_(edm::LuminosityBlockPrincipal&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#6 0x00007ffff7c46e0e in void edm::callWithTryCatchAndPrint<void>(std::function<void ()>, char const*, bool) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7 0x00007ffff7c47071 in edm::InputSource::readLuminosityBlock(edm::LuminosityBlockPrincipal&, edm::HistoryAppender&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#8 0x00007ffff7beebaf in edm::EventProcessor::readLuminosityBlock(std::shared_ptr<edm::RunPrincipal>) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#9 0x00007ffff7beffcf in edm::EventProcessor::beginLumiAsync(edm::IOVSyncValue const&, std::shared_ptr<edm::RunProcessingStatus>, edm::WaitingTaskHolder)::{lambda(std::__exception_ptr::exception_ptr const*, auto:1)#2}::operator()<edm::WaitingTaskHolder>(std::__exception_ptr::exception_ptr const*, edm::WaitingTaskHolder) const::{lambda(edm::LimitedTaskQueue::Resumer)#1}::operator()(edm::LimitedTaskQueue::Resumer)::{lambda()#1}::operator()() [clone .lto_priv.0] () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#10 0x00007ffff7c11243 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::EventProcessor::beginLumiAsync(edm::IOVSyncValue const&, std::shared_ptr<edm::RunProcessingStatus>, edm::WaitingTaskHolder)::{lambda(std::__exception_ptr::exception_ptr const*, auto:1)#2}::operator()<edm::WaitingTaskHolder>(std::__exception_ptr::exception_ptr const*,edm::WaitingTaskHolder) const::{lambda(edm::LimitedTaskQueue::Resumer)#1}::operator()(edm::LimitedTaskQueue::Resumer)::{lambda()#1}>(tbb::detail::d2::task_group&, edm::EventProcessor::beginLumiAsync(edm::IOVSyncValue const&, std::shared_ptr<edm::RunProcessingStatus>, edm::WaitingTaskHolder)::{lambda(std::__exception_ptr::exception_ptr const*, auto:1)#2}::operator()<edm::WaitingTaskHolder>(std::__exception_ptr::exception_ptr const*, edm::WaitingTaskHolder) const::{lambda(edm::LimitedTaskQueue::Resumer)#1}::operator()(edm::LimitedTaskQueue::Resumer)::{lambda()#1}&&)::{lambda()#1}>::execute() [clone .lto_priv.0] () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#11 0x00007ffff7ecb205 in tbb::detail::d2::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#12 0x00007ffff70aa60a in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=<optimized out>, waiter=..., this=0x7ffff4bcfb80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/task_dispatcher.h:334
#13 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x7ffff4bcfb80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/task_dispatcher.h:470
#14 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/arena.cpp:215
#15 tbb::detail::r1::thread_dispatcher_client::process (td=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/thread_dispatcher_client.h:41
#16 tbb::detail::r1::thread_dispatcher::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/thread_dispatcher.cpp:195
#17 0x00007ffff70a26b8 in tbb::detail::r1::rml::private_worker::run (this=0x7ffff1cecf80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/private_server.cpp:271
#18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7ffff1cecf80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/private_server.cpp:221
#19 0x00007ffff62ea1ca in start_thread () from /lib64/libpthread.so.0
#20 0x00007ffff6ac08d3 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fff9e9ee700 (LWP 65173) "cmsRun"):
#0 0x00007ffff6f0a2f1 in __cxxabiv1::__cxa_throw (obj=0x7fff76219700, tinfo=0x7ffff7e9d860 <typeinfo for edm::Exception>, dest=0x7ffff7e63450 <edm::Exception::~Exception()>) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:81
#1 0x00007ffff187b956 in (anonymous namespace)::RootErrorHandlerImpl(int, char const*, char const*) [clone .cold] () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2 0x00007ffff7813a1e in ErrorHandler(Int_t, const char *, const char *, typedef __va_list_tag __va_list_tag *) (level=3000, location=0x7fff9e9eabec "TMap::GetValue", fmt=0x7ffff78ddf40 "argument is a null pointer", ap=0x7fff9e9e6b68) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d931bc5cddb5087cc8416931c48ac2a5/root-6.32.13/core/foundation/src/TError.cxx:148
#3 0x00007ffff774c7fe in TObject::DoError (this=0x7fffa1e2e980, level=3000, location=0x7ffff78dec26 "GetValue", fmt=0x7ffff78ddf40 "argument is a null pointer", va=0x7fff9e9e6b68) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d931bc5cddb5087cc8416931c48ac2a5/root-6.32.13/core/base/src/TObject.cxx:954
#4 0x00007ffff774ca67 in TObject::Error (this=0x7fffa1e2e980, location=0x7ffff78dec26 "GetValue", fmt=0x7ffff78ddf40 "argument is a null pointer") at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d931bc5cddb5087cc8416931c48ac2a5/root-6.32.13/core/base/src/TObject.cxx:991
#5 0x00007ffff77cf616 in TCollection::IsArgNull (this=0x7fffa1e2e980, where=0x7ffff78dec26 "GetValue", obj=0x0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d931bc5cddb5087cc8416931c48ac2a5/root-6.32.13/core/cont/src/TCollection.cxx:374
#6 0x00007ffff77de6b8 in TMap::GetValue (this=0x7fffa1e2e980, key=0x0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d931bc5cddb5087cc8416931c48ac2a5/root-6.32.13/core/cont/src/TMap.cxx:248
#7 0x00007ffff0aa3c5d in TFile::GetCacheRead (this=0x7fffa1786e80, tree=0x0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d931bc5cddb5087cc8416931c48ac2a5/root-6.32.13/io/io/src/TFile.cxx:1273
#8 0x00007fffeed79a63 in TStorageFactoryFile::ReadBuffer(char*, int) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libIOPoolTFileAdaptor.so
#9 0x00007ffff0ee0da6 in TBasket::ReadBasketBuffers (this=0x7fff76fe5700, pos=86396606, len=11511, file=0x7fffa1786e80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d931bc5cddb5087cc8416931c48ac2a5/root-6.32.13/tree/tree/src/TBasket.cxx:547
#10 0x00007ffff0ef12f7 in TBranch::GetBasketImpl (this=0x7fffa276eb00, basketnumber=214, user_buffer=0x0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d931bc5cddb5087cc8416931c48ac2a5/root-6.32.13/tree/tree/src/TBranch.cxx:1268
#11 0x00007ffff0ef1ad8 in TBranch::GetBasketAndFirst (this=0x7fffa276eb00, basket=@0x7fff9e9e7030: 0x0, first=@0x7fff9e9e7028: 2497, user_buffer=0x0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d931bc5cddb5087cc8416931c48ac2a5/root-6.32.13/tree/tree/src/TBranch.cxx:1390
#12 0x00007ffff0ef28ca in TBranch::GetEntry (this=0x7fffa276eb00, entry=2498, getall=0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d931bc5cddb5087cc8416931c48ac2a5/root-6.32.13/tree/tree/src/TBranch.cxx:1716
#13 0x00007ffff0f0618c in TBranchElement::GetEntry (this=0x7fffa276eb00, entry=2498, getall=0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d931bc5cddb5087cc8416931c48ac2a5/root-6.32.13/tree/tree/src/TBranchElement.cxx:2783
#14 0x00007ffff0f05d4b in TBranchElement::GetEntry (this=0x7fffa276aa00, entry=2498, getall=0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d931bc5cddb5087cc8416931c48ac2a5/root-6.32.13/tree/tree/src/TBranchElement.cxx:2727
#15 0x00007ffff0f05d4b in TBranchElement::GetEntry (this=0x7fffa276a500, entry=2498, getall=0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d931bc5cddb5087cc8416931c48ac2a5/root-6.32.13/tree/tree/src/TBranchElement.cxx:2727
#16 0x00007fff9dfbd718 in edm::RootTree::getEntryUsingCache(TBranch*, long long, TTreeCache*) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#17 0x00007fff9df9c315 in edm::RootDelayedReader::getProduct_(edm::BranchID const&, edm::EDProductGetter const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginIOPoolInput.so
#18 0x00007ffff7bca27b in edm::DelayedReader::getProduct(edm::BranchID const&, edm::EDProductGetter const*, edm::ModuleCallingContext const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#19 0x00007ffff7c7945f in edm::DelayedReaderInputProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}::operator()() const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#20 0x00007ffff7c795ec in edm::DelayedReaderInputProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#21 0x00007ffff7bddfc6 in edm::EventPrincipal::getByProductID(edm::ProductID const&) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#22 0x00007ffff7bde246 in edm::EventPrincipal::getIt(edm::ProductID const&) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#23 0x00007ffff6a541f9 in edm::RefCore::tryToGetProductPtr(std::type_info const&, edm::EDProductGetter const*) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libDataFormatsCommon.so
#24 0x00007fff9cf9de18 in edm::refitem::GetRefPtrImpl<std::vector<reco::TrackExtra, std::allocator<reco::TrackExtra> >, reco::TrackExtra, edm::refhelper::FindUsingAdvance<std::vector<reco::TrackExtra, std::allocator<reco::TrackExtra> >, reco::TrackExtra>, unsigned int>::getRefPtr_(edm::RefCore const&, unsigned int) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#25 0x00007fff9cfbe766 in TrackProducerAlgorithm<reco::Track>::runWithTrack(TrackingGeometry const*, MagneticField const*, edm::View<reco::Track> const&, TrajectoryFitter const*, Propagator const*, TransientTrackingRecHitBuilder const*, reco::BeamSpot const&, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#26 0x00007fff9cfc3e1b in TrackRefitter::produce(edm::Event&, edm::EventSetup const&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#27 0x00007ffff7cef1e5 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#28 0x00007ffff7cd5acc in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#29 0x00007ffff7c5fb59 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#30 0x00007ffff7c61251 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#31 0x00007ffff7ec9828 in tbb::detail::d2::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#32 0x00007ffff70aa60a in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=<optimized out>, waiter=..., this=0x7ffff4bcfc80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/task_dispatcher.h:334
#33 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x7ffff4bcfc80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/task_dispatcher.h:470
#34 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/arena.cpp:215
#35 tbb::detail::r1::thread_dispatcher_client::process (td=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/thread_dispatcher_client.h:41
#36 tbb::detail::r1::thread_dispatcher::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/thread_dispatcher.cpp:195
#37 0x00007ffff70a26b8 in tbb::detail::r1::rml::private_worker::run (this=0x7ffff1ced100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/private_server.cpp:271
#38 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7ffff1ced100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/private_server.cpp:221
#39 0x00007ffff62ea1ca in start_thread () from /lib64/libpthread.so.0
#40 0x00007ffff6ac08d3 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fff9f3ef700 (LWP 65172) "cmsRun"):
#0 0x00007ffff62f34cd in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007ffff62ecb94 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007ffff7c794cf in edm::DelayedReaderInputProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}::operator()() const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#3 0x00007ffff7c795ec in edm::DelayedReaderInputProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#4 0x00007ffff7bddfc6 in edm::EventPrincipal::getByProductID(edm::ProductID const&) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5 0x00007ffff7bde246 in edm::EventPrincipal::getIt(edm::ProductID const&) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6 0x00007ffff6a541f9 in edm::RefCore::tryToGetProductPtr(std::type_info const&, edm::EDProductGetter const*) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libDataFormatsCommon.so
#7 0x00007fff9cf9de18 in edm::refitem::GetRefPtrImpl<std::vector<reco::TrackExtra, std::allocator<reco::TrackExtra> >, reco::TrackExtra, edm::refhelper::FindUsingAdvance<std::vector<reco::TrackExtra, std::allocator<reco::TrackExtra> >, reco::TrackExtra>, unsigned int>::getRefPtr_(edm::RefCore const&, unsigned int) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#8 0x00007fff9cfbe766 in TrackProducerAlgorithm<reco::Track>::runWithTrack(TrackingGeometry const*, MagneticField const*, edm::View<reco::Track> const&, TrajectoryFitter const*, Propagator const*, TransientTrackingRecHitBuilder const*, reco::BeamSpot const&, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#9 0x00007fff9cfc3e1b in TrackRefitter::produce(edm::Event&, edm::EventSetup const&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#10 0x00007ffff7cef1e5 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#11 0x00007ffff7cd5acc in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#12 0x00007ffff7c5fb59 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#13 0x00007ffff7c61251 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#14 0x00007ffff7ec9828 in tbb::detail::d2::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#15 0x00007ffff70aa60a in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=<optimized out>, waiter=..., this=0x7ffff4bcfa80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/task_dispatcher.h:334
#16 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x7ffff4bcfa80) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/task_dispatcher.h:470
#17 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/arena.cpp:215
#18 tbb::detail::r1::thread_dispatcher_client::process (td=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/thread_dispatcher_client.h:41
#19 tbb::detail::r1::thread_dispatcher::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/thread_dispatcher.cpp:195
#20 0x00007ffff70a26b8 in tbb::detail::r1::rml::private_worker::run (this=0x7ffff1ced080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/private_server.cpp:271
#21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7ffff1ced080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/private_server.cpp:221
#22 0x00007ffff62ea1ca in start_thread () from /lib64/libpthread.so.0
#23 0x00007ffff6ac08d3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7ffff5553580 (LWP 65075) "cmsRun"):
#0 0x00007ffff62f34cd in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007ffff62ecb94 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007ffff7c794cf in edm::DelayedReaderInputProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const::{lambda()#1}::operator()() const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#3 0x00007ffff7c795ec in edm::DelayedReaderInputProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#4 0x00007ffff7bddfc6 in edm::EventPrincipal::getByProductID(edm::ProductID const&) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#5 0x00007ffff7bde246 in edm::EventPrincipal::getIt(edm::ProductID const&) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#6 0x00007ffff6a541f9 in edm::RefCore::tryToGetProductPtr(std::type_info const&, edm::EDProductGetter const*) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libDataFormatsCommon.so
#7 0x00007fff9cf9de18 in edm::refitem::GetRefPtrImpl<std::vector<reco::TrackExtra, std::allocator<reco::TrackExtra> >, reco::TrackExtra, edm::refhelper::FindUsingAdvance<std::vector<reco::TrackExtra, std::allocator<reco::TrackExtra> >, reco::TrackExtra>, unsigned int>::getRefPtr_(edm::RefCore const&, unsigned int) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#8 0x00007fff9cfbe766 in TrackProducerAlgorithm<reco::Track>::runWithTrack(TrackingGeometry const*, MagneticField const*, edm::View<reco::Track> const&, TrajectoryFitter const*, Propagator const*, TransientTrackingRecHitBuilder const*, reco::BeamSpot const&, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#9 0x00007fff9cfc3e1b in TrackRefitter::produce(edm::Event&, edm::EventSetup const&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#10 0x00007ffff7cef1e5 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#11 0x00007ffff7cd5acc in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#12 0x00007ffff7c5fb59 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#13 0x00007ffff7c61251 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#14 0x00007ffff7ec9828 in tbb::detail::d2::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#15 0x00007ffff70a787b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x7ffff4aeb900, this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/task_dispatcher.h:334
#16 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/task_dispatcher.h:470
#17 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/task_dispatcher.cpp:168
#18 0x00007ffff7bdfe0f in edm::FinalWaitingTask::wait() () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#19 0x00007ffff7befbc1 in edm::EventProcessor::processRuns() () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#20 0x00007ffff7be8f81 in edm::EventProcessor::runToCompletion() () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#21 0x0000000000408556 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#22 0x00007ffff7095f71 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2022.0.0-fe4e5fc27f2e8eae5da4b66ede97f22b/tbb-v2022.0.0/src/tbb/arena.cpp:821
#23 0x000000000040a283 in main::{lambda()#1}::operator()() const ()
#24 0x00000000004051b8 in main ()
Here job was reading std::vector<reco::TrackExtra>
----- Begin Fatal Exception 01-Jun-2025 14:27:24 CEST-----------------------
An exception of category 'FileReadError' occurred while
[0] Processing Event run: 1 lumi: 42 event: 4200 stream: 2
[1] Running path 'p1'
[2] Calling method for module TrackRefitter/'refittedVtxTracks'
[3] Processing Event run: 1 lumi: 42 event: 4200 stream: 2
[4] Running path 'p1'
[5] Calling method for module TrackRefitter/'refittedVtxTracks'
[6] While reading from source std::vector<reco::TrackExtra> ALCARECOTkAlDiMuonVertexTracks '' ALCA
[7] Reading branch recoTrackExtras_ALCARECOTkAlDiMuonVertexTracks__ALCA.
Additional Info:
[a] Fatal Root Error: @SUB=TMap::GetValue
argument is a null pointer
----- End Fatal Exception -------------------------------------------------
Inspecting the state in thread 4 the error seems to be related to https://github.com/cms-sw/cmssw/pull/48088 and how the TTreeCache switching was changed.
In this stack frame
#7 0x00007ffff0aa3c5d in TFile::GetCacheRead (this=0x7fffa1786e80, tree=0x0) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d931bc5cddb5087cc8416931c48ac2a5/root-6.32.13/io/io/src/TFile.cxx:1273
The functions looks like
TFileCacheRead *TFile::GetCacheRead(const TObject* tree) const
{
if (!tree) {
if (!fCacheRead && fCacheReadMap->GetSize() == 1) {
TIter next(fCacheReadMap);
return (TFileCacheRead *)fCacheReadMap->GetValue(next()); // <--- this is line 1273
}
return fCacheRead;
}
TFileCacheRead *cache = (TFileCacheRead *)fCacheReadMap->GetValue(tree);
if (!cache) return fCacheRead;
return cache;
}
so it looks like somehow the iterator next() returns a null pointer. Inspecting the fCacheReadMap shows
(gdb) p this->fCacheReadMap->fSize
$8 = 1
(gdb) p this->fCacheReadMap->fTable->GetSize()
$7 = 0
which feels weird.
So that's from https://github.com/cms-sw/cmssw/blob/450f248b8d24194a8962f492f6030147fd70e9dc/IOPool/TFileAdaptor/src/TStorageFactoryFile.cc#L261-L264
One of the goals of https://github.com/cms-sw/cmssw/pull/48088 (and my changes as well) is to always pass in a tree to TFile::{G,S}etCacheRead(), so I guess the nullptr call in TStorageFactoryFile is either tickling some bug or violating some invariant since the SetCacheRead behavior was changed.
I think this stack points to the problem
Thread 5 (Thread 0x7fff9d9ff700 (LWP 65174) "cmsRun"):
#0 0x00007ffff62f34cd in __lll_lock_wait () from /lib64/[libpthread.so](http://libpthread.so/).0
#1 0x00007ffff62ecb94 in pthread_mutex_lock () from /lib64/[libpthread.so](http://libpthread.so/).0
#2 0x00007ffff7c7ae83 in edm::DelayedReaderInputProductResolver::retrieveAndMerge_(edm::Principal const&, edm::MergeableRunProductMetadata const*) const () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/[libFWCoreFramework.so](http://libfwcoreframework.so/)
#3 0x00007ffff7c6e70c in edm::Principal::readAllFromSourceAndMergeImmediately(edm::MergeableRunProductMetadata const*) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/[libFWCoreFramework.so](http://libfwcoreframework.so/)
#4 0x00007fff9dfaa75a in edm::RootFile::readLuminosityBlock_(edm::LuminosityBlockPrincipal&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/[pluginIOPoolInput.so](http://pluginiopoolinput.so/)
#5 0x00007fff9df8e283 in edm::PoolSource::readLuminosityBlock_(edm::LuminosityBlockPrincipal&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/[pluginIOPoolInput.so](http://pluginiopoolinput.so/)
#6 0x00007ffff7c46e0e in void edm::callWithTryCatchAndPrint<void>(std::function<void ()>, char const*, bool) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/[libFWCoreFramework.so](http://libfwcoreframework.so/)
#7 0x00007ffff7c47071 in edm::InputSource::readLuminosityBlock(edm::LuminosityBlockPrincipal&, edm::HistoryAppender&) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/[libFWCoreFramework.so](http://libfwcoreframework.so/)
#8 0x00007ffff7beebaf in edm::EventProcessor::readLuminosityBlock(std::shared_ptr<edm::RunPrincipal>) () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/[libFWCoreFramework.so](http://libfwcoreframework.so/)
#9 0x00007ffff7beffcf in edm::EventProcessor::beginLumiAsync(edm::IOVSyncValue const&, std::shared_ptr<edm::RunProcessingStatus>, edm::WaitingTaskHolder)::{lambda(std::__exception_ptr::exception_ptr const*, auto:1)#2}::operator()<edm::WaitingTaskHolder>(std::__exception_ptr::exception_ptr const*, edm::WaitingTaskHolder) const::{lambda(edm::LimitedTaskQueue::Resumer)#1}::operator()(edm::LimitedTaskQueue::Resumer)::{lambda()#1}::operator()() [clone .lto_priv.0] () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/[libFWCoreFramework.so](http://libfwcoreframework.so/)
#10 0x00007ffff7c11243 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::EventProcessor::beginLumiAsync(edm::IOVSyncValue const&, std::shared_ptr<edm::RunProcessingStatus>, edm::WaitingTaskHolder)::{lambda(std::__exception_ptr::exception_ptr const*, auto:1)#2}::operator()<edm::WaitingTaskHolder>(std::__exception_ptr::exception_ptr const*,edm::WaitingTaskHolder) const::{lambda(edm::LimitedTaskQueue::Resumer)#1}::operator()(edm::LimitedTaskQueue::Resumer)::{lambda()#1}>(tbb::detail::d2::task_group&, edm::EventProcessor::beginLumiAsync(edm::IOVSyncValue const&, std::shared_ptr<edm::RunProcessingStatus>, edm::WaitingTaskHolder)::{lambda(std::__exception_ptr::exception_ptr const*, auto:1)#2}::operator()<edm::WaitingTaskHolder>(std::__exception_ptr::exception_ptr const*, edm::WaitingTaskHolder) const::{lambda(edm::LimitedTaskQueue::Resumer)#1}::operator()(edm::LimitedTaskQueue::Resumer)::{lambda()#1}&&)::{lambda()#1}>::execute() [clone .lto_priv.0] () from /build/mkortela/debug/issue48171/CMSSW_15_1_X_2025-05-27-1100/lib/el8_amd64_gcc12/[libFWCoreFramework.so](http://libfwcoreframework.so/)
The lock is taken too late. The lock should happen before the call to edm::PoolSource::readLuminosityBlock_. This points to a concurrency problem in dealing with concurrent LuminosityBlocks.
The lock is taken too late. The lock should happen before the call to
edm::PoolSource::readLuminosityBlock_. This points to a concurrency problem in dealing with concurrent LuminosityBlocks.
So we have overlapping SetCacheRead() calls and the GetCacheRead(nullptr) in TStorageFactoryFile is getting the wrong cache or looking at the CacheReadMap in an inconsistent state/
@cms-sw/core-l2 , re-iterating on the above:
Has https://github.com/cms-sw/cmssw/pull/48270 solved this issue?
If yes, please sign / close the issue. Thanks