cmssw [AARCH64] SIGSEGV in CkfTrackCandidateMaker:lowPtTripletStepTrackCandidates

Unit test DQMOffline/Configuration/TestDQMOnlineClient-visualization_secondInstance failed for arm64 in CMSSW_13_1_X-2023-03-02-2300 with segmentation violation:

#1  0x000040002d4b80f0 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#2  0x000040002d4bf0d0 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3  0x000040002708d0bc in std::execute_native_thread_routine (__p=0x40002927d850) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x00004000272d78b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400027333afc in thread_start () from /lib64/libc.so.6
Thread 1 (Thread 0x400026f1d030 (LWP 3933761) "cmsRun"):
#0  0x00004000273b5934 in nanosleep () from /lib64/libc.so.6
#1  0x00004000273b57d8 in sleep () from /lib64/libc.so.6
#2  0x000040002d4b28b8 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040003f82d460 in HelixArbitraryPlaneCrossing::pathLength(Plane const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libTrackingToolsGeomPropagators.so
#5  0x000040003f836298 in AnalyticalPropagator::propagateWithHelixCrossing(HelixPlaneCrossing&, Plane const&, float, Point3DBase<float, GlobalTag>&, Vector3DBase<float, GlobalTag>&, double&) const [clone .constprop.0] () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libTrackingToolsGeomPropagators.so
#6  0x000040003f833274 in AnalyticalPropagator::propagateWithPath(FreeTrajectoryState const&, Plane const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libTrackingToolsGeomPropagators.so
#7  0x000040003f82a884 in Propagator::propagateWithPath(TrajectoryStateOnSurface const&, Plane const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libTrackingToolsGeomPropagators.so
#8  0x000040004d746934 in PropagatorWithMaterial::propagateWithPath(TrajectoryStateOnSurface const&, Plane const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libTrackingToolsMaterialEffects.so
#9  0x000040003f7fa45c in GeomDetCompatibilityChecker::isCompatible(GeomDet const*, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libTrackingToolsDetLayers.so
#10 0x000040004cd78acc in CompatibleDetToGroupAdder::add(GeomDet const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector<DetGroup, std::allocator<DetGroup> >&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libRecoTrackerTkDetLayers.so
#11 0x000040004cd80980 in TIBRing::groupedCompatibleDetsV(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector<DetGroup, std::allocator<DetGroup> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libRecoTrackerTkDetLayers.so
#12 0x000040004cd79538 in CompatibleDetToGroupAdder::add(GeometricSearchDet const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector<DetGroup, std::allocator<DetGroup> >&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libRecoTrackerTkDetLayers.so
#13 0x000040004cd8051c in TBLayer::groupedCompatibleDetsV(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector<DetGroup, std::allocator<DetGroup> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libRecoTrackerTkDetLayers.so
#14 0x000040003f7f7488 in GeometricSearchDet::groupedCompatibleDets(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libTrackingToolsDetLayers.so
#15 0x000040004cd07e68 in LayerMeasurements::groupedMeasurements(DetLayer const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libTrackingToolsMeasurementDet.so
#16 0x00004000a948d818 in GroupedCkfTrajectoryBuilder::advanceOneLayer(TrajectorySeed const&, TempTrajectory&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const [clone .constprop.0] () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#17 0x00004000a9482490 in GroupedCkfTrajectoryBuilder::groupedLimitedCandidates(TrajectorySeed const&, TempTrajectory const&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#18 0x00004000a9484dcc in GroupedCkfTrajectoryBuilder::buildTrajectories(TrajectorySeed const&, std::vector<Trajectory, std::allocator<Trajectory> >&, unsigned int&, TrajectoryFilter const*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#19 0x00004000a564d72c in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&)::{lambda(unsigned long)#1}::operator()(unsigned long) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libRecoTrackerCkfPattern.so
#20 0x00004000a564ecf4 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libRecoTrackerCkfPattern.so
#21 0x00004000254846fc in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#22 0x000040002546aaa0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#23 0x00004000253f5f78 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#24 0x00004000253f66e0 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#25 0x0000400025915160 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libFWCoreConcurrency.so
#26 0x0000400026dc8508 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x40009c626a00, this=0x400027bf3900) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-00a1da2ea6897843151bdc54c2bd1c06/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
#27 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x400027bf3900) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-00a1da2ea6897843151bdc54c2bd1c06/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
#28 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-00a1da2ea6897843151bdc54c2bd1c06/tbb-v2021.8.0/src/tbb/task_dispatcher.cpp:168
#29 0x000040002536ec84 in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#30 0x000040002537bcc8 in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#31 0x000040002537c250 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02774/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-02-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#32 0x000000000040b9c8 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#33 0x0000400026dc0f5c in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins_b/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-00a1da2ea6897843151bdc54c2bd1c06/tbb-v2021.8.0/src/tbb/arena.cpp:694
#34 0x000000000040f73c in main::{lambda()#1}::operator()() const ()
#35 0x0000000000406fc4 in main ()

Current Modules:

Module: CkfTrackCandidateMaker:lowPtQuadStepTrackCandidates (crashed)
Module: CkfTrackCandidateMaker:lowPtQuadStepTrackCandidates
Module: CkfTrackCandidateMaker:lowPtQuadStepTrackCandidates
Module: TrackTfClassifier:initialStep
Module: CkfTrackCandidateMaker:lowPtQuadStepTrackCandidates
Module: CkfTrackCandidateMaker:lowPtQuadStepTrackCandidates
Module: CkfTrackCandidateMaker:lowPtQuadStepTrackCandidates
Module: CkfTrackCandidateMaker:lowPtQuadStepTrackCandidates

Full log: link

Mar 03 '23 07:03 iarspider

assign dqm

Mar 03 '23 07:03 iarspider

New categories assigned: dqm

@micsucmed,@rvenditti,@emanueleusai,@syuvivida,@pmandrik you have been requested to review this Pull request/Issue and eventually sign? Thanks

Mar 03 '23 07:03 cmsbuild

A new Issue was created by @iarspider .

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

Mar 03 '23 07:03 cmsbuild

assign reconstruction

Crash occurs in reconstruction code

Mar 03 '23 14:03 makortel

New categories assigned: reconstruction

@mandrenguyen,@clacaputo you have been requested to review this Pull request/Issue and eventually sign? Thanks

Mar 03 '23 14:03 cmsbuild

FYI @cms-sw/tracking-pog-l2 Looks like it's coming from lowPtQuadStepTrackCandidates

Mar 03 '23 14:03 mandrenguyen

A possibly related crash in CMSSW_13_1_X_2023-03-12-2300 250204.17 step3

Thread 5 (Thread 0x400063a89250 (LWP 928526) "cmsRun"):
#3  0x000040001961a1ec in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000040003c2d9544 in CompatibleDetToGroupAdder::add(GeometricSearchDet const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector<DetGroup, std::allocator<DetGroup> >&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libRecoTrackerTkDetLayers.so
#6  0x000040003c2df7bc in PixelForwardLayerPhase1::groupedCompatibleDetsV(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector<DetGroup, std::allocator<DetGroup> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libRecoTrackerTkDetLayers.so
#7  0x000040001eb27488 in GeometricSearchDet::groupedCompatibleDets(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libTrackingToolsDetLayers.so
#8  0x000040003c267e68 in LayerMeasurements::groupedMeasurements(DetLayer const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libTrackingToolsMeasurementDet.so
#9  0x000040009f16d818 in GroupedCkfTrajectoryBuilder::advanceOneLayer(TrajectorySeed const&, TempTrajectory&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const [clone .constprop.0] () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#10 0x000040009f162490 in GroupedCkfTrajectoryBuilder::groupedLimitedCandidates(TrajectorySeed const&, TempTrajectory const&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#11 0x000040009f166020 in GroupedCkfTrajectoryBuilder::rebuildTrajectories(TempTrajectory const&, TrajectorySeed const&, std::vector<Trajectory, std::allocator<Trajectory> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#12 0x000040009dacdad8 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&)::{lambda(unsigned long)#1}::operator()(unsigned long) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libRecoTrackerCkfPattern.so
#13 0x000040009dacecf4 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libRecoTrackerCkfPattern.so
#14 0x0000400012113f7c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#15 0x00004000120fa320 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#16 0x0000400012085ca8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#17 0x0000400012086420 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so

Thread 4 (Thread 0x400063079250 (LWP 928525) "cmsRun"):
#2  0x00004000196169cc in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000400041021eb4 in tensorflow::monitoring::CounterCell* tensorflow::monitoring::Counter<0>::GetCell<>() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/external/el8_aarch64_gcc11/lib/libtensorflow_cc.so.2
#5  0x0000400047b4dfbc in tensorflow::DirectSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*, tensorflow::thread::ThreadPoolOptions const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/external/el8_aarch64_gcc11/lib/libtensorflow_cc.so.2
#6  0x000040003e16cf14 in tensorflow::run(tensorflow::Session*, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::thread::ThreadPoolOptions const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libPhysicsToolsTensorFlow.so
#7  0x000040003e16d10c in tensorflow::run(tensorflow::Session*, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::thread::ThreadPoolInterface*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libPhysicsToolsTensorFlow.so
#8  0x000040005ea29398 in MkFitOutputConverter::computeDNNs(std::vector<TrackCandidate, std::allocator<TrackCandidate> > const&, std::vector<TrajectoryStateOnSurface, std::allocator<TrajectoryStateOnSurface> > const&, reco::BeamSpot const*, std::vector<reco::Vertex, std::allocator<reco::Vertex> > const*, tensorflow::Session const*, std::vector<float, std::allocator<float> > const&, bool) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerMkFitPlugins.so
#9  0x000040005ea2afec in MkFitOutputConverter::convertCandidates(MkFitOutputWrapper const&, mkfit::EventOfHits const&, MkFitClusterIndexToHit const&, MkFitClusterIndexToHit const&, edm::View<TrajectorySeed> const&, MagneticField const&, Propagator const&, Propagator const&, TkClonerImpl const&, std::vector<DetLayer const*, std::allocator<DetLayer const*> > const&, std::vector<mkfit::Track, std::allocator<mkfit::Track> > const&, reco::BeamSpot const*, std::vector<reco::Vertex, std::allocator<reco::Vertex> > const*, tensorflow::Session const*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerMkFitPlugins.so
#10 0x000040005ea246bc in MkFitOutputConverter::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerMkFitPlugins.so
#11 0x00004000120ffc90 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#12 0x00004000120f96f0 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so

Thread 3 (Thread 0x400062669250 (LWP 928524) "cmsRun"):
#2  0x00004000196169cc in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040005eb2e014 in mkfit::helixAtRFromIterativeCCS(Matriplex::Matriplex<float, 6, 1, 8> const&, Matriplex::Matriplex<int, 1, 1, 8> const&, Matriplex::Matriplex<float, 1, 1, 8> const&, Matriplex::Matriplex<float, 6, 1, 8>&, Matriplex::Matriplex<float, 6, 6, 8>&, Matriplex::Matriplex<int, 1, 1, 8>&, int, mkfit::PropagationFlags) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#5  0x000040005eb2f70c in mkfit::propagateHelixToRMPlex(Matriplex::MatriplexSym<float, 6, 8> const&, Matriplex::Matriplex<float, 6, 1, 8> const&, Matriplex::Matriplex<int, 1, 1, 8> const&, Matriplex::Matriplex<float, 1, 1, 8> const&, Matriplex::MatriplexSym<float, 6, 8>&, Matriplex::Matriplex<float, 6, 1, 8>&, Matriplex::Matriplex<int, 1, 1, 8>&, int, mkfit::PropagationFlags, Matriplex::Matriplex<int, 1, 1, 8> const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#6  0x000040005eb17ed4 in mkfit::kalmanPropagateAndComputeChi2(Matriplex::MatriplexSym<float, 6, 8> const&, Matriplex::Matriplex<float, 6, 1, 8> const&, Matriplex::Matriplex<int, 1, 1, 8> const&, Matriplex::MatriplexSym<float, 3, 8> const&, Matriplex::Matriplex<float, 3, 1, 8> const&, Matriplex::Matriplex<float, 1, 1, 8>&, Matriplex::Matriplex<float, 6, 1, 8>&, Matriplex::Matriplex<int, 1, 1, 8>&, int, mkfit::PropagationFlags, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#7  0x000040005eb28e24 in mkfit::MkFinder::findCandidatesCloneEngine(mkfit::LayerOfHits const&, mkfit::CandCloner&, int, int, mkfit::FindingFoos const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#8  0x000040005eb236b8 in mkfit::MkBuilder::find_tracks_in_layers(mkfit::CandCloner&, mkfit::MkFinder*, mkfit::SteeringParams::IterationType_e, int, int, int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#9  0x000040005eb239f4 in mkfit::MkBuilder::findTracksCloneEngine(mkfit::SteeringParams::IterationType_e)::{lambda(int)#1}::operator()(int) const::{lambda(tbb::detail::d1::blocked_range<int> const&)#1}::operator()(tbb::detail::d1::blocked_range<int> const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#10 0x000040005eb23e7c in tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<int>, mkfit::MkBuilder::findTracksCloneEngine(mkfit::SteeringParams::IterationType_e)::{lambda(int)#1}::operator()(int) const::{lambda(tbb::detail::d1::blocked_range<int> const&)#1}, tbb::detail::d1::auto_partitioner const>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#11 0x0000400013a58508 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x400014759800, this=0x400014883e80) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-d1db7ed5e1d50722d8d27a149fd6e0b9/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x400014883e80) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-d1db7ed5e1d50722d8d27a149fd6e0b9/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
#13 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-d1db7ed5e1d50722d8d27a149fd6e0b9/tbb-v2021.8.0/src/tbb/task_dispatcher.cpp:168
#14 0x000040005eb1f8a0 in void tbb::detail::d1::dynamic_grainsize_mode<tbb::detail::d1::adaptive_mode<tbb::detail::d1::auto_partition_type> >::work_balance<tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned long>, tbb::detail::d2::parallel_for_body_wrapper<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, mkfit::MkBuilder::findTracksCloneEngine(mkfit::SteeringParams::IterationType_e)::{lambda(int)#1}, int>, tbb::detail::d1::auto_partitioner const>, tbb::detail::d1::blocked_range<unsigned long> >(tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned long>, tbb::detail::d2::parallel_for_body_wrapper<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, mkfit::MkBuilder::findTracksCloneEngine(mkfit::SteeringParams::IterationType_e)::{lambda(int)#1}, int>, tbb::detail::d1::auto_partitioner const>&, tbb::detail::d1::blocked_range<unsigned long>&, tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#15 0x000040005eb1fcfc in tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned long>, tbb::detail::d2::parallel_for_body_wrapper<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, mkfit::MkBuilder::findTracksCloneEngine(mkfit::SteeringParams::IterationType_e)::{lambda(int)#1}, int>, tbb::detail::d1::auto_partitioner const>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#16 0x0000400013a58508 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x400014758700, this=0x400014883e80) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-d1db7ed5e1d50722d8d27a149fd6e0b9/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
#17 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x400014883e80) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-d1db7ed5e1d50722d8d27a149fd6e0b9/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
#18 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-d1db7ed5e1d50722d8d27a149fd6e0b9/tbb-v2021.8.0/src/tbb/task_dispatcher.cpp:168
#19 0x000040005eb1ce68 in tbb::detail::d2::for_each_root_task<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, mkfit::MkBuilder::findTracksCloneEngine(mkfit::SteeringParams::IterationType_e)::{lambda(int)#1}, int, std::random_access_iterator_tag>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#20 0x0000400013a58508 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x400062667d80, this=0x400014883e80) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-d1db7ed5e1d50722d8d27a149fd6e0b9/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
#21 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x400014883e80) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-d1db7ed5e1d50722d8d27a149fd6e0b9/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
#22 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-d1db7ed5e1d50722d8d27a149fd6e0b9/tbb-v2021.8.0/src/tbb/task_dispatcher.cpp:168
#23 0x000040005eb1e064 in mkfit::MkBuilder::findTracksCloneEngine(mkfit::SteeringParams::IterationType_e) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#24 0x000040005eabb940 in mkfit::run_OneIteration(mkfit::TrackerInfo const&, mkfit::IterationConfig const&, mkfit::EventOfHits const&, std::vector<std::vector<bool, std::allocator<bool> > const*, std::allocator<std::vector<bool, std::allocator<bool> > const*> > const&, mkfit::MkBuilder&, std::vector<mkfit::Track, std::allocator<mkfit::Track> >&, std::vector<mkfit::Track, std::allocator<mkfit::Track> >&, bool, bool, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCMS.so
#25 0x000040005ea382b0 in tbb::detail::d1::task_arena_function<MkFitProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const::{lambda()#1}, void>::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerMkFitPlugins.so
#26 0x0000400013a51554 in operator() (__closure=<optimized out>) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-d1db7ed5e1d50722d8d27a149fd6e0b9/tbb-v2021.8.0/src/tbb/arena.cpp:767
#27 tbb::detail::d0::try_call_proxy<tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, intptr_t)::<lambda()> >::on_completion<tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, intptr_t)::<lambda()> > (on_completion_body=..., this=<optimized out>) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-d1db7ed5e1d50722d8d27a149fd6e0b9/tbb-v2021.8.0/src/tbb/../../include/oneapi/tbb/detail/_template_helpers.h:230
#28 tbb::detail::r1::isolate_within_arena (d=..., isolation=<optimized out>) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-d1db7ed5e1d50722d8d27a149fd6e0b9/tbb-v2021.8.0/src/tbb/arena.cpp:768
#29 0x000040005ea3d2e4 in MkFitProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerMkFitPlugins.so
#30 0x00004000120ffc90 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#31 0x00004000120f96f0 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#32 0x0000400012085ca8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#33 0x0000400012086420 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so

Thread 1 (Thread 0x400013bdca30 (LWP 926082) "cmsRun"):
#2  0x00004000196169cc in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040003cadf008 in CellularAutomaton::evolve(unsigned int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libRecoPixelVertexingPixelTriplets.so
#5  0x000040003cad64a0 in CAHitQuadrupletGenerator::hitNtuplets(IntermediateHitDoublets const&, std::vector<OrderedHitSeeds, std::allocator<OrderedHitSeeds> >&, SeedingLayerSetsHits const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libRecoPixelVertexingPixelTriplets.so
#6  0x00004000b0075878 in CAHitNtupletEDProducerT<CAHitQuadrupletGenerator>::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/pluginRecoPixelVertexingPixelTripletsPlugins.so
#7  0x0000400012113f7c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#8  0x00004000120fa320 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#9  0x0000400012085ca8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#10 0x0000400012086420 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week0/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-12-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so

Current Modules:
Module: CkfTrackCandidateMaker:lowPtTripletStepTrackCandidates (crashed)
Module: MkFitProducer:detachedTripletStepTrackCandidatesMkFit
Module: MkFitOutputConverter:highPtTripletStepTrackCandidates
Module: CAHitQuadrupletEDProducer:detachedQuadStepHitQuadruplets

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_aarch64_gcc11/CMSSW_13_1_X_2023-03-12-2300/pyRelValMatrixLogs/run/250204.17_QQH1352T_13UP17/step3_QQH1352T_13UP17.log#/

Mar 13 '23 09:03 makortel

it would be nice to get a stack trace with line numbers

Mar 13 '23 12:03 slava77

are other builds showing a crash as well, or is this ARM-specific?

Mar 13 '23 12:03 slava77

type tracking

Mar 13 '23 12:03 slava77

These are specific to ARM (as the issue title tries to convey)

Mar 13 '23 12:03 makortel

@slava77 the crash was only for ARM, and the test didn't crash in any recent IB, so could be a one-time issue. The RelVal @makortel mentioned also only failed once so far.

Mar 13 '23 12:03 iarspider

Happened again in RelVal 1001.2:

Thread 4 (Thread 0x400069609250 (LWP 2325874) "cmsRun"):
#0  0x000040001a2fb5cc in poll () from /lib64/libc.so.6
#1  0x000040001f6fc39c in full_read.constprop () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02778/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-29-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#2  0x000040001f6cbbc4 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02778/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-29-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3  0x000040001f6c71f8 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02778/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-29-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00004000b08bd764 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&)::{lambda(unsigned long)#1}::operator()(unsigned long) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02778/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-29-2300/lib/el8_aarch64_gcc11/libRecoTrackerCkfPattern.so
#6  0x00004000b08bedc4 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02778/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-29-2300/lib/el8_aarch64_gcc11/libRecoTrackerCkfPattern.so
#7  0x00004000183a43ac in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02778/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-29-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#8  0x000040001838a750 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02778/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-29-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#9  0x0000400018315f48 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02778/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-29-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#10 0x00004000183166b0 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02778/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-29-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#11 0x0000400018835160 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02778/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_1_X_2023-03-29-2300/lib/el8_aarch64_gcc11/libFWCoreConcurrency.so
#12 0x0000400019ce2a64 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x4000c9d1f300, waiter=<synthetic pointer>..., this=0x40001ab13e00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_13_1_0_pre1_LossyCompression-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre1_LossyCompression-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-d1db7ed5e1d50722d8d27a149fd6e0b9/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
#13 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x40001ab13e00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_13_1_0_pre1_LossyCompression-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre1_LossyCompression-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-d1db7ed5e1d50722d8d27a149fd6e0b9/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
#14 tbb::detail::r1::arena::process (tls=..., this=0x40001ab13780) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_13_1_0_pre1_LossyCompression-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre1_LossyCompression-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-d1db7ed5e1d50722d8d27a149fd6e0b9/tbb-v2021.8.0/src/tbb/arena.cpp:137
#15 tbb::detail::r1::market::process (this=0x40001ab3b080, j=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_13_1_0_pre1_LossyCompression-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre1_LossyCompression-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-d1db7ed5e1d50722d8d27a149fd6e0b9/tbb-v2021.8.0/src/tbb/market.cpp:599
#16 0x0000400019ceac5c in tbb::detail::r1::rml::private_worker::run (this=0x40001e67c100) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_13_1_0_pre1_LossyCompression-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre1_LossyCompression-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-d1db7ed5e1d50722d8d27a149fd6e0b9/tbb-v2021.8.0/src/tbb/private_server.cpp:271
#17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x40001e67c100) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_13_1_0_pre1_LossyCompression-el8_aarch64_gcc11/build/CMSSW_13_1_0_pre1_LossyCompression-build/BUILD/el8_aarch64_gcc11/external/tbb/v2021.8.0-d1db7ed5e1d50722d8d27a149fd6e0b9/tbb-v2021.8.0/src/tbb/private_server.cpp:221
#18 0x000040001a1f78b8 in start_thread () from /lib64/libpthread.so.0
#19 0x000040001a253afc in thread_start () from /lib64/libc.so.6

Mar 31 '23 07:03 iarspider

Notice that it doesn't fail every time, and it's hard to reproduce

Mar 31 '23 09:03 iarspider

Notice that it doesn't fail every time, and it's hard to reproduce

it may be practical to have the ARM builds with line info (-g). Or is the cost too large?

Mar 31 '23 12:03 slava77

Another one in workflow 136.804 step 3 in CMSSW_13_2_X_2023-06-22-2300 on el8_aarch64_gcc11

 Thread 5 (Thread 0x400071a09260 (LWP 943510) "cmsRun"):
#2  0x00004000221633f8 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040004651c258 in TrajectoryStateOnSurface (anonymous namespace)::lupdate<2u>(TrajectoryStateOnSurface const&, TrackingRecHit const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libTrackingToolsKalmanUpdators.so
#5  0x0000400046520220 in KFUpdator::update(TrajectoryStateOnSurface const&, TrackingRecHit const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libTrackingToolsKalmanUpdators.so
#6  0x0000400065290f7c in KFTrajectorySmoother::trajectory(Trajectory const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libTrackingToolsTrackFitters.so
#7  0x0000400065219808 in (anonymous namespace)::KFFittingSmoother::smoothingStep(Trajectory&&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/pluginTrackingToolsTrackFittersPlugins.so
#8  0x000040006521d7b8 in (anonymous namespace)::KFFittingSmoother::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/pluginTrackingToolsTrackFittersPlugins.so
#9  0x0000400065216d78 in (anonymous namespace)::FlexibleKFFittingSmoother::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/pluginTrackingToolsTrackFittersPlugins.so
#10 0x00004000950b7dbc in TrackProducerAlgorithm<reco::Track>::buildTrack(TrajectoryFitter const*, Propagator const*, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > >&, TrajectoryStateOnSurface&, TrajectorySeed const&, float, reco::BeamSpot const&, edm::RefToBase<TrajectorySeed>, int, signed char) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libRecoTrackerTrackProducer.so
#11 0x0000400094f8f61c in TrackProducerAlgorithm<reco::Track>::runWithCandidate(TrackingGeometry const*, MagneticField const*, std::vector<TrackCandidate, std::allocator<TrackCandidate> > const&, TrajectoryFitter const*, Propagator const*, TransientTrackingRecHitBuilder const*, reco::BeamSpot const&, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerTrackProducerPlugins.so
#12 0x0000400094f8ddac in TrackProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerTrackProducerPlugins.so
#13 0x000040001afc4ccc in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#14 0x000040001afaaeb0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so

Thread 4 (Thread 0x40006ebb9260 (LWP 943509) "cmsRun"):
#3  0x0000400022167cc8 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00004000abf5dd18 in GroupedCkfTrajectoryBuilder::advanceOneLayer(TrajectorySeed const&, TempTrajectory&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const [clone .constprop.0] () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#6  0x00004000abf52550 in GroupedCkfTrajectoryBuilder::groupedLimitedCandidates(TrajectorySeed const&, TempTrajectory const&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#7  0x00004000abf54e3c in GroupedCkfTrajectoryBuilder::buildTrajectories(TrajectorySeed const&, std::vector<Trajectory, std::allocator<Trajectory> >&, unsigned int&, TrajectoryFilter const*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerCkfPatternPlugins.so
#8  0x00004000aa4fd874 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&)::{lambda(unsigned long)#1}::operator()(unsigned long) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libRecoTrackerCkfPattern.so
#9  0x00004000aa4feda4 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libRecoTrackerCkfPattern.so
#10 0x000040001afc4ccc in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#11 0x000040001afaaeb0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so

Thread 3 (Thread 0x40006e1a9260 (LWP 943508) "cmsRun"):
#2  0x00004000221633f8 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000400069d4b464 in mkfit::applyMaterialEffects(Matriplex::Matriplex<float, 1, 1, 8> const&, Matriplex::Matriplex<float, 1, 1, 8> const&, Matriplex::Matriplex<float, 1, 1, 8> const&, Matriplex::MatriplexSym<float, 6, 8>&, Matriplex::Matriplex<float, 6, 1, 8>&, int, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#5  0x0000400069d4b7ac in mkfit::propagateHelixToRMPlex(Matriplex::MatriplexSym<float, 6, 8> const&, Matriplex::Matriplex<float, 6, 1, 8> const&, Matriplex::Matriplex<int, 1, 1, 8> const&, Matriplex::Matriplex<float, 1, 1, 8> const&, Matriplex::MatriplexSym<float, 6, 8>&, Matriplex::Matriplex<float, 6, 1, 8>&, Matriplex::Matriplex<int, 1, 1, 8>&, int, mkfit::PropagationFlags const&, Matriplex::Matriplex<int, 1, 1, 8> const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#6  0x0000400069d4412c in mkfit::MkFinder::bkFitFitTracks(mkfit::EventOfHits const&, mkfit::SteeringParams const&, int, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#7  0x0000400069d36110 in mkfit::MkBuilder::fit_cands(mkfit::MkFinder*, int, int, int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#8  0x0000400069d3634c in tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<int>, mkfit::MkBuilder::backwardFit()::{lambda(int)#1}::operator()(int) const::{lambda(tbb::detail::d1::blocked_range<int> const&)#1}, tbb::detail::d1::auto_partitioner const>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#9  0x000040001ca08774 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x4000de133000, this=0x40001d8e3e80) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-d9b6b79f96fc04849cfc544d9852057d/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#10 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x40001d8e3e80) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-d9b6b79f96fc04849cfc544d9852057d/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#11 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-d9b6b79f96fc04849cfc544d9852057d/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#12 0x0000400069d3aaa8 in void tbb::detail::d1::dynamic_grainsize_mode<tbb::detail::d1::adaptive_mode<tbb::detail::d1::auto_partition_type> >::work_balance<tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned long>, tbb::detail::d2::parallel_for_body_wrapper<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, mkfit::MkBuilder::backwardFit()::{lambda(int)#1}, int>, tbb::detail::d1::auto_partitioner const>, tbb::detail::d1::blocked_range<unsigned long> >(tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned long>, tbb::detail::d2::parallel_for_body_wrapper<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, mkfit::MkBuilder::backwardFit()::{lambda(int)#1}, int>, tbb::detail::d1::auto_partitioner const>&, tbb::detail::d1::blocked_range<unsigned long>&, tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#13 0x0000400069d3b0bc in tbb::detail::d1::start_for<tbb::detail::d1::blocked_range<unsigned long>, tbb::detail::d2::parallel_for_body_wrapper<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, mkfit::MkBuilder::backwardFit()::{lambda(int)#1}, int>, tbb::detail::d1::auto_partitioner const>::execute(tbb::detail::d1::execution_data&) [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#14 0x000040001ca08774 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x4000de132800, this=0x40001d8e3e80) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-d9b6b79f96fc04849cfc544d9852057d/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#15 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x40001d8e3e80) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-d9b6b79f96fc04849cfc544d9852057d/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#16 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-d9b6b79f96fc04849cfc544d9852057d/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#17 0x0000400069d2cf3c in tbb::detail::d2::for_each_root_task<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, mkfit::MkBuilder::backwardFit()::{lambda(int)#1}, int, std::random_access_iterator_tag>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#18 0x000040001ca08774 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x40006e1a7d80, this=0x40001d8e3e80) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-d9b6b79f96fc04849cfc544d9852057d/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#19 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x40001d8e3e80) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-d9b6b79f96fc04849cfc544d9852057d/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#20 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-d9b6b79f96fc04849cfc544d9852057d/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#21 0x0000400069d2d978 in void tbb::detail::d2::parallel_for_each<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, mkfit::MkBuilder::backwardFit()::{lambda(int)#1}>(__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, __gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, mkfit::MkBuilder::backwardFit()::{lambda(int)#1} const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#22 0x0000400069d2f5fc in mkfit::MkBuilder::backwardFit() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCore.so
#23 0x0000400069cbba2c in mkfit::run_OneIteration(mkfit::TrackerInfo const&, mkfit::IterationConfig const&, mkfit::EventOfHits const&, std::vector<std::vector<bool, std::allocator<bool> > const*, std::allocator<std::vector<bool, std::allocator<bool> > const*> > const&, mkfit::MkBuilder&, std::vector<mkfit::Track, std::allocator<mkfit::Track> >&, std::vector<mkfit::Track, std::allocator<mkfit::Track> >&, bool, bool, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libRecoTrackerMkFitCMS.so
#24 0x0000400069c292c0 in tbb::detail::d1::task_arena_function<MkFitProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const::{lambda()#1}, void>::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerMkFitPlugins.so
#25 0x000040001ca016e4 in operator() (__closure=<optimized out>) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-d9b6b79f96fc04849cfc544d9852057d/tbb-v2021.9.0/src/tbb/arena.cpp:757
#26 tbb::detail::d0::try_call_proxy<tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, intptr_t)::<lambda()> >::on_completion<tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, intptr_t)::<lambda()> > (on_completion_body=..., this=<optimized out>) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-d9b6b79f96fc04849cfc544d9852057d/tbb-v2021.9.0/src/tbb/../../include/oneapi/tbb/detail/_template_helpers.h:230
#27 tbb::detail::r1::isolate_within_arena (d=..., isolation=<optimized out>) at /data/cmsbld/jenkins_a/workspace/build-any-ib/w/BUILD/el8_aarch64_gcc11/external/tbb/v2021.9.0-d9b6b79f96fc04849cfc544d9852057d/tbb-v2021.9.0/src/tbb/arena.cpp:758
#28 0x0000400069c2e3c4 in MkFitProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerMkFitPlugins.so
#29 0x000040001afb05c0 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#30 0x000040001afaa280 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so

Thread 1 (Thread 0x40001cbacf30 (LWP 942338) "cmsRun"):
#2  0x00004000221633f8 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040006355d578 in tensorflow::OpKernelContext::OpKernelContext(tensorflow::OpKernelContext::Params*, int) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/external/el8_aarch64_gcc11/lib/libtensorflow_framework.so.2
#5  0x000040006379f0b8 in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::Process(tensorflow::SimplePropagatorState::TaggedNode, long) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/external/el8_aarch64_gcc11/lib/libtensorflow_framework.so.2
#6  0x000040004b76e4a4 in tensorflow::thread::ThreadPool::Schedule(std::function<void ()>) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/external/el8_aarch64_gcc11/lib/libtensorflow_cc.so.2
#7  0x0000400052319bdc in std::_Function_handler<void (std::function<void ()>), tensorflow::DirectSession::RunInternal(long, tensorflow::RunOptions const&, tensorflow::CallFrameInterface*, tensorflow::DirectSession::ExecutorsAndKeys*, tensorflow::RunMetadata*, tensorflow::thread::ThreadPoolOptions const&)::{lambda(std::function<void ()>)#6}>::_M_invoke(std::_Any_data const&, std::function<void ()>&&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/external/el8_aarch64_gcc11/lib/libtensorflow_cc.so.2
#8  0x0000400063792e3c in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::ScheduleReady(absl::lts_20210324::InlinedVector<tensorflow::SimplePropagatorState::TaggedNode, 8ul, std::allocator<tensorflow::SimplePropagatorState::TaggedNode> >*, tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/external/el8_aarch64_gcc11/lib/libtensorflow_framework.so.2
#9  0x000040006379790c in tensorflow::(anonymous namespace)::ExecutorImpl::RunAsync(tensorflow::Executor::Args const&, std::function<void (tensorflow::Status const&)>) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/external/el8_aarch64_gcc11/lib/libtensorflow_framework.so.2
#10 0x00004000523290e0 in tensorflow::DirectSession::RunInternal(long, tensorflow::RunOptions const&, tensorflow::CallFrameInterface*, tensorflow::DirectSession::ExecutorsAndKeys*, tensorflow::RunMetadata*, tensorflow::thread::ThreadPoolOptions const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/external/el8_aarch64_gcc11/lib/libtensorflow_cc.so.2
#11 0x000040005232b170 in tensorflow::DirectSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*, tensorflow::thread::ThreadPoolOptions const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/external/el8_aarch64_gcc11/lib/libtensorflow_cc.so.2
#12 0x000040004892cde4 in tensorflow::run(tensorflow::Session*, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::thread::ThreadPoolOptions const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libPhysicsToolsTensorFlow.so
#13 0x000040004892cfdc in tensorflow::run(tensorflow::Session*, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::thread::ThreadPoolInterface*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libPhysicsToolsTensorFlow.so
#14 0x0000400069c197e8 in MkFitOutputConverter::computeDNNs(std::vector<TrackCandidate, std::allocator<TrackCandidate> > const&, std::vector<TrajectoryStateOnSurface, std::allocator<TrajectoryStateOnSurface> > const&, reco::BeamSpot const*, std::vector<reco::Vertex, std::allocator<reco::Vertex> > const*, tensorflow::Session const*, std::vector<float, std::allocator<float> > const&, bool) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerMkFitPlugins.so
#15 0x0000400069c1b43c in MkFitOutputConverter::convertCandidates(MkFitOutputWrapper const&, mkfit::EventOfHits const&, MkFitClusterIndexToHit const&, MkFitClusterIndexToHit const&, edm::View<TrajectorySeed> const&, MagneticField const&, Propagator const&, Propagator const&, TkClonerImpl const&, std::vector<DetLayer const*, std::allocator<DetLayer const*> > const&, std::vector<mkfit::Track, std::allocator<mkfit::Track> > const&, reco::BeamSpot const*, std::vector<reco::Vertex, std::allocator<reco::Vertex> > const*, tensorflow::Session const*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerMkFitPlugins.so
#16 0x0000400069c1d51c in MkFitOutputConverter::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/pluginRecoTrackerMkFitPlugins.so
#17 0x000040001afb05c0 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so
#18 0x000040001afaa280 in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02790/el8_aarch64_gcc11/cms/cmssw/CMSSW_13_2_X_2023-06-22-2300/lib/el8_aarch64_gcc11/libFWCoreFramework.so

Current Modules:
Module: CkfTrackCandidateMaker:lowPtTripletStepTrackCandidates (crashed)
Module: MkFitOutputConverter:highPtTripletStepTrackCandidates
Module: TrackProducer:highPtTripletStepTracks
Module: MkFitProducer:highPtTripletStepTrackCandidatesMkFit

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_aarch64_gcc11/CMSSW_13_2_X_2023-06-22-2300/pyRelValMatrixLogs/run/136.804_RunHLTPhy2017D/step3_RunHLTPhy2017D.log#/

Jun 23 '23 13:06 makortel

some cleanups to bad states during tracking were done in #42320 , merged July 20. Is this crash still happening?

Jul 29 '23 16:07 slava77

It seems there is a new occurrence of this crash in relval 250204.17 for el8_aarch64_gcc11. See full log.

Aug 08 '23 09:08 aandvalenzuela

It seems there is a new occurrence of this crash in relval 250204.17 for el8_aarch64_gcc11. See full log.

This is in a very different code path MkFitProducer crashed, instead of CkfTrackCandidateMaker. This looks more similar to #42071

Aug 08 '23 12:08 slava77

Another failure: log

Thread 9 (Thread 0x40006f419250 (LWP 3913943) "cmsRun"):
#0  0x00004000227a4b3c in nanosleep () from /lib64/libc.so.6
#1  0x00004000227a49e0 in sleep () from /lib64/libc.so.6
#2  0x00004000287c927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00004000427bde78 in BasicTrajectoryState::~BasicTrajectoryState() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libTrackingToolsTrajectoryState.so
#5  0x000040005ed0c858 in Trajectory::~Trajectory() [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginTrackingToolsTrackFittersPlugins.so
#6  0x000040005ed0cb78 in (anonymous namespace)::KFFittingSmoother::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginTrackingToolsTrackFittersPlugins.so
#7  0x000040005ed06488 in (anonymous namespace)::FlexibleKFFittingSmoother::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginTrackingToolsTrackFittersPlugins.so
#8  0x00004000969cbb90 in TrackProducerAlgorithm<reco::Track>::buildTrack(TrajectoryFitter const*, Propagator const*, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > >&, TrajectoryStateOnSurface&, TrajectorySeed const&, float, reco::BeamSpot const&, edm::RefToBase<TrajectorySeed>, int, signed char) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerTrackProducer.so
#9  0x00004000a2a4d590 in TrackProducerAlgorithm<reco::Track>::runWithCandidate(TrackingGeometry const*, MagneticField const*, std::vector<TrackCandidate, std::allocator<TrackCandidate> > const&, TrajectoryFitter const*, Propagator const*, TransientTrackingRecHitBuilder const*, reco::BeamSpot const&, std::vector<AlgoProductTraits<reco::Track>::AlgoProduct, std::allocator<AlgoProductTraits<reco::Track>::AlgoProduct> >&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#10 0x00004000a2a4bf18 in TrackProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginRecoTrackerTrackProducerPlugins.so
#11 0x00004000206da44c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#12 0x00004000206befd0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#13 0x000040002064c074 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#14 0x000040002064c894 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#15 0x0000400020be50dc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#16 0x0000400022132dfc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x4000cd873b00, waiter=<synthetic pointer>..., this=0x4000251b3c80) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#17 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x4000251b3c80) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#18 tbb::detail::r1::arena::process (tls=..., this=0x4000251b2f00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:137
#19 tbb::detail::r1::market::process (this=0x4000251db080, j=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/market.cpp:599
#20 0x000040002213af2c in tbb::detail::r1::rml::private_worker::run (this=0x400027bbbf00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#21 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400027bbbf00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#22 0x00004000226c78b8 in start_thread () from /lib64/libpthread.so.0
#23 0x0000400022723afc in thread_start () from /lib64/libc.so.6
Thread 8 (Thread 0x40006be29250 (LWP 3913942) "cmsRun"):
#0  0x00004000227a4b3c in nanosleep () from /lib64/libc.so.6
#1  0x00004000227a49e0 in sleep () from /lib64/libc.so.6
#2  0x00004000287c927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000400020927c90 in edm::HandleBase::id() const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libDataFormatsCommon.so
#5  0x000040004b499e20 in TkStripMeasurementDet::empty(MeasurementTrackerEvent const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginRecoTrackerMeasurementDetPlugins.so
#6  0x000040004b494f14 in TkGluedMeasurementDet::measurements(TrajectoryStateOnSurface const&, MeasurementEstimator const&, MeasurementTrackerEvent const&, tracking::TempMeasurements&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginRecoTrackerMeasurementDetPlugins.so
#7  0x000040004b597b68 in LayerMeasurements::groupedMeasurements(DetLayer const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libTrackingToolsMeasurementDet.so
#8  0x00004000a2bcceec in GroupedCkfTrajectoryBuilder::advanceOneLayer(TrajectorySeed const&, TempTrajectory&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const [clone .constprop.0] () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#9  0x00004000a2bc3108 in GroupedCkfTrajectoryBuilder::groupedLimitedCandidates(TrajectorySeed const&, TempTrajectory const&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#10 0x00004000a2bc49e4 in GroupedCkfTrajectoryBuilder::buildTrajectories(TrajectorySeed const&, std::vector<Trajectory, std::allocator<Trajectory> >&, unsigned int&, TrajectoryFilter const*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#11 0x0000400096a2b3e4 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&)::{lambda(unsigned long)#1}::operator()(unsigned long) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerCkfPattern.so
#12 0x0000400096a2c47c in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerCkfPattern.so
#13 0x00004000206da44c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#14 0x00004000206befd0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#15 0x000040002064c074 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#16 0x000040002064c894 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#17 0x0000400020be50dc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#18 0x0000400022132dfc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x4000cd96f100, waiter=<synthetic pointer>..., this=0x4000251b3b00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#19 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x4000251b3b00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#20 tbb::detail::r1::arena::process (tls=..., this=0x4000251b2f00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:137
#21 tbb::detail::r1::market::process (this=0x4000251db080, j=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/market.cpp:599
#22 0x000040002213af2c in tbb::detail::r1::rml::private_worker::run (this=0x400027bbc000) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#23 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400027bbc000) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#24 0x00004000226c78b8 in start_thread () from /lib64/libpthread.so.0
#25 0x0000400022723afc in thread_start () from /lib64/libc.so.6
Thread 7 (Thread 0x40006ea09250 (LWP 3913941) "cmsRun"):
#0  0x00004000227ca91c in poll () from /lib64/libc.so.6
#1  0x000040002881333c in full_read.constprop () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00004000287cc5e4 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x00004000287ccf5c in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000400021c20bc8 in cache_bin_dalloc_easy (ptr=0x40014fcb8560, bin=<optimized out>) at include/jemalloc/internal/cache_bin.h:435
#6  tcache_dalloc_small (slow_path=false, binind=<optimized out>, ptr=0x40014fcb8560, tcache=0x40006ea0f1d8, tsd=0x40006ea0ee40) at include/jemalloc/internal/tcache_inlines.h:158
#7  arena_sdalloc (slow_path=<optimized out>, caller_alloc_ctx=<optimized out>, tcache=<optimized out>, size=<optimized out>, ptr=<optimized out>, tsdn=<optimized out>) at include/jemalloc/internal/arena_inlines_b.h:418
#8  isdalloct (slow_path=<optimized out>, alloc_ctx=<optimized out>, tcache=<optimized out>, size=<optimized out>, ptr=<optimized out>, tsdn=<optimized out>) at include/jemalloc/internal/jemalloc_internal_inlines_c.h:133
#9  isfree (slow_path=<optimized out>, tcache=<optimized out>, usize=<optimized out>, ptr=<optimized out>, tsd=<optimized out>) at src/jemalloc.c:2982
#10 je_sdallocx_default (ptr=0x40014fcb8560, size=<optimized out>, flags=<optimized out>) at src/jemalloc.c:3924
#11 0x0000400021c71310 in sizedDeleteImpl (size=<optimized out>, ptr=<optimized out>) at src/jemalloc_cpp.cpp:195
#12 operator delete (ptr=<optimized out>, size=<optimized out>) at src/jemalloc_cpp.cpp:200
#13 0x0000400096a22528 in CachingSeedCleanerBySharedInput::done() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerCkfPattern.so
#14 0x0000400096a2c4a8 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerCkfPattern.so
#15 0x00004000206da44c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#16 0x00004000206befd0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#17 0x000040002064c074 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#18 0x000040002064c894 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#19 0x0000400020be50dc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#20 0x0000400022132dfc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x4000cd837400, waiter=<synthetic pointer>..., this=0x4000251b3b80) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#21 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x4000251b3b80) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#22 tbb::detail::r1::arena::process (tls=..., this=0x4000251b2f00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:137
#23 tbb::detail::r1::market::process (this=0x4000251db080, j=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/market.cpp:599
#24 0x000040002213af2c in tbb::detail::r1::rml::private_worker::run (this=0x400027bbbe00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#25 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400027bbbe00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#26 0x00004000226c78b8 in start_thread () from /lib64/libpthread.so.0
#27 0x0000400022723afc in thread_start () from /lib64/libc.so.6
Thread 6 (Thread 0x40006b419250 (LWP 3913940) "cmsRun"):
#0  0x00004000227a4b3c in nanosleep () from /lib64/libc.so.6
#1  0x00004000227a49e0 in sleep () from /lib64/libc.so.6
#2  0x00004000287c927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00004000225e3280 in __atan2_finite () from /lib64/libm.so.6
#5  0x00004000615cebe4 in (anonymous namespace)::TfDnn::operator()(std::vector<reco::Track, std::allocator<reco::Track> > const&, reco::BeamSpot const&, std::vector<reco::Vertex, std::allocator<reco::Vertex> > const&) const [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginRecoTrackerFinalTrackSelectorsPlugins.so
#6  0x00004000615d0c9c in TrackMVAClassifier<(anonymous namespace)::TfDnn, void>::computeMVA(std::vector<reco::Track, std::allocator<reco::Track> > const&, reco::BeamSpot const&, std::vector<reco::Vertex, std::allocator<reco::Vertex> > const&, std::vector<std::pair<float, bool>, std::allocator<std::pair<float, bool> > >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginRecoTrackerFinalTrackSelectorsPlugins.so
#7  0x000040006165d0a0 in TrackMVAClassifierBase::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerFinalTrackSelectors.so
#8  0x00004000206da44c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#9  0x00004000206befd0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#10 0x000040002064c074 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#11 0x000040002064c894 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#12 0x0000400020be50dc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#13 0x0000400022132dfc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x4000cd8c7700, waiter=<synthetic pointer>..., this=0x4000251b3a80) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#14 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x4000251b3a80) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#15 tbb::detail::r1::arena::process (tls=..., this=0x4000251b2f00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:137
#16 tbb::detail::r1::market::process (this=0x4000251db080, j=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/market.cpp:599
#17 0x000040002213af2c in tbb::detail::r1::rml::private_worker::run (this=0x400027bbbe80) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#18 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400027bbbe80) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#19 0x00004000226c78b8 in start_thread () from /lib64/libpthread.so.0
#20 0x0000400022723afc in thread_start () from /lib64/libc.so.6
Thread 5 (Thread 0x400069a09250 (LWP 3913936) "cmsRun"):
#0  0x00004000227a4b3c in nanosleep () from /lib64/libc.so.6
#1  0x00004000227a49e0 in sleep () from /lib64/libc.so.6
#2  0x00004000287c927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040004277db60 in HelixBarrelPlaneCrossingByCircle::chooseSolution(Basic2DVector<double> const&, Basic2DVector<double> const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libTrackingToolsGeomPropagators.so
#5  0x00004000427837a0 in HelixBarrelPlaneCrossingByCircle::pathLength(Plane const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libTrackingToolsGeomPropagators.so
#6  0x000040004b631d28 in TOBRod::groupedCompatibleDetsV(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector<DetGroup, std::allocator<DetGroup> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerTkDetLayers.so
#7  0x000040004b629154 in CompatibleDetToGroupAdder::add(GeometricSearchDet const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector<DetGroup, std::allocator<DetGroup> >&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerTkDetLayers.so
#8  0x000040004b62f560 in TBLayer::groupedCompatibleDetsV(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector<DetGroup, std::allocator<DetGroup> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerTkDetLayers.so
#9  0x0000400042737508 in GeometricSearchDet::groupedCompatibleDets(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libTrackingToolsDetLayers.so
#10 0x000040004b597754 in LayerMeasurements::groupedMeasurements(DetLayer const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libTrackingToolsMeasurementDet.so
#11 0x00004000a2bcceec in GroupedCkfTrajectoryBuilder::advanceOneLayer(TrajectorySeed const&, TempTrajectory&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const [clone .constprop.0] () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#12 0x00004000a2bc3108 in GroupedCkfTrajectoryBuilder::groupedLimitedCandidates(TrajectorySeed const&, TempTrajectory const&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#13 0x00004000a2bc49e4 in GroupedCkfTrajectoryBuilder::buildTrajectories(TrajectorySeed const&, std::vector<Trajectory, std::allocator<Trajectory> >&, unsigned int&, TrajectoryFilter const*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#14 0x0000400096a2b3e4 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&)::{lambda(unsigned long)#1}::operator()(unsigned long) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerCkfPattern.so
#15 0x0000400096a2c47c in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerCkfPattern.so
#16 0x00004000206da44c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#17 0x00004000206befd0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#18 0x000040002064c074 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#19 0x000040002064c894 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#20 0x0000400020be50dc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#21 0x0000400022132dfc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x4000250a0a00, waiter=<synthetic pointer>..., this=0x4000251b3980) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#22 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x4000251b3980) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#23 tbb::detail::r1::arena::process (tls=..., this=0x4000251b2f00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:137
#24 tbb::detail::r1::market::process (this=0x4000251db080, j=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/market.cpp:599
#25 0x000040002213af2c in tbb::detail::r1::rml::private_worker::run (this=0x400027bbbf80) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#26 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400027bbbf80) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#27 0x00004000226c78b8 in start_thread () from /lib64/libpthread.so.0
#28 0x0000400022723afc in thread_start () from /lib64/libc.so.6
Thread 4 (Thread 0x400065879250 (LWP 3913935) "cmsRun"):
#0  0x00004000227a4b3c in nanosleep () from /lib64/libc.so.6
#1  0x00004000227a49e0 in sleep () from /lib64/libc.so.6
#2  0x00004000287c927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000400096a224c4 in CachingSeedCleanerBySharedInput::done() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerCkfPattern.so
#5  0x0000400096a2c4a8 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerCkfPattern.so
#6  0x00004000206da44c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#7  0x00004000206befd0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#8  0x000040002064c074 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#9  0x000040002064c894 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#10 0x0000400020be50dc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#11 0x0000400022132dfc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x4000cd8e4500, waiter=<synthetic pointer>..., this=0x4000251b3c00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#12 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x4000251b3c00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#13 tbb::detail::r1::arena::process (tls=..., this=0x4000251b2f00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:137
#14 tbb::detail::r1::market::process (this=0x4000251db080, j=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/market.cpp:599
#15 0x000040002213af2c in tbb::detail::r1::rml::private_worker::run (this=0x400027bbc100) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#16 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400027bbc100) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#17 0x00004000226c78b8 in start_thread () from /lib64/libpthread.so.0
#18 0x0000400022723afc in thread_start () from /lib64/libc.so.6
Thread 3 (Thread 0x400064e69250 (LWP 3913934) "cmsRun"):
#0  0x00004000227a4b3c in nanosleep () from /lib64/libc.so.6
#1  0x00004000227a49e0 in sleep () from /lib64/libc.so.6
#2  0x00004000287c927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000040004b63314c in CompositeTECWedge::findClosestDet(Point3DBase<float, GlobalTag> const&, int) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerTkDetLayers.so
#5  0x000040004b628e30 in CompositeTECWedge::groupedCompatibleDetsV(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector<DetGroup, std::allocator<DetGroup> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerTkDetLayers.so
#6  0x000040004b629154 in CompatibleDetToGroupAdder::add(GeometricSearchDet const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector<DetGroup, std::allocator<DetGroup> >&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerTkDetLayers.so
#7  0x000040004b6298e0 in CompositeTECPetal::searchNeighbors(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, SubLayerCrossing const&, float, std::vector<DetGroup, std::allocator<DetGroup> >&, bool) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerTkDetLayers.so
#8  0x000040004b629c3c in CompositeTECPetal::groupedCompatibleDetsV(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector<DetGroup, std::allocator<DetGroup> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerTkDetLayers.so
#9  0x000040004b629154 in CompatibleDetToGroupAdder::add(GeometricSearchDet const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector<DetGroup, std::allocator<DetGroup> >&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerTkDetLayers.so
#10 0x000040004b630684 in TECLayer::groupedCompatibleDetsV(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, std::vector<DetGroup, std::allocator<DetGroup> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerTkDetLayers.so
#11 0x0000400042737508 in GeometricSearchDet::groupedCompatibleDets(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libTrackingToolsDetLayers.so
#12 0x000040004b597754 in LayerMeasurements::groupedMeasurements(DetLayer const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libTrackingToolsMeasurementDet.so
#13 0x00004000a2bcceec in GroupedCkfTrajectoryBuilder::advanceOneLayer(TrajectorySeed const&, TempTrajectory&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const [clone .constprop.0] () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#14 0x00004000a2bc3108 in GroupedCkfTrajectoryBuilder::groupedLimitedCandidates(TrajectorySeed const&, TempTrajectory const&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#15 0x00004000a2bc49e4 in GroupedCkfTrajectoryBuilder::buildTrajectories(TrajectorySeed const&, std::vector<Trajectory, std::allocator<Trajectory> >&, unsigned int&, TrajectoryFilter const*) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#16 0x0000400096a2b3e4 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&)::{lambda(unsigned long)#1}::operator()(unsigned long) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerCkfPattern.so
#17 0x0000400096a2c47c in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerCkfPattern.so
#18 0x00004000206da44c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#19 0x00004000206befd0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#20 0x000040002064c074 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#21 0x000040002064c894 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#22 0x0000400020be50dc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#23 0x0000400022132dfc in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x4000cd939000, waiter=<synthetic pointer>..., this=0x4000251b3a00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#24 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (waiter=<synthetic pointer>..., t=0x0, this=0x4000251b3a00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#25 tbb::detail::r1::arena::process (tls=..., this=0x4000251b2f00) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:137
#26 tbb::detail::r1::market::process (this=0x4000251db080, j=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/market.cpp:599
#27 0x000040002213af2c in tbb::detail::r1::rml::private_worker::run (this=0x400027bbc080) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#28 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x400027bbc080) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#29 0x00004000226c78b8 in start_thread () from /lib64/libpthread.so.0
#30 0x0000400022723afc in thread_start () from /lib64/libc.so.6
Thread 2 (Thread 0x400042529250 (LWP 3912646) "cmsRun"):
#0  0x00004000226d2578 in waitpid () from /lib64/libpthread.so.0
#1  0x00004000287c93f0 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00004000287cc550 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000040002245eb8c in std::execute_native_thread_routine (__p=0x40003eea4110) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x00004000226c78b8 in start_thread () from /lib64/libpthread.so.0
#5  0x0000400022723afc in thread_start () from /lib64/libc.so.6
Thread 1 (Thread 0x40002230ca20 (LWP 3909718) "cmsRun"):
#0  0x00004000227a4b3c in nanosleep () from /lib64/libc.so.6
#1  0x00004000227a49e0 in sleep () from /lib64/libc.so.6
#2  0x00004000287c927c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00004000608bd5f0 in PixelCPEGenericBase::createClusterParam(SiPixelCluster const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoLocalTrackerSiPixelRecHits.so
#5  0x000040006089a3c0 in PixelCPEBase::getParameters(SiPixelCluster const&, GeomDet const&, LocalTrajectoryParameters const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoLocalTrackerSiPixelRecHits.so
#6  0x000040004b49d584 in TkPixelMeasurementDet::compHits(TrajectoryStateOnSurface const&, MeasurementTrackerEvent const&, float, float) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginRecoTrackerMeasurementDetPlugins.so
#7  0x000040004b49dba4 in TkPixelMeasurementDet::measurements(TrajectoryStateOnSurface const&, MeasurementEstimator const&, MeasurementTrackerEvent const&, tracking::TempMeasurements&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginRecoTrackerMeasurementDetPlugins.so
#8  0x000040004b597b68 in LayerMeasurements::groupedMeasurements(DetLayer const&, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libTrackingToolsMeasurementDet.so
#9  0x00004000a2bcceec in GroupedCkfTrajectoryBuilder::advanceOneLayer(TrajectorySeed const&, TempTrajectory&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const [clone .constprop.0] () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#10 0x00004000a2bc3108 in GroupedCkfTrajectoryBuilder::groupedLimitedCandidates(TrajectorySeed const&, TempTrajectory const&, TrajectoryFilter const*, Propagator const*, bool, std::vector<TempTrajectory, std::allocator<TempTrajectory> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#11 0x00004000a2bc59dc in GroupedCkfTrajectoryBuilder::rebuildTrajectories(TempTrajectory const&, TrajectorySeed const&, std::vector<Trajectory, std::allocator<Trajectory> >&) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/pluginRecoTrackerCkfPatternPlugins.so
#12 0x0000400096a2b778 in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&)::{lambda(unsigned long)#1}::operator()(unsigned long) const () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerCkfPattern.so
#13 0x0000400096a2c47c in cms::CkfTrackCandidateMakerBase::produceBase(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libRecoTrackerCkfPattern.so
#14 0x00004000206da44c in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#15 0x00004000206befd0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#16 0x000040002064c074 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#17 0x000040002064c894 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#18 0x0000400020be50dc in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreConcurrency.so
#19 0x0000400022138774 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x4000cd8d7700, this=0x4000251b3900) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#20 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x4000251b3900) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#21 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#22 0x00004000205c6fc4 in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#23 0x00004000205d05e8 in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#24 0x00004000205d0b78 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/week1/el8_aarch64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-03-2300/lib/el8_aarch64_gcc12/libFWCoreFramework.so
#25 0x0000000000406fe4 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#26 0x00004000221310d8 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins_a/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_aarch64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_aarch64_gcc12/external/tbb/v2021.9.0-16196f107c456d12eb9d248792d8cb7b/tbb-v2021.9.0/src/tbb/arena.cpp:688
#27 0x0000000000408854 in main::{lambda()#1}::operator()() const ()
#28 0x0000000000404954 in main ()

Current Modules:

Module: CkfTrackCandidateMaker:lowPtQuadStepTrackCandidates (crashed)
Module: CkfTrackCandidateMaker:lowPtQuadStepTrackCandidates
Module: CkfTrackCandidateMaker:lowPtQuadStepTrackCandidates
Module: TrackTfClassifier:initialStep
Module: TrackProducer:lowPtQuadStepTracks
Module: CkfTrackCandidateMaker:lowPtQuadStepTrackCandidates
Module: CkfTrackCandidateMaker:lowPtQuadStepTrackCandidates
Module: CkfTrackCandidateMaker:lowPtQuadStepTrackCandidates

A fatal system signal has occurred: segmentation violation
/data/cmsbuild/jenkins_c/workspace/ib-run-qa/CMSSW_14_1_X_2024-03-03-2300/src/DQM/Integration/test/runtest.sh: line 28: 3909718 Segmentation fault      (core dumped) cmsRun $CLIENTS_DIR/$1 unitTest=True

Mar 04 '24 10:03 iarspider

cmssw cmssw copied to clipboard

[AARCH64] SIGSEGV in CkfTrackCandidateMaker:lowPtTripletStepTrackCandidates

cmssw
cmssw copied to clipboard