Matti Kortelainen

The only place adding elements to `listLayer_` seems to be in `GEMDQMHarvester::getGeometryInfo()` https://github.com/cms-sw/cmssw/blob/9dab6a97de58a930407ae847dcd4210b1946807f/DQM/GEM/plugins/GEMDQMHarvester.cc#L333-L339 which is called for every `dqmEndLuminosityBlock()`. The `listLayer_` is cleared in the other branch of this if-else.

So the `getGeometry()` calls `listLayer_.clear()` (and then fills it again) if the argument `MonitorElement*` is not `nullptr` https://github.com/cms-sw/cmssw/blob/9dab6a97de58a930407ae847dcd4210b1946807f/DQM/GEM/plugins/GEMDQMHarvester.cc#L315-L317 I checked at the caller https://github.com/cms-sw/cmssw/blob/9dab6a97de58a930407ae847dcd4210b1946807f/DQM/GEM/plugins/GEMDQMHarvester.cc#L216 that the `h2SrcStatusEOH` is always `nullptr`...

In CMSSW_12_6_ROOT626_X_2022-09-18-2300 el8_amd64_gcc10 step 2 crashed with ``` %MSG-e Root_Error: MixingModule:mix TUnixSystem::DispatchSignals() 19-Sep-2022 09:05:14 CEST Run: 1 Event: 4 floating point exception %MSG %MSG-e Root_Error: MixingModule:mix TUnixSystem::DispatchSignals() 19-Sep-2022 09:05:15 CEST...

In CMSSW_12_6_X_2022-09-18-0000 el8_amd64_gcc10 step 2 crashed with ``` %MSG-e Root_Error: MixingModule:mix TUnixSystem::DispatchSignals() 18-Sep-2022 14:56:16 CEST Run: 1 Event: 3 floating point exception %MSG %MSG-e Root_Error: MixingModule:mix TUnixSystem::DispatchSignals() 18-Sep-2022 14:56:16 CEST...

In CMSSW_12_6_X_2022-09-17-1100 el8_amd64_gcc10 step 3 crashed with ``` Thread 5 (Thread 0x2ba7c0000700 (LWP 22037) "cmsRun"): #2 0x00002ba77b0213a0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02750/el8_amd64_gcc10/cms/cmssw/CMSSW_12_6_X_2022-09-17-1100/lib/el8_amd64_gcc10/pluginFWCoreServicesPlugins.so #3 #4 0x00002ba773e3483b in ?? () from /cvmfs/cms-ib.cern.ch/nweek-02750/el8_amd64_gcc10/cms/cmssw/CMSSW_12_6_X_2022-09-17-1100/external/el8_amd64_gcc10/lib/libz.so.1...

In CMSSW_12_6_X_2022-09-16-2300 slc7_amd64_gcc10 step 3 crashed with ``` Thread 5 (Thread 0x2b2e81c00700 (LWP 5869) "cmsRun"): #2 0x00002b2e3c5083b0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02750/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_6_X_2022-09-16-2300/lib/slc7_amd64_gcc10/pluginFWCoreServicesPlugins.so #3 #4 0x00002b2e67b462f8 in void EcalPulseCovariance::serialize(eos::portable_iarchive&, unsigned int)...

In CMSSW_12_6_X_2022-09-16-1100 slc7_amd64_gcc10 step 2 crashed with ``` %MSG-e Root_Error: MixingModule:mix TUnixSystem::DispatchSignals() 16-Sep-2022 16:47:24 CEST Run: 1 Event: 4 floating point exception %MSG %MSG-e Root_Error: MixingModule:mix TUnixSystem::DispatchSignals() 16-Sep-2022 16:47:24 CEST...

In CMSSW_12_6_UBSAN_X_2022-09-16-1100 el8_amd64_gcc11 step 2 crashed with ``` Thread 5 (Thread 0x2b8ffd600700 (LWP 18959) "cmsRun"): #2 0x00002b8f59ecc9de in (anonymous namespace)::sig_pause_for_stacktrace (sig=) at /pool/condor/dir_2153/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/6dda80287ded6cb30cbcf21c830565ab/opt/cmssw/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_UBSAN_X_2022-09-16-1100/src/FWCore/Services/plugins/InitRootHandlers.cc:448 #3 #4 0x00002b908469abbd in std::hash::operator() (__val=2560316554, this=)...

For CMSSW_12_6_X_2022-09-15-2300 el9_amd64_gcc11 step 2 the IB dashboard reports a timeout, but we did get some stack traces ``` %MSG-e Root_Error: MixingModule:mix TUnixSystem::DispatchSignals() 16-Sep-2022 06:58:44 CEST Run: 1 Event: 3...

CMSSW_12_6_X_2022-09-15-2300 seems to be the first IB where these crashes appeared. The `HcalDDDRecConstants::getHCID()` seems to play some role in most of them.