[DO NOT MERGE] Fix crashes in HGCAL HLT/RECO with v19 geometry
PR description:
This PR should go on top of #48287 fixing the HLT and RECO part for V19 geometry. Also in this case the PR is not ready but it would benefit from the CI/CD tests to spot additional errors in moving to the new geometry.
####: For testing this PR:
- Include #48287
- Include #48301
- Test on V19 workflow : 34034.0
FYI: @felicepantaleo @pfs @jbsauvan @EmyrClement @indra-ehep @bsunanda
cms-bot internal usage
-code-checks
Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48303/45166
- There are other open Pull requests which might conflict with changes you have proposed:
- File HLTrigger/Configuration/python/HLT_75e33/psets/HGCAL_chargeCollectionEfficiencies_cfi.py modified in PR(s): #48287
- File HLTrigger/Configuration/python/HLT_75e33/psets/HGCAL_noise_fC_cfi.py modified in PR(s): #48287
- File L1Trigger/L1THGCal/python/l1tHGCalVFEProducer_cfi.py modified in PR(s): #48287
- File L1Trigger/L1THGCal/src/veryfrontend/HGCalVFESummationImpl.cc modified in PR(s): #48287
- File SimCalorimetry/HGCalSimProducers/interface/HGCDigitizerBase.h modified in PR(s): #48287
- File SimCalorimetry/HGCalSimProducers/interface/HGCFEElectronics.h modified in PR(s): #47447
- File SimCalorimetry/HGCalSimProducers/plugins/HGCDigitizer.cc modified in PR(s): #47447
- File SimCalorimetry/HGCalSimProducers/python/hgcalDigitizer_cfi.py modified in PR(s): #48287
- File SimCalorimetry/HGCalSimProducers/src/HGCFEElectronics.cc modified in PR(s): #47447
Code check has found code style and quality issues which could be resolved by applying following patch(s)
- code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48303/45166/code-format.patch
e.g.
curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48303/45166/code-format.patch | patch -p1You can also runscram build code-formatto apply code format directly
allow @waredjeb test rights
+code-checks
Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48303/45167
- There are other open Pull requests which might conflict with changes you have proposed:
- File HLTrigger/Configuration/python/HLT_75e33/psets/HGCAL_chargeCollectionEfficiencies_cfi.py modified in PR(s): #48287
- File HLTrigger/Configuration/python/HLT_75e33/psets/HGCAL_noise_fC_cfi.py modified in PR(s): #48287
- File L1Trigger/L1THGCal/python/l1tHGCalVFEProducer_cfi.py modified in PR(s): #48287
- File L1Trigger/L1THGCal/src/veryfrontend/HGCalVFESummationImpl.cc modified in PR(s): #48287
- File SimCalorimetry/HGCalSimProducers/interface/HGCDigitizerBase.h modified in PR(s): #48287
- File SimCalorimetry/HGCalSimProducers/interface/HGCFEElectronics.h modified in PR(s): #47447
- File SimCalorimetry/HGCalSimProducers/plugins/HGCDigitizer.cc modified in PR(s): #47447
- File SimCalorimetry/HGCalSimProducers/python/hgcalDigitizer_cfi.py modified in PR(s): #48287
A new Pull Request was created by @waredjeb for master.
It involves the following packages:
- HLTrigger/Configuration (hlt)
- L1Trigger/L1THGCal (l1, upgrade)
- RecoLocalCalo/HGCalRecProducers (upgrade, reconstruction)
- SimCalorimetry/HGCalSimProducers (simulation, upgrade)
@BenjaminRS, @Martin-Grunewald, @Moanwar, @civanch, @cmsbuild, @jfernan2, @kpedro88, @mandrenguyen, @mdhildreth, @mmusich, @quinnanm, @srimanob, @subirsarkar can you please review it and eventually sign? Thanks. @Martin-Grunewald, @SohamBhattacharya, @VourMa, @amarini, @apsallid, @bsunanda, @cseez, @denizsun, @edjtscott, @felicepantaleo, @hatakeyamak, @jbsauvan, @lecriste, @lgray, @missirol, @mmusich, @pfs, @rovere, @salimcerci, @sameasy, @sethzenz, @vandreev11, @youyingli this is something you requested to watch as well. @antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here
test parameters:
- pull_request = https://github.com/cms-sw/cmssw/pull/48287, https://github.com/cms-sw/cmssw/pull/48301
- relvals_opt = --what upgrade
- workflows = 34034.0
@cmsbuild, please test
-1
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c826bf/46689/summary.html
COMMIT: f450b19fb066b4293fea0a7135da794141b6d78f
CMSSW: CMSSW_15_1_X_2025-06-12-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/48303/46689/install.sh to create a dev area with all the needed externals and cmssw changes.
This pull request cannot be automatically merged, could you please rebase it? You can see the log for git cms-merge-topic here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c826bf/46689/git-merge-result
@cmsbuild, please test
-1
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c826bf/46695/summary.html
COMMIT: f450b19fb066b4293fea0a7135da794141b6d78f
CMSSW: CMSSW_15_1_X_2025-06-12-1500/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/48303/46695/install.sh to create a dev area with all the needed externals and cmssw changes.
This pull request cannot be automatically merged, could you please rebase it? You can see the log for git cms-merge-topic here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c826bf/46695/git-merge-result
+code-checks
Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48303/45173
- There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/ForwardDetId/interface/HGCalTriggerDetId.h modified in PR(s): #48287
- File DataFormats/ForwardDetId/interface/HGCalTriggerModuleDetId.h modified in PR(s): #48287
- File HLTrigger/Configuration/python/HLT_75e33/psets/HGCAL_chargeCollectionEfficiencies_cfi.py modified in PR(s): #48287
- File HLTrigger/Configuration/python/HLT_75e33/psets/HGCAL_noise_fC_cfi.py modified in PR(s): #48287
- File L1Trigger/L1THGCal/interface/HGCalCoarseTriggerCellMapping.h modified in PR(s): #48287
- File L1Trigger/L1THGCal/interface/HGCalTriggerTools.h modified in PR(s): #48287
- File L1Trigger/L1THGCal/interface/concentrator/HGCalConcentratorCoarsenerImpl.h modified in PR(s): #48287
- File L1Trigger/L1THGCal/interface/concentrator/HGCalConcentratorProcessorSelection.h modified in PR(s): #48287
- File L1Trigger/L1THGCal/interface/concentrator/HGCalConcentratorSuperTriggerCellImpl.h modified in PR(s): #48287
- File L1Trigger/L1THGCal/plugins/concentrator/HGCalConcentratorProcessorSelection.cc modified in PR(s): #48287
- File L1Trigger/L1THGCal/plugins/veryfrontend/HGCalVFEProcessorSums.cc modified in PR(s): #48287
- File L1Trigger/L1THGCal/python/l1tHGCalConcentratorProducer_cfi.py modified in PR(s): #48287
- File L1Trigger/L1THGCal/python/l1tHGCalVFEProducer_cfi.py modified in PR(s): #48287
- File L1Trigger/L1THGCal/src/HGCalTriggerTools.cc modified in PR(s): #48287
- File L1Trigger/L1THGCal/src/concentrator/HGCalConcentratorCoarsenerImpl.cc modified in PR(s): #48287
- File L1Trigger/L1THGCal/src/concentrator/HGCalConcentratorSuperTriggerCellImpl.cc modified in PR(s): #48287
- File L1Trigger/L1THGCal/src/veryfrontend/HGCalVFESummationImpl.cc modified in PR(s): #48287
- File RecoLocalCalo/HGCalRecAlgos/src/RecHitTools.cc modified in PR(s): #47859, #48287
- File RecoLocalCalo/HGCalRecProducers/python/HGCalUncalibRecHit_cfi.py modified in PR(s): #48287
- File SimCalorimetry/HGCalSimProducers/interface/HGCDigitizerBase.h modified in PR(s): #48287
- File SimCalorimetry/HGCalSimProducers/interface/HGCFEElectronics.h modified in PR(s): #47447
- File SimCalorimetry/HGCalSimProducers/plugins/HGCDigitizer.cc modified in PR(s): #47447
- File SimCalorimetry/HGCalSimProducers/plugins/HGCDigitizerBase.cc modified in PR(s): #48287
- File SimCalorimetry/HGCalSimProducers/python/hgcalDigitizer_cfi.py modified in PR(s): #48287
Pull request #48303 was updated.
test parameters:
- pull_request = #48301
- relvals_opt = --what upgrade
- workflows = 34034.0
@cmsbuild, please test
-1
Failed Tests: RelVals
Size: This PR adds an extra 36KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c826bf/46696/summary.html
COMMIT: 45ce846c0f0cfe5f96fa4fb3e690563f4a51e0b5
CMSSW: CMSSW_15_1_X_2025-06-12-1500/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/48303/46696/install.sh to create a dev area with all the needed externals and cmssw changes.
RelVals
----- Begin Fatal Exception 12-Jun-2025 23:08:08 CEST-----------------------
An exception of category 'BadConfiguration' occurred while
[0] Constructing the EventProcessor
[1] Constructing module: class=MixingModule label='mix'
Exception Message:
HGCFEElectronics wrong size for ToA thresholds
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 12-Jun-2025 23:09:55 CEST-----------------------
An exception of category 'BadConfiguration' occurred while
[0] Constructing the EventProcessor
[1] Constructing module: class=MixingModule label='mix'
Exception Message:
HGCFEElectronics wrong size for ToA thresholds
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 12-Jun-2025 23:12:12 CEST-----------------------
An exception of category 'BadConfiguration' occurred while
[0] Constructing the EventProcessor
[1] Constructing module: class=MixingModule label='mix'
Exception Message:
HGCFEElectronics wrong size for ToA thresholds
----- End Fatal Exception -------------------------------------------------
+code-checks
Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48303/45181
- There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/ForwardDetId/interface/HGCalTriggerDetId.h modified in PR(s): #48287
- File DataFormats/ForwardDetId/interface/HGCalTriggerModuleDetId.h modified in PR(s): #48287
- File HLTrigger/Configuration/python/HLT_75e33/psets/HGCAL_chargeCollectionEfficiencies_cfi.py modified in PR(s): #48287
- File HLTrigger/Configuration/python/HLT_75e33/psets/HGCAL_noise_fC_cfi.py modified in PR(s): #48287
- File L1Trigger/L1THGCal/interface/HGCalCoarseTriggerCellMapping.h modified in PR(s): #48287
- File L1Trigger/L1THGCal/interface/HGCalTriggerTools.h modified in PR(s): #48287
- File L1Trigger/L1THGCal/interface/concentrator/HGCalConcentratorCoarsenerImpl.h modified in PR(s): #48287
- File L1Trigger/L1THGCal/interface/concentrator/HGCalConcentratorProcessorSelection.h modified in PR(s): #48287
- File L1Trigger/L1THGCal/interface/concentrator/HGCalConcentratorSuperTriggerCellImpl.h modified in PR(s): #48287
- File L1Trigger/L1THGCal/plugins/concentrator/HGCalConcentratorProcessorSelection.cc modified in PR(s): #48287
- File L1Trigger/L1THGCal/plugins/veryfrontend/HGCalVFEProcessorSums.cc modified in PR(s): #48287
- File L1Trigger/L1THGCal/python/l1tHGCalConcentratorProducer_cfi.py modified in PR(s): #48287
- File L1Trigger/L1THGCal/python/l1tHGCalVFEProducer_cfi.py modified in PR(s): #48287
- File L1Trigger/L1THGCal/src/HGCalTriggerTools.cc modified in PR(s): #48287
- File L1Trigger/L1THGCal/src/concentrator/HGCalConcentratorCoarsenerImpl.cc modified in PR(s): #48287
- File L1Trigger/L1THGCal/src/concentrator/HGCalConcentratorSuperTriggerCellImpl.cc modified in PR(s): #48287
- File L1Trigger/L1THGCal/src/veryfrontend/HGCalVFESummationImpl.cc modified in PR(s): #48287
- File RecoLocalCalo/HGCalRecAlgos/src/RecHitTools.cc modified in PR(s): #47859, #48287
- File RecoLocalCalo/HGCalRecProducers/python/HGCalUncalibRecHit_cfi.py modified in PR(s): #48287
- File SimCalorimetry/HGCalSimProducers/interface/HGCDigitizerBase.h modified in PR(s): #48287
- File SimCalorimetry/HGCalSimProducers/plugins/HGCDigitizerBase.cc modified in PR(s): #48287
- File SimCalorimetry/HGCalSimProducers/python/hgcalDigitizer_cfi.py modified in PR(s): #48287
Pull request #48303 was updated.
@cmsbuild, please test
-1
Failed Tests: RelVals
Size: This PR adds an extra 48KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c826bf/46703/summary.html
COMMIT: 58912f6af30a9ea41f1ce8c5ae56b60e0449c152
CMSSW: CMSSW_15_1_X_2025-06-12-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/48303/46703/install.sh to create a dev area with all the needed externals and cmssw changes.
RelVals
+code-checks
Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48303/45188
- There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/ForwardDetId/interface/HGCalTriggerDetId.h modified in PR(s): #48287
- File DataFormats/ForwardDetId/interface/HGCalTriggerModuleDetId.h modified in PR(s): #48287
- File HLTrigger/Configuration/python/HLT_75e33/psets/HGCAL_chargeCollectionEfficiencies_cfi.py modified in PR(s): #48287
- File HLTrigger/Configuration/python/HLT_75e33/psets/HGCAL_noise_fC_cfi.py modified in PR(s): #48287
- File L1Trigger/L1THGCal/interface/HGCalCoarseTriggerCellMapping.h modified in PR(s): #48287
- File L1Trigger/L1THGCal/interface/HGCalTriggerTools.h modified in PR(s): #48287
- File L1Trigger/L1THGCal/interface/concentrator/HGCalConcentratorCoarsenerImpl.h modified in PR(s): #48287
- File L1Trigger/L1THGCal/interface/concentrator/HGCalConcentratorProcessorSelection.h modified in PR(s): #48287
- File L1Trigger/L1THGCal/interface/concentrator/HGCalConcentratorSuperTriggerCellImpl.h modified in PR(s): #48287
- File L1Trigger/L1THGCal/plugins/concentrator/HGCalConcentratorProcessorSelection.cc modified in PR(s): #48287
- File L1Trigger/L1THGCal/plugins/veryfrontend/HGCalVFEProcessorSums.cc modified in PR(s): #48287
- File L1Trigger/L1THGCal/python/l1tHGCalConcentratorProducer_cfi.py modified in PR(s): #48287
- File L1Trigger/L1THGCal/python/l1tHGCalVFEProducer_cfi.py modified in PR(s): #48287
- File L1Trigger/L1THGCal/src/HGCalTriggerTools.cc modified in PR(s): #48287
- File L1Trigger/L1THGCal/src/concentrator/HGCalConcentratorCoarsenerImpl.cc modified in PR(s): #48287
- File L1Trigger/L1THGCal/src/concentrator/HGCalConcentratorSuperTriggerCellImpl.cc modified in PR(s): #48287
- File L1Trigger/L1THGCal/src/veryfrontend/HGCalVFESummationImpl.cc modified in PR(s): #48287
- File RecoLocalCalo/HGCalRecAlgos/src/RecHitTools.cc modified in PR(s): #47859, #48287
- File RecoLocalCalo/HGCalRecProducers/python/HGCalUncalibRecHit_cfi.py modified in PR(s): #48287
- File SimCalorimetry/HGCalSimProducers/interface/HGCDigitizerBase.h modified in PR(s): #48287
- File SimCalorimetry/HGCalSimProducers/plugins/HGCDigitizerBase.cc modified in PR(s): #48287
- File SimCalorimetry/HGCalSimProducers/python/hgcalDigitizer_cfi.py modified in PR(s): #48287
@cmsbuild, please test
Pull request #48303 was updated.
-1
Failed Tests: RelVals
Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c826bf/46710/summary.html
COMMIT: 5d8dacfa45ff5555d6d63b3546db99e29ac10b8e
CMSSW: CMSSW_15_1_X_2025-06-13-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/48303/46710/install.sh to create a dev area with all the needed externals and cmssw changes.
RelVals
Some remarks: 29634.75 and 29634.911 crash in the DIGI step with D110, while 34034.0 crashes in the RECO step with D120.
One thing that puzzles me re-running locally 29634.911. Using directly the cmdLog I can reproduce the crash. However step2 runs fine if I either:
- use cmsRun (cmsDriver used to create cfg only)
- run cmsDriver removing the mem profiling options, namely
--maxmem_profile --customise Validation/Performance/TimeMemoryJobReport.customiseWithTimeMemoryJobReport --maxmem_profile(btw I noticemaxmem_profileis called twice, not sure if intentional).
The crash in 34034.0 might be fixed by my last commit (it works for me locally). I am looking now at the other 2.
@cmsbuild, please test
+code-checks
Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48303/45202
- There are other open Pull requests which might conflict with changes you have proposed:
- File RecoLocalCalo/HGCalRecAlgos/src/RecHitTools.cc modified in PR(s): #47859
Pull request #48303 was updated.
-1
Failed Tests: RelVals
Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c826bf/46729/summary.html
COMMIT: fd88a4ed11be4b7029f98e591354df9fdf378f0b
CMSSW: CMSSW_15_1_X_2025-06-15-0000/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/48303/46729/install.sh to create a dev area with all the needed externals and cmssw changes.
RelVals
- 29696.0
A fatal system signal has occurred: segmentation violation
Some remarks: 29634.75 and 29634.911 crash in the DIGI step with D110, while 34034.0 crashes in the RECO step with D120.
One thing that puzzles me re-running locally 29634.911. Using directly the cmdLog I can reproduce the crash. However step2 runs fine if I either:
* use cmsRun (cmsDriver used to create cfg only) * run cmsDriver removing the mem profiling options, namely ` --maxmem_profile --customise Validation/Performance/TimeMemoryJobReport.customiseWithTimeMemoryJobReport --maxmem_profile` (btw I notice `maxmem_profile` is called twice, not sure if intentional).
I see the same behaviour on 29696.0
Running on the step1.root produced in the tests, everything works fine without the memory profiling. With the memory profiling I get the crash here
#2 0x00007f4d0ce33164 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02893/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_X_2025-06-10-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3 <signal handler called>
#4 0x00007f4c20a28fd9 in HGCFEElectronics<HGCDataFrame<DetId, HGCSample> >::runShaperWithToT(HGCDataFrame<DetId, HGCSample>&, std::array<float, 15ul>&, std::array<float, 15ul>&, CLHEP::HepRandomEngine*, unsigned int, float, unsigned i
nt, float, int, float, float, std::array<float, 6ul> const&) () from /data/wredjeb/FixV19/testPR/CMSSW_15_1_X_2025-06-11-1100/lib/el8_amd64_gcc12/libSimCalorimetryHGCalSimProducers.so
#5 0x00007f4c20a562d7 in HGCDigitizerBase::runSimple(std::unique_ptr<edm::SortedCollection<HGCDataFrame<DetId, HGCSample>, edm::StrictWeakOrdering<HGCDataFrame<DetId, HGCSample> > >, std::default_delete<edm::SortedCollection<HGCDataFr
ame<DetId, HGCSample>, edm::StrictWeakOrdering<HGCDataFrame<DetId, HGCSample> > > > >&, std::unordered_map<unsigned int, hgc_digi::HGCCellInfo, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int
const, hgc_digi::HGCCellInfo> > >&, CaloSubdetectorGeometry const*, std::unordered_set<DetId, std::hash<DetId>, std::equal_to<DetId>, std::allocator<DetId> > const&, CLHEP::HepRandomEngine*) () from /data/wredjeb/FixV19/testPR/CMSSW_1
5_1_X_2025-06-11-1100/lib/el8_amd64_gcc12/pluginSimCalorimetryHGCalSimProducersPlugins.so
#6 0x00007f4c20a481fb in HGCDigitizer::finalizeEvent(edm::Event&, edm::EventSetup const&, CLHEP::HepRandomEngine*) () from /data/wredjeb/FixV19/testPR/CMSSW_15_1_X_2025-06-11-1100/lib/el8_amd64_gcc12/pluginSimCalorimetryHGCalSimProduc
ersPlugins.so
#7 0x00007f4c20a498f4 in HGCDigiProducer::finalizeEvent(edm::Event&, edm::EventSetup const&) () from /data/wredjeb/FixV19/testPR/CMSSW_15_1_X_2025-06-11-1100/lib/el8_amd64_gcc12/pluginSimCalorimetryHGCalSimProducersPlugins.so
#8 0x00007f4c20cfd76b in edm::MixingModule::finalizeEvent(edm::Event&, edm::EventSetup const&) () from /data/wredjeb/FixV19/testPR/CMSSW_15_1_X_2025-06-11-1100/lib/el8_amd64_gcc12/pluginSimGeneralMixingModulePlugins.so
#9 0x00007f4c20c62341 in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02893/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_X_2025-06-10-2300/lib/el8_amd64_gcc12/libMixingBase.so
#10 0x00007f4d13467455 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02893/el8_amd64_gcc12/cms/cm
ssw/CMSSW_15_1_X_2025-06-10-2300/lib/el8_amd64_gcc12/libFWCoreFramework.so