cmssw TICL-barrel: run CLUE in the barrel calorimeters and first workflows

PR description:

This PR adds the possibility of running part of the TICL reconstruction framework in the barrel region. In particular, it introduces the ticl_barrel process modifier with which one can run CLUE in ECAL and HCAL (HO not included), also at HLT, and the corresponding associators and validation chain.

With this PR also the first workflows for TICL-barrel are introduced. These workflows run the modules mentioned above at step2 and step3, and with the HARVESTING one can produce the DQM plots for the LayerClusters in the barrel calorimeters.

PR validation:

Tested on wfs 29694.203, 29634.0 (these two with and without ticl_barrel process modifier) and 29753.209 (introduced by this PR).

@felicepantaleo @rovere @amecca

Apr 13 '25 14:04 brusale

cms-bot internal usage

Apr 13 '25 14:04 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47859/44499

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): #47090, #47113, #47124
- File HLTrigger/Configuration/python/HLT_75e33_cff.py modified in PR(s): #47611
- File HLTrigger/Configuration/python/HLT_75e33_timing_cff.py modified in PR(s): #47611

Apr 13 '25 14:04 cmsbuild

A new Pull Request was created by @brusale for master.

It involves the following packages:

Configuration/Generator (generators)
Configuration/ProcessModifiers (operations)
Configuration/PyReleaseValidation (upgrade, pdmv)
DataFormats/CaloRecHit (reconstruction)
DataFormats/HGCalReco (upgrade, reconstruction)
HLTrigger/Configuration (hlt)
RecoHGCal/TICL (upgrade, reconstruction)
RecoLocalCalo/HGCalRecAlgos (upgrade, reconstruction)
RecoLocalCalo/HGCalRecProducers (upgrade, reconstruction)
RecoParticleFlow/PFClusterProducer (reconstruction)
SimCalorimetry/HGCalAssociatorProducers (simulation, upgrade)
SimGeneral/CaloAnalysis (simulation)
Validation/Configuration (simulation, dqm)
Validation/HGCalValidation (dqm)

@AdrianoDee, @Martin-Grunewald, @Moanwar, @antoniovagnerini, @antoniovilela, @bbilin, @civanch, @cmsbuild, @davidlange6, @dickychant, @fabiocos, @jfernan2, @kpedro88, @lviliani, @mandrenguyen, @mdhildreth, @menglu21, @miquork, @mkirsano, @mmusich, @rappoccio, @rseidita, @srimanob, @subirsarkar can you please review it and eventually sign? Thanks. @Martin-Grunewald, @ReyerBand, @SohamBhattacharya, @VourMa, @apsallid, @argiro, @bsunanda, @cseez, @denizsun, @edjtscott, @fabiocos, @felicepantaleo, @forthommel, @hatakeyamak, @lecriste, @lgray, @makortel, @martinamalberti, @missirol, @mmarionncern, @mmusich, @pfs, @rchatter, @rovere, @salimcerci, @sameasy, @seemasharmafnal, @sethzenz, @slomeo, @sobhatta, @thomreis, @vandreev11, @wang0jin, @youyingli this is something you requested to watch as well. @antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

Apr 13 '25 14:04 cmsbuild

test parameters:

relvals_opt = --what upgrade
workflows = 29694.203, 29634.0, 29753.209, 29634.756
enable = hlt_p2_integration, hlt_p2_timing

Apr 13 '25 14:04 mmusich

@cmsbuild, please test

Apr 13 '25 14:04 mmusich

-1

Failed Tests: ClangBuild Size: This PR adds an extra 512KB to repository Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-724d13/45531/summary.html COMMIT: 33de3138324ea3654b255c926f1d18aeee9896b0 CMSSW: CMSSW_15_1_X_2025-04-13-0000/el8_amd64_gcc12 Additional Tests: HLT_P2_INTEGRATION,HLT_P2_TIMING User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/47859/45531/install.sh to create a dev area with all the needed externals and cmssw changes.

Clang Build

I found compilation warning while trying to compile with clang. Command used:

USER_CUDA_FLAGS='--expt-relaxed-constexpr' USER_CXXFLAGS='-Wno-register -fsyntax-only' /usr/bin/time -v scram build -k -j 32 COMPILER='llvm compile'

See details on the summary page.

Apr 13 '25 15:04 cmsbuild

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47859/44537

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): #47090, #47113, #47124, #47872, #47891
- File HLTrigger/Configuration/python/HLT_75e33_cff.py modified in PR(s): #47611
- File HLTrigger/Configuration/python/HLT_75e33_timing_cff.py modified in PR(s): #47611
- File SimGeneral/CaloAnalysis/plugins/CaloTruthAccumulator.cc modified in PR(s): #47883

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47859/44537/code-format.patch e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47859/44537/code-format.patch | patch -p1 You can also run scram build code-format to apply code format directly

Apr 16 '25 19:04 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47859/44542

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): #47090, #47113, #47124, #47891
- File HLTrigger/Configuration/python/HLT_75e33_cff.py modified in PR(s): #47611
- File HLTrigger/Configuration/python/HLT_75e33_timing_cff.py modified in PR(s): #47611
- File RecoLocalCalo/HGCalRecProducers/plugins/RecHitMapProducer.cc modified in PR(s): #47892
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToCPAssociatorByEnergyScoreProducer.cc modified in PR(s): #47892
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToSCAssociatorByEnergyScoreProducer.cc modified in PR(s): #47892
- File SimGeneral/CaloAnalysis/plugins/CaloTruthAccumulator.cc modified in PR(s): #47883

Apr 17 '25 06:04 cmsbuild

Pull request #47859 was updated. @AdrianoDee, @Martin-Grunewald, @Moanwar, @antoniovagnerini, @antoniovilela, @bbilin, @civanch, @cmsbuild, @davidlange6, @dickychant, @fabiocos, @jfernan2, @kpedro88, @lviliani, @mandrenguyen, @mdhildreth, @menglu21, @miquork, @mkirsano, @mmusich, @rappoccio, @rseidita, @srimanob, @subirsarkar can you please check and sign again.

Apr 17 '25 06:04 cmsbuild

@cmsbuild, please test

Apr 17 '25 06:04 mmusich

+1

Size: This PR adds an extra 248KB to repository Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-724d13/45600/summary.html COMMIT: 021dcd86734a18c0a9cdcf3b18b7935132d98e9d CMSSW: CMSSW_15_1_X_2025-04-16-2300/el8_amd64_gcc12 Additional Tests: HLT_P2_INTEGRATION,HLT_P2_TIMING User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/47859/45600/install.sh to create a dev area with all the needed externals and cmssw changes.

DAS Queries: The DAS query tests failed, see the summary page for details.
HLT P2 Timing: chart

Comparison Summary

Summary:

You potentially removed 196 lines from the logs
Reco comparison results: 30 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3916361
DQMHistoTests: Total failures: 166
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3916175
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 215 log files, 184 edm output root files, 50 DQM output files
TriggerResults: found differences in 6 / 48 workflows

Apr 17 '25 09:04 cmsbuild

Some observations:

the HLT integration test failed log, but the bot didn't pick that up (hopefully it should be fixed https://github.com/cms-sw/cms-bot/pull/2479). For the record, the problem is of the type:

    ----- Begin Fatal Exception 17-Apr-2025 12:05:09 CEST-----------------------
An exception of category 'ScheduleExecutionFailure' occurred while
   [0] Calling beginJob
Exception Message:
Unrunnable schedule
The Path/EndPath configuration could cause the job to deadlock
  module 'hltParticleFlowClusterHBHE' is on path 'HLT_IsoMu24_FromL1TkMuon' and depends on module 'hltParticleFlowRecHitHBHE'
  module 'hltParticleFlowRecHitHBHE' is on path 'HLT_IsoMu24_FromL1TkMuon' and follows module 'hltParticleFlowClusterHBHE' on the path
----- End Fatal Exception -------------------------------------------------

I am not sure I understand the changes picked up the Validation e.g. in HGCAL/HGCalValidator (see link).

Apr 17 '25 10:04 mmusich

test parameters:

pull_request = https://github.com/cms-sw/cms-bot/pull/2479
enable = hlt_p2_integration

Apr 17 '25 14:04 mmusich

@cmsbuild, please test

Apr 17 '25 14:04 mmusich

-1

Failed Tests: HLTP2Integration Size: This PR adds an extra 16KB to repository Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-724d13/45617/summary.html COMMIT: 021dcd86734a18c0a9cdcf3b18b7935132d98e9d CMSSW: CMSSW_15_1_X_2025-04-16-2300/el8_amd64_gcc12 Additional Tests: HLT_P2_INTEGRATION User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/47859/45617/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 199 lines from the logs
Reco comparison results: 30 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3916361
DQMHistoTests: Total failures: 166
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3916175
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 215 log files, 184 edm output root files, 50 DQM output files
TriggerResults: found differences in 6 / 48 workflows

Apr 17 '25 17:04 cmsbuild

test parameters:

relvals_opt = --what upgrade
workflows = 29694.203, 29634.0, 29753.209, 29634.756
enable = hlt_p2_integration, hlt_p2_timing

Apr 22 '25 06:04 mmusich

Some observations: [...] * I am not sure I understand the changes picked up the Validation e.g. in HGCAL/HGCalValidator (see link).

Taking a closer look, the plot (e.g. the second one here) are actually fixed by this PR. The Number of Layers plot shows how many layers a CaloParticle is spanning and it is physically impossible for a particle in HGCAL to span more than 47 layers. I updated the PR using the endcap_hits_and_fractions to make sure the layer calculation does not produce unexpected results.

Apr 24 '25 21:04 brusale

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47859/44611

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): #47090, #47891
- File HLTrigger/Configuration/python/HLT_75e33_cff.py modified in PR(s): #47611
- File HLTrigger/Configuration/python/HLT_75e33_timing_cff.py modified in PR(s): #47611
- File RecoLocalCalo/HGCalRecProducers/plugins/RecHitMapProducer.cc modified in PR(s): #47892
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToCPAssociatorByEnergyScoreProducer.cc modified in PR(s): #47892
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToSCAssociatorByEnergyScoreProducer.cc modified in PR(s): #47892

Apr 24 '25 21:04 cmsbuild

Pull request #47859 was updated. @AdrianoDee, @Martin-Grunewald, @Moanwar, @antoniovagnerini, @antoniovilela, @bbilin, @civanch, @cmsbuild, @davidlange6, @dickychant, @fabiocos, @jfernan2, @kpedro88, @lviliani, @mandrenguyen, @mdhildreth, @menglu21, @miquork, @mkirsano, @mmusich, @rappoccio, @rseidita, @srimanob, @subirsarkar can you please check and sign again.

Apr 24 '25 21:04 cmsbuild

@cmsbuild, please test

Apr 24 '25 21:04 mmusich

+1

Size: This PR adds an extra 552KB to repository Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-724d13/45712/summary.html COMMIT: e9af0dac22af54e21f6f57e5c6c62a781a6609b2 CMSSW: CMSSW_15_1_X_2025-04-24-1100/el8_amd64_gcc12 Additional Tests: HLT_P2_INTEGRATION,HLT_P2_TIMING User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/47859/45712/install.sh to create a dev area with all the needed externals and cmssw changes.

DAS Queries: The DAS query tests failed, see the summary page for details.
HLT P2 Timing: chart

Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 32 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3920066
DQMHistoTests: Total failures: 458
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3919588
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
Checked 218 log files, 186 edm output root files, 51 DQM output files
TriggerResults: found differences in 8 / 49 workflows

Apr 24 '25 23:04 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47859/44617

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): #47090, #47891
- File HLTrigger/Configuration/python/HLT_75e33_cff.py modified in PR(s): #47611
- File HLTrigger/Configuration/python/HLT_75e33_timing_cff.py modified in PR(s): #47611
- File RecoLocalCalo/HGCalRecProducers/plugins/RecHitMapProducer.cc modified in PR(s): #47892
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToCPAssociatorByEnergyScoreProducer.cc modified in PR(s): #47892
- File SimCalorimetry/HGCalAssociatorProducers/plugins/LCToSCAssociatorByEnergyScoreProducer.cc modified in PR(s): #47892

Apr 25 '25 13:04 cmsbuild

test parameters:

relvals_opt = --what upgrade
workflows = 29694.203, 29634.0, 29753.209, 29634.758
enable = hlt_p2_integration, hlt_p2_timing

Apr 25 '25 13:04 mmusich

Pull request #47859 was updated. @AdrianoDee, @Martin-Grunewald, @Moanwar, @antoniovagnerini, @antoniovilela, @bbilin, @civanch, @cmsbuild, @davidlange6, @dickychant, @fabiocos, @jfernan2, @kpedro88, @lviliani, @mandrenguyen, @mdhildreth, @menglu21, @miquork, @mkirsano, @mmusich, @rappoccio, @rseidita, @srimanob, @subirsarkar can you please check and sign again.

Apr 25 '25 13:04 cmsbuild

@cmsbuild, please test

Apr 25 '25 13:04 mmusich

allow @brusale test rights

Apr 25 '25 13:04 mmusich

+1

Size: This PR adds an extra 36KB to repository Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-724d13/45717/summary.html COMMIT: 71325f18889499722342a8507ae7d34191107cda CMSSW: CMSSW_15_1_X_2025-04-25-1100/el8_amd64_gcc12 Additional Tests: HLT_P2_INTEGRATION,HLT_P2_TIMING User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/47859/45717/install.sh to create a dev area with all the needed externals and cmssw changes.

DAS Queries: The DAS query tests failed, see the summary page for details.
HLT P2 Timing: chart

Comparison Summary

Summary:

You potentially added 4 lines to the logs
Reco comparison results: 28 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3913297
DQMHistoTests: Total failures: 450
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3912827
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 215 log files, 184 edm output root files, 50 DQM output files
TriggerResults: found differences in 7 / 48 workflows

Apr 25 '25 16:04 cmsbuild

code-checks

May 05 '25 18:05 mmusich

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47859/44701

ERROR: Unable to merge PR.

See log https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47859/44701/cms-checkout-topic.log

May 05 '25 18:05 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47859/44775

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): #47090
- File HLTrigger/Configuration/python/HLT_75e33_cff.py modified in PR(s): #47611
- File HLTrigger/Configuration/python/HLT_75e33_timing_cff.py modified in PR(s): #47611

May 11 '25 18:05 cmsbuild