cmssw Further fixes for HGCal validation at HLT

PR description:

This PR is a further follow-up to https://github.com/cms-sw/cmssw/pull/47892 and https://github.com/cms-sw/cmssw/pull/48114. It solves the problem spotted at https://github.com/cms-sw/cmssw/pull/47708#issuecomment-2884694870 in which wf 29634.75 is failing in the current IB on ARM platforms. I also reproduced the same issue by running runTheMatrix.py --what upgrade -l 29634.751 -t 4 -j 8 in a regular IB with arch el9_amd64_gcc12.

PR validation:

Tested successfully via runTheMatrix.py --what upgrade -l 29634.751 --nEvents 500 -t 4 -j 8 both under el9_amd64_gcc12 and el8_aarch64_gcc12.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Not a backport, not to be backported.

May 31 '25 08:05 mmusich

please test for el8_aarch64_gcc12

May 31 '25 08:05 mmusich

cms-bot internal usage

May 31 '25 08:05 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48221/45017

May 31 '25 08:05 cmsbuild

-1

Failed Tests: UnitTests Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1f4996/46470/summary.html COMMIT: 973c1c07b6b853397b90170018bc79ca21ef7b1e CMSSW: CMSSW_15_1_X_2025-05-30-2300/el8_aarch64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/48221/46470/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 1 errors in the following unit tests:

---> test TestDQMGUIUpload had ERRORS

May 31 '25 09:05 cmsbuild

test parameters:

relvals_opt = --what upgrade
workflows = 29634.0,29634.75,29634.203,29634.752,29634.77

May 31 '25 16:05 mmusich

@cmsbuild, please test

Jun 02 '25 08:06 mmusich

+1

Size: This PR adds an extra 16KB to repository Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1f4996/46492/summary.html COMMIT: 973c1c07b6b853397b90170018bc79ca21ef7b1e CMSSW: CMSSW_15_1_X_2025-06-01-2300/el8_amd64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/48221/46492/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 5 lines from the logs
Reco comparison results: 3 differences found in the comparisons
DQMHistoTests: Total files compared: 53
DQMHistoTests: Total histograms compared: 4221612
DQMHistoTests: Total failures: 390
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4221202
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
Checked 226 log files, 192 edm output root files, 53 DQM output files
TriggerResults: no differences found

Jun 02 '25 11:06 cmsbuild

please test for el8_aarch64_gcc12

Aug 06 '25 06:08 mmusich

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1f4996/47547/summary.html COMMIT: 973c1c07b6b853397b90170018bc79ca21ef7b1e CMSSW: CMSSW_15_1_X_2025-08-05-2300/el8_aarch64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/48221/47547/install.sh to create a dev area with all the needed externals and cmssw changes.

Aug 06 '25 08:08 cmsbuild

please test

now to get fresh comparisons

Aug 06 '25 09:08 mmusich

+1

Size: This PR adds an extra 16KB to repository Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1f4996/47551/summary.html COMMIT: 973c1c07b6b853397b90170018bc79ca21ef7b1e CMSSW: CMSSW_15_1_X_2025-08-05-2300/el8_amd64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/48221/47551/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 6 lines from the logs
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 53
DQMHistoTests: Total histograms compared: 4263640
DQMHistoTests: Total failures: 299
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4263321
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
Checked 226 log files, 192 edm output root files, 53 DQM output files
TriggerResults: no differences found

Aug 06 '25 11:08 cmsbuild

test parameters:

workflows = ph2_hlt

Sep 05 '25 11:09 mmusich

please test

Sep 05 '25 11:09 mmusich

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48221/45989

There are other open Pull requests which might conflict with changes you have proposed:
- File RecoHGCal/TICL/plugins/SimTrackstersProducer.cc modified in PR(s): #48319
- File Validation/HGCalValidation/plugins/HGCalValidator.cc modified in PR(s): #48565

Sep 05 '25 11:09 cmsbuild

Pull request #48221 was updated.

Sep 05 '25 11:09 cmsbuild

+1

Size: This PR adds an extra 20KB to repository Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1f4996/47982/summary.html COMMIT: 918a3eb848e01f4ba9b48a4115421a4a58c8fed8 CMSSW: CMSSW_15_1_X_2025-09-05-1100/el8_amd64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/48221/47982/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 21 lines from the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 63
DQMHistoTests: Total histograms compared: 4464071
DQMHistoTests: Total failures: 1457
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4462594
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 62 files compared)
Checked 260 log files, 219 edm output root files, 63 DQM output files
TriggerResults: no differences found

Sep 05 '25 13:09 cmsbuild

type ngt, bug-fix

Sep 05 '25 13:09 mmusich

A new Pull Request was created by @mmusich for master.

It involves the following packages:

RecoHGCal/TICL (reconstruction, upgrade)
Validation/HGCalValidation (dqm)

@Moanwar, @antoniovagnerini, @ctarricone, @jfernan2, @mandrenguyen, @rseidita, @srimanob, @subirsarkar can you please review it and eventually sign? Thanks. @apsallid, @bsunanda, @cseez, @denizsun, @felicepantaleo, @forthommel, @hatakeyamak, @lecriste, @lgray, @missirol, @pfs, @rovere, @salimcerci, @sameasy, @sethzenz, @sobhatta, @vandreev11, @youyingli this is something you requested to watch as well. @ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

Sep 05 '25 13:09 cmsbuild

@cms-sw/dqm-l2 @cms-sw/reconstruction-l2 @cms-sw/upgrade-l2 do you have any comments?

Sep 09 '25 07:09 mmusich

+1

Sep 09 '25 08:09 jfernan2

Milestone for this pull request has been moved to CMSSW_16_0_X. Please open a backport if it should also go in to CMSSW_15_1_X.

Sep 10 '25 06:09 cmsbuild

Milestone for this pull request has been moved to CMSSW_16_0_X. Please open a backport if it should also go in to CMSSW_15_1_X.

I am confused by this message. Despite it gitHub still shows CMSSW_15_1_X as milestone.

Can please @cms-sw/orp-l2 clarify?

Sep 10 '25 07:09 mmusich

@mmusich I'll look into it (it happens every time we start a new release series)

Sep 10 '25 07:09 iarspider

+Upgrade

Sep 10 '25 12:09 Moanwar

@cms-sw/dqm-l2 can you please have a look?

Sep 16 '25 09:09 mmusich

+dqm

Sep 16 '25 12:09 rseidita

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @mandrenguyen, @sextonkennedy, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)

Sep 16 '25 12:09 cmsbuild

+1

Sep 17 '25 12:09 ftenchini