cmssw Update heavy-ion eras for 2024 + add MC RelVal workflows

Sets up heavy-ion eras for 2024:

Implements HI reco, i.e.,pp_on_PbPb type eras and their corresponding MC relval workflows (161,161.2,161.02, etc)
Updates the UPC eras to be based on Run3_2024, rather than just Run3, although I think for the moment they're equivalent

Note that certain sim customizations for the ZDC and HF fine-grained bit are for now just copied from the 2023 era. They are extended to the UPC era, although this era is so far only used for the reco step, not for the digi one. In case the UPC era is used for digi at some point, it's better to change this now.

The heavy-ion embedding (event overlay) workflows are not extended to 2024 yet. This will be done in a subsequent PR.

No backport will be needed.

Jun 26 '24 02:06 mandrenguyen

cms-bot internal usage

Jun 26 '24 02:06 cmsbuild

test parameters:

workflows = 161, 161.02, 161.03, 161.1, 180, 180.1, 181, 181.1, 142.901, 142.902

Jun 26 '24 02:06 mandrenguyen

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45307/40717

This PR adds an extra 92KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/relval_standard.py modified in PR(s): #45055
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): #45055
- File Configuration/StandardSequences/python/Eras.py modified in PR(s): #33532, #45240

Jun 26 '24 02:06 cmsbuild

A new Pull Request was created by @mandrenguyen for master.

It involves the following packages:

Configuration/Eras (operations)
Configuration/PyReleaseValidation (pdmv, upgrade)
Configuration/StandardSequences (operations)
SimCalorimetry/HcalTrigPrimProducers (l1)
SimG4Core/Application (simulation)

@srimanob, @subirsarkar, @aloeliger, @civanch, @rappoccio, @fabiocos, @davidlange6, @kskovpen, @mdhildreth, @miquork, @cmsbuild, @AdrianoDee, @sunilUIET, @antoniovilela, @epalencia can you please review it and eventually sign? Thanks. @missirol, @VinInn, @rovere, @mmusich, @GiacomoSguazzoni, @AnnikaStein, @VourMa, @dgulhan, @slomeo, @bsunanda, @fabiocos, @Martin-Grunewald, @sameasy, @abdoulline, @JanFSchulte, @mtosi, @makortel, @felicepantaleo this is something you requested to watch as well. @antoniovilela, @sextonkennedy, @rappoccio you are the release manager for this.

cms-bot commands are listed here

Jun 26 '24 02:06 cmsbuild

please test

Jun 26 '24 02:06 mandrenguyen

-1

Failed Tests: UnitTests RelVals RelVals-INPUT AddOn Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a0d7d8/40078/summary.html COMMIT: a0ea6a00188b93904edb8261f4c2c498a07b7038 CMSSW: CMSSW_14_1_X_2024-06-25-2300/el8_amd64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/45307/40078/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 4 errors in the following unit tests:

---> test test_MC_22_crosscheck had ERRORS
---> test test_MC_23_crosscheck had ERRORS
---> test test_MC_22_setup had ERRORS
and more ...

RelVals

139.001

----- Begin Fatal Exception 26-Jun-2024 04:57:39 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
   [1] While attempting to load plugin L1TRawToDigi
Exception Message:
unable to load /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02843/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-06-25-2300/lib/el8_amd64_gcc12/poisoned/plugin-poisoned-EventFilterL1TRawToDigiAuto.so because /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02843/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-06-25-2300/lib/el8_amd64_gcc12/poisoned/plugin-poisoned-EventFilterL1TRawToDigiAuto.so: cannot open shared object file: No such file or directory
----- End Fatal Exception -------------------------------------------------

136.731

----- Begin Fatal Exception 26-Jun-2024 04:57:39 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
   [1] While attempting to load plugin L1TRawToDigi
Exception Message:
unable to load /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02843/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-06-25-2300/lib/el8_amd64_gcc12/poisoned/plugin-poisoned-EventFilterL1TRawToDigiAuto.so because /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02843/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-06-25-2300/lib/el8_amd64_gcc12/poisoned/plugin-poisoned-EventFilterL1TRawToDigiAuto.so: cannot open shared object file: No such file or directory
----- End Fatal Exception -------------------------------------------------

140.023

----- Begin Fatal Exception 26-Jun-2024 04:57:39 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
   [1] While attempting to load plugin L1TRawToDigi
Exception Message:
unable to load /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02843/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-06-25-2300/lib/el8_amd64_gcc12/poisoned/plugin-poisoned-EventFilterL1TRawToDigiAuto.so because /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02843/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-06-25-2300/lib/el8_amd64_gcc12/poisoned/plugin-poisoned-EventFilterL1TRawToDigiAuto.so: cannot open shared object file: No such file or directory
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

RelVals-INPUT

134.801134.801_RunHLTPhy2015C/step2_RunHLTPhy2015C.log
134.802134.802_RunDoubleEG2015C/step2_RunDoubleEG2015C.log
134.803134.803_RunDoubleMuon2015C/step2_RunDoubleMuon2015C.log

Expand to see more relval errors ...

AddOn Tests

fastsim1

----- Begin Fatal Exception 26-Jun-2024 04:55:28 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
   [1] While attempting to load plugin L1TDigiToRaw
Exception Message:
unable to load /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02843/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-06-25-2300/lib/el8_amd64_gcc12/poisoned/plugin-poisoned-EventFilterL1TRawToDigiAuto.so because /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02843/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-06-25-2300/lib/el8_amd64_gcc12/poisoned/plugin-poisoned-EventFilterL1TRawToDigiAuto.so: cannot open shared object file: No such file or directory
----- End Fatal Exception -------------------------------------------------

fastsim2

----- Begin Fatal Exception 26-Jun-2024 04:55:29 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
   [1] While attempting to load plugin L1TDigiToRaw
Exception Message:
unable to load /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02843/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-06-25-2300/lib/el8_amd64_gcc12/poisoned/plugin-poisoned-EventFilterL1TRawToDigiAuto.so because /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02843/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-06-25-2300/lib/el8_amd64_gcc12/poisoned/plugin-poisoned-EventFilterL1TRawToDigiAuto.so: cannot open shared object file: No such file or directory
----- End Fatal Exception -------------------------------------------------

hlt_mc_Fake1

----- Begin Fatal Exception 26-Jun-2024 04:55:03 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
   [1] While attempting to load plugin L1TDigiToRaw
Exception Message:
unable to load /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02843/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-06-25-2300/lib/el8_amd64_gcc12/poisoned/plugin-poisoned-EventFilterL1TRawToDigiAuto.so because /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02843/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-06-25-2300/lib/el8_amd64_gcc12/poisoned/plugin-poisoned-EventFilterL1TRawToDigiAuto.so: cannot open shared object file: No such file or directory
----- End Fatal Exception -------------------------------------------------

Expand to see more addon errors ...

Jun 26 '24 03:06 cmsbuild

please test These tests seem to run file locally. I'm not sure what this poisoned plugin error mentioned in the logs is about. For lack of a better idea, I'm just gonna try to the tests again and see if this was transient. If anyone has some insight please enlighten me.

Jun 26 '24 14:06 mandrenguyen

For lack of a better idea, I'm just gonna try to the tests again and see if this was transient.

this issue was caused by a severely broken IB CMSSW_14_1_X_2024-06-25-2300 due to this PR. See discussion there for a rationale. It was fixed in CMSSW_14_1_X_2024-06-26-1200

Jun 26 '24 14:06 mmusich

please abort

Jun 26 '24 16:06 mandrenguyen

please test Thanks @stahlleiton for spotting a typo, which is fixed now

Jun 26 '24 16:06 mandrenguyen

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45307/40732

This PR adds an extra 60KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/relval_standard.py modified in PR(s): #45055
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): #45055
- File Configuration/StandardSequences/python/Eras.py modified in PR(s): #33532, #45240

Jun 26 '24 16:06 cmsbuild

Pull request #45307 was updated. @AdrianoDee, @aloeliger, @antoniovilela, @civanch, @davidlange6, @epalencia, @fabiocos, @kskovpen, @mdhildreth, @miquork, @rappoccio, @srimanob, @subirsarkar, @sunilUIET can you please check and sign again.

Jun 26 '24 16:06 cmsbuild

-1

Failed Tests: RelVals Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a0d7d8/40100/summary.html COMMIT: fdd45d786cd85c33d330ac1ae0ff103929f3e3ac CMSSW: CMSSW_14_1_X_2024-06-26-1200/el8_amd64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/45307/40100/install.sh to create a dev area with all the needed externals and cmssw changes.

DAS Queries: The DAS query tests failed, see the summary page for details.

RelVals

161.02

----- Begin Fatal Exception 26-Jun-2024 20:14:25 CEST-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Calling InputSource::readFile_
   [1] Opening DQM Root file
   [2] Calling StorageFactory::open()
   [3] Calling File::sysopen()
Exception Message:
Failed to open the file 'step3_inDQM.root'
Input file file:step3_inDQM.root was not found, could not be opened, or is corrupted.
   Additional Info:
      [a] open() failed with system error 'No such file or directory' (error code 2)
----- End Fatal Exception -------------------------------------------------

Jun 26 '24 18:06 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45307/40735

This PR adds an extra 68KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/relval_standard.py modified in PR(s): #45055
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): #45055
- File Configuration/StandardSequences/python/Eras.py modified in PR(s): #33532, #45240

Jun 26 '24 19:06 cmsbuild

Pull request #45307 was updated. @AdrianoDee, @aloeliger, @antoniovilela, @civanch, @cmsbuild, @davidlange6, @epalencia, @fabiocos, @kskovpen, @mdhildreth, @miquork, @rappoccio, @srimanob, @subirsarkar, @sunilUIET can you please check and sign again.

Jun 26 '24 19:06 cmsbuild

please test

Jun 26 '24 19:06 mandrenguyen

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a0d7d8/40105/summary.html COMMIT: de912384fc3815c9787db863145194084faf4928 CMSSW: CMSSW_14_1_X_2024-06-26-1200/el8_amd64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/45307/40105/install.sh to create a dev area with all the needed externals and cmssw changes.

DAS Queries: The DAS query tests failed, see the summary page for details.

Comparison Summary

Summary:

You potentially added 5 lines to the logs
Reco comparison results: 12 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3345088
DQMHistoTests: Total failures: 9
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3345059
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: no differences found

Jun 26 '24 23:06 cmsbuild

@cms-sw/pdmv-l2 can you have look please? This one mainly concerns relval. It would be good to get it in for the next pre-release

Jun 27 '24 18:06 mandrenguyen

@cms-sw/pdmv-l2 @cms-sw/l1-l2 @cms-sw/simulation-l2 @cms-sw/upgrade-l2 Kindly have a look.

Jul 02 '24 14:07 mandrenguyen

+l1

No real changes to L1, just introducing additional era modifiers.

Jul 02 '24 15:07 aloeliger

+1

Jul 02 '24 15:07 civanch

ping @cms-sw/pdmv-l2 @cms-sw/upgrade-l2

Jul 08 '24 19:07 mandrenguyen

+Upgrade

Jul 08 '24 20:07 srimanob

@cmsbuild please test

Retrigger the test after 2 weeks, before merging.

Jul 08 '24 21:07 srimanob

+1

Size: This PR adds an extra 12KB to repository Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a0d7d8/40287/summary.html COMMIT: de912384fc3815c9787db863145194084faf4928 CMSSW: CMSSW_14_1_X_2024-07-08-1100/el8_amd64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/45307/40287/install.sh to create a dev area with all the needed externals and cmssw changes.

DAS Queries: The DAS query tests failed, see the summary page for details.

Comparison Summary

Summary:

You potentially added 2 lines to the logs
Reco comparison results: 11 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3345088
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3345062
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: no differences found

Jul 09 '24 00:07 cmsbuild

+pdmv

Jul 09 '24 07:07 AdrianoDee

+1

Jul 16 '24 15:07 rappoccio

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.

Jul 16 '24 15:07 cmsbuild