cmssw Update PU for Run3 wfs

PR description:

Update MinBias datasets for 2021, 2023, 2024, and 2021FS Run3 PU wfs, following the discussion in #38760.

PR validation:

Tested on 11834.0,12634.0,13034.0,13434.0,12234.0

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Not a backport.

Aug 11 '22 19:08 kskovpen

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39041/31527

This PR adds an extra 48KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): #38760

Aug 11 '22 20:08 cmsbuild

A new Pull Request was created by @kskovpen for master.

It involves the following packages:

Configuration/PyReleaseValidation (pdmv, upgrade)

@jordan-martins, @bbilin, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen can you please review it and eventually sign? Thanks. @makortel, @kpedro88, @fabiocos, @Martin-Grunewald, @missirol, @trtomei, @beaucero, @slomeo this is something you requested to watch as well. @perrotta, @dpiparo, @qliphy, @rappoccio you are the release manager for this.

cms-bot commands are listed here

Aug 11 '22 20:08 cmsbuild

test parameters:

workflow = 11834.0,12634.0,13034.0

Aug 11 '22 20:08 kskovpen

please test

Aug 11 '22 20:08 kskovpen

-1

Failed Tests: RelVals-INPUT Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ca1e69/26774/summary.html COMMIT: 860d91e3b5b2b117dff5dd08589afa0873663c4c CMSSW: CMSSW_12_5_X_2022-08-11-1100/el8_amd64_gcc10 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39041/26774/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@dilsonjd cms-sw/cmssw#38974
@jaimeleonh cms-sw/cmssw#39020
@mmusich cms-sw/cmssw#39025

You can see more details here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ca1e69/26774/git-recent-commits.json https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ca1e69/26774/git-merge-result

DAS Queries: The DAS query tests failed, see the summary page for details.

RelVals-INPUT

11601.0DAS Error
11602.0DAS Error
11603.0DAS Error

Expand to see more relval errors ...

11604.0
11605.0
11606.0
11630.0
11725.0
11925.0

Comparison Summary

Summary:

You potentially added 235 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 9336 differences found in the comparisons
DQMHistoTests: Total files compared: 53
DQMHistoTests: Total histograms compared: 4016058
DQMHistoTests: Total failures: 23138
DQMHistoTests: Total nulls: 4
DQMHistoTests: Total successes: 3992894
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 3.4220000000000006 KiB( 52 files compared)
DQMHistoSizes: changed ( 11834.0 ): 3.900 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 12634.0 ): 0.811 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 13034.0 ): -1.289 KiB SiStrip/MechanicalView
Checked 220 log files, 59 edm output root files, 53 DQM output files
TriggerResults: found differences in 4 / 52 workflows

Aug 12 '22 01:08 cmsbuild

@kskovpen How do you plan for FastSim? Thanks.

Aug 12 '22 02:08 srimanob

@kskovpen How do you plan for FastSim? Thanks.

It's coming.

Aug 12 '22 06:08 kskovpen

@kskovpen How do you plan for FastSim? Thanks.

It's coming.

Thanks. To confirm the cmsDriver with CMSSW_12_5_0_pre4, it should come from runTheMatrix.py --what upgrade -l 13240.303 --command="--beamspot Realistic25ns13p6TeVEarly2022Collision" --wm init

Aug 12 '22 06:08 srimanob

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39041/31543

This PR adds an extra 52KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): #38760

Aug 13 '22 09:08 cmsbuild

Pull request #39041 was updated. @jordan-martins, @bbilin, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen can you please check and sign again.

Aug 13 '22 09:08 cmsbuild

test parameters:

workflow = 11834.0,12634.0,13034.0,13434.0

Aug 13 '22 09:08 kskovpen

please test

Aug 13 '22 09:08 kskovpen

-1

Failed Tests: RelVals RelVals-INPUT Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ca1e69/26796/summary.html COMMIT: e1428dac1d913967070c1bfe7a75f96b40555eb3 CMSSW: CMSSW_12_5_X_2022-08-12-2300/el8_amd64_gcc10 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39041/26796/install.sh to create a dev area with all the needed externals and cmssw changes.

DAS Queries: The DAS query tests failed, see the summary page for details.

RelVals

13434.0

----- Begin Fatal Exception 13-Aug-2022 12:51:12 CEST-----------------------
An exception of category 'NoSecondaryFiles' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=MixingModule label='mix'
Exception Message:
RootEmbeddedFileSequence no input files specified for secondary input source.
----- End Fatal Exception -------------------------------------------------

11834.0

----- Begin Fatal Exception 13-Aug-2022 12:55:02 CEST-----------------------
An exception of category 'NoSecondaryFiles' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=MixingModule label='mix'
Exception Message:
RootEmbeddedFileSequence no input files specified for secondary input source.
----- End Fatal Exception -------------------------------------------------

13434.0

----- Begin Fatal Exception 13-Aug-2022 12:51:12 CEST-----------------------
An exception of category 'NoSecondaryFiles' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=MixingModule label='mix'
Exception Message:
RootEmbeddedFileSequence no input files specified for secondary input source.
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

RelVals-INPUT

11601.0DAS Error
11602.0DAS Error
11603.0DAS Error

Expand to see more relval errors ...

11604.0
11605.0
11606.0
11607.0
11608.0
11609.0
11630.0
11634.0
11634.1
11634.24
11634.5
11634.501
11634.505
11634.511
11634.521
11634.601
11634.7
11634.71
11634.72
11634.91
11640.0
11643.0
11646.0
11650.0
11650.501
11650.505
11723.17
11725.0
11834.0
11834.13
11834.19
11834.21
11834.24
11834.99
11846.0
11846.19
11925.0
12034.0
12434.0
12634.0
12634.99
12834.0
13034.0
13034.99

Aug 13 '22 12:08 cmsbuild

Apart from -v1 issue I've mentioned separately, FastSim PU dataset seems to be wrong.

cmsDriver of FastSim PU looks very wrong, and that cause the issue in the PR test. It tried to look for /RelValMinBias_14TeV/CMSSW_12_5_0_pre4-124X_mcRun3_2022_realistic_v10_Run3FastSim_FastSim-v11/GEN-SIM-RECO but what produced from https://dmytro.web.cern.ch/dmytro/cmsprodmon/workflows.php?campaign=CMSSW_12_5_0_pre4__fastsim_noPU_2022_14TeV-1660286681 is /RelValMinBias_14TeV/CMSSW_12_5_0_pre4-124X_mcRun3_2022_realistic_v10_Run3FastSim_FastSim-v1/GEN-SIM

From the workflow, I see the following

MinBias_14TeV_pythia8_TuneCP5_cfi --beamspot Realistic25ns13p6TeVEarly2022Collision --conditions auto:phase1_2022_realistic --datatier GEN-SIM --era Run3_FastSim --eventcontent FASTPU --fast --fileout file:step1.root --geometry DB:Extended --nStreams 2 --nThreads 8 --no_exec --number 10 --python_filename step_1_cfg.py --relval 90000,3000 --step GEN,SIM

However, workflow 13240.303 should give (as I try locally)

MinBias_14TeV_pythia8_TuneCP5_cfi -s GEN,SIM,RECOBEFMIX --fast --era Run3_FastSim --beamspot Realistic25ns13p6TeVEarly2022Collision --conditions auto:phase1_2022_realistic --geometry DB:Extended --eventcontent FASTPU --datatier GEN-SIM-RECO --relval 90000,100 --io MinBias_14TeV_pythia8_TuneCP5_2021FS_FastSimRun3_Run3FSMBMixing.io --python MinBias_14TeV_pythia8_TuneCP5_2021FS_FastSimRun3_Run3FSMBMixing.py --beamspot Realistic25ns13p6TeVEarly2022Collision -n 10 --no_exec --fileout file:step1.root

step and datatier were wrong in the relval workflow that produce MinBias for FastSim.

Aug 13 '22 13:08 srimanob

Apart from -v1 issue I've mentioned separately, FastSim PU dataset seems to be wrong.

cmsDriver of FastSim PU looks very wrong, and that cause the issue in the PR test. It tried to look for /RelValMinBias_14TeV/CMSSW_12_5_0_pre4-124X_mcRun3_2022_realistic_v10_Run3FastSim_FastSim-v11/GEN-SIM-RECO but what produced from https://dmytro.web.cern.ch/dmytro/cmsprodmon/workflows.php?campaign=CMSSW_12_5_0_pre4__fastsim_noPU_2022_14TeV-1660286681 is /RelValMinBias_14TeV/CMSSW_12_5_0_pre4-124X_mcRun3_2022_realistic_v10_Run3FastSim_FastSim-v1/GEN-SIM

From the workflow, I see the following
MinBias_14TeV_pythia8_TuneCP5_cfi --beamspot Realistic25ns13p6TeVEarly2022Collision --conditions auto:phase1_2022_realistic --datatier GEN-SIM --era Run3_FastSim --eventcontent FASTPU --fast --fileout file:step1.root --geometry DB:Extended --nStreams 2 --nThreads 8 --no_exec --number 10 --python_filename step_1_cfg.py --relval 90000,3000 --step GEN,SIM
However, workflow 13240.303 should give (as I try locally)
MinBias_14TeV_pythia8_TuneCP5_cfi -s GEN,SIM,RECOBEFMIX --fast --era Run3_FastSim --beamspot Realistic25ns13p6TeVEarly2022Collision --conditions auto:phase1_2022_realistic --geometry DB:Extended --eventcontent FASTPU --datatier GEN-SIM-RECO --relval 90000,100 --io MinBias_14TeV_pythia8_TuneCP5_2021FS_FastSimRun3_Run3FSMBMixing.io --python MinBias_14TeV_pythia8_TuneCP5_2021FS_FastSimRun3_Run3FSMBMixing.py --beamspot Realistic25ns13p6TeVEarly2022Collision -n 10 --no_exec --fileout file:step1.root
step and datatier were wrong in the relval workflow that produce MinBias for FastSim.

Thanks, indeed, this is fixed now.

Aug 13 '22 13:08 kskovpen

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39041/31550

This PR adds an extra 52KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): #38760

Aug 14 '22 07:08 cmsbuild

Pull request #39041 was updated. @jordan-martins, @bbilin, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen can you please check and sign again.

Aug 14 '22 07:08 cmsbuild

please test

Aug 14 '22 07:08 kskovpen

Hi @kskovpen Do you tried to run locally? Or at least, test with --wm init and look at the cmsDriver(s).

If I do

cmsrel CMSSW_12_5_X_2022-08-14-0000
cd CMSSW_12_5_X_2022-08-14-0000/src/
cmsenv
git cms-merge-topic kskovpen:run3pu
scram b -j 8
runTheMatrix.py --what upgrade -l 13434.0 --wm init

I will get following cmsDriver, which the PU dataset is wrong. cmsDriver.py step2 -s SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,PAT,NANO,VALIDATION:@standardValidation+@miniAODValidation,DQM:@standardDQMFS+@miniAODDQM+@nanoAODDQM --fast --era Run3_FastSim --beamspot Run3RoundOptics25ns13TeVLowSigmaZ --conditions auto:phase1_2022_realistic --geometry DB:Extended --eventcontent FEVTDEBUGHLT,MINIAODSIM,NANOEDMAODSIM,DQM --datatier GEN-SIM-DIGI-RECO,MINIAODSIM,NANOAODSIM,DQMIO -n 10 --pileup Run3_Flat55To75_PoissonOOTPU --pileup_input das:/RelValMinBias_14TeV/CMSSW_12_5_0_pre4-124X_mcRun3_2022_realistic_v10_FastSim-v11/GEN-SIM-RECO --io FastSimRun3PU_2021FSPU.io --python FastSimRun3PU_2021FSPU.py --no_exec --filein file:step1.root --fileout file:step2.root > step2_TTbar_14TeV+2021FSPU+TTbar_14TeV_TuneCP5_Gen+FastSimRun3PU+HARVESTFastRun3PU.log 2>&1

or for FullSim (11834.0), it will call das:/RelValMinBias_14TeV/CMSSW_12_5_0_pre4-124X_mcRun3_2022_realistic_v10-v21/GEN-SIM

If you want to overwrite the version, you should deal with the following lines, https://github.com/cms-sw/cmssw/blob/master/Configuration/PyReleaseValidation/python/relval_steps.py#L3727-L3737, and leave -v without number in the definition of defaultDataSets.

Aug 14 '22 07:08 srimanob

abort

Aug 14 '22 08:08 kskovpen

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39041/31553

This PR adds an extra 52KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): #38760

Aug 14 '22 11:08 cmsbuild

Pull request #39041 was updated. @jordan-martins, @bbilin, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen can you please check and sign again.

Aug 14 '22 11:08 cmsbuild

test parameters:

workflow = 11834.0,12634.0,13034.0,13434.0,12234.0

Aug 14 '22 11:08 kskovpen

please test

Aug 14 '22 11:08 kskovpen

-1

Failed Tests: RelVals-INPUT Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ca1e69/26805/summary.html COMMIT: de3d13961aaca376d618cce97078cb7ff2a53f84 CMSSW: CMSSW_12_5_X_2022-08-14-0000/el8_amd64_gcc10 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39041/26805/install.sh to create a dev area with all the needed externals and cmssw changes.

DAS Queries: The DAS query tests failed, see the summary page for details.

RelVals-INPUT

11601.0DAS Error
11602.0DAS Error
11603.0DAS Error

Expand to see more relval errors ...

11604.0
11605.0
11606.0
11630.0
11725.0
11925.0
12034.0

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 9305 differences found in the comparisons
DQMHistoTests: Total files compared: 53
DQMHistoTests: Total histograms compared: 4016058
DQMHistoTests: Total failures: 52829
DQMHistoTests: Total nulls: 7
DQMHistoTests: Total successes: 3963200
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.8420000000000001 KiB( 52 files compared)
DQMHistoSizes: changed ( 11834.0 ): 0.708 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 12634.0 ): 0.059 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 13034.0 ): -1.605 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 220 log files, 59 edm output root files, 53 DQM output files
TriggerResults: found differences in 4 / 52 workflows

Aug 14 '22 18:08 cmsbuild

Failure from INPUT-DAS of 11601.0, 11602.0, 11603.0 because the change on defaultDataSets['2021']='CMSSW_12_0_0_pre4-120X_mcRun3_2021_realistic_v2-v' We need also missing relvals to avoid this.

Note that, as mentioned above, now we will start mixing BS in the INPUT relvals as all of them use the same string, CMSSW_12_5_0_pre4-124X_mcRun3_2022_realistic_v10-v. For example

/RelValTTbar_14TeV/CMSSW_12_5_0_pre4-124X_mcRun3_2022_realistic_v10-v1/GEN-SIM ==> Old BS
/RelValSingleElectronPt10/CMSSW_12_5_0_pre4-124X_mcRun3_2022_realistic_v10-v1/GEN-SIM ==> New or old BS as you will submit
/RelValMinBias_14TeV/CMSSW_12_5_0_pre4-124X_mcRun3_2022_realistic_v10-v2/GEN-SIM ==> New BS. -v2 should not be used a separator of BS (in my opinion).

Not that easy, and avoid changing in overwrite condition by just submit full RelVals with processing string (e.g. BS2022). So that, all will be v1 and very clean and separate from existing relvals of 12_5_0_pre4.

Aug 15 '22 02:08 srimanob

Failure from INPUT-DAS of 11601.0, 11602.0, 11603.0 because the change on defaultDataSets['2021']='CMSSW_12_0_0_pre4-120X_mcRun3_2021_realistic_v2-v' We need also missing relvals to avoid this.

Note that, as mentioned above, now we will start mixing BS in the INPUT relvals as all of them use the same string, CMSSW_12_5_0_pre4-124X_mcRun3_2022_realistic_v10-v. For example

/RelValTTbar_14TeV/CMSSW_12_5_0_pre4-124X_mcRun3_2022_realistic_v10-v1/GEN-SIM ==> Old BS

/RelValSingleElectronPt10/CMSSW_12_5_0_pre4-124X_mcRun3_2022_realistic_v10-v1/GEN-SIM ==> New or old BS as you will submit

/RelValMinBias_14TeV/CMSSW_12_5_0_pre4-124X_mcRun3_2022_realistic_v10-v2/GEN-SIM ==> New BS. -v2 should not be used a separator of BS (in my opinion).

Not that easy, and avoid changing in overwrite condition by just submit full RelVals with processing string (e.g. BS2022). So that, all will be v1 and very clean and separate from existing relvals of 12_5_0_pre4.

Thanks @srimanob. Let's try, indeed. Just submitted the full batch of noPU samples that use BS2022 as processing string.

Aug 15 '22 06:08 kskovpen

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39041/31635

This PR adds an extra 52KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): #38760, #39059

Aug 18 '22 11:08 cmsbuild

Pull request #39041 was updated. @jordan-martins, @bbilin, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen can you please check and sign again.

Aug 18 '22 11:08 cmsbuild

please test

Aug 18 '22 11:08 kskovpen