Adding PUPPI jets and METs to offline DQM

[rseidita]

PR description:

Adding the PUPPI collection for both jets and METs to the offline DQM JetMET sequence, as presented here.

PR validation:

echo '{
"369978" : [[1, 800]]
}' > step1_lumiRanges.log  2>&1

(dasgoclient --limit 0 --query 'lumi,file dataset=/JetMET0/Run2023D-v1/RAW run=369978' --format json | das-selected-lumis.py 1,800 ) | sort -u > step1_dasquery.log  2>&1

cmsDriver.py step2  --process reHLT -s L1REPACK:Full,HLT:@relval2023 --conditions auto:run3_hlt_relval --data  --eventcontent FEVTDEBUGHLT --datatier FEVTDEBUGHLT --era Run3_2023 -n 1000 --filein filelist:step1_dasquery.log --lumiToProcess step1_lumiRanges.log --fileout file:step2.root  --nThreads 8

cmsDriver.py step3  --conditions auto:run3_data_prompt_relval -s RAW2DIGI,L1Reco,RECO,PAT,NANO,DQM:@jetmet --datatier RECO,MINIAOD,NANOAOD,DQMIO --eventcontent RECO,MINIAOD,NANOEDMAOD,DQM --data  --process reRECO --scenario pp --era Run3_2023 --customise Configuration/DataProcessing/RecoTLR.customisePostEra_Run3 --hltProcess reHLT -n 1000 --filein  file:step2.root  --fileout file:step3.root  --nThreads 8

cmsDriver.py step4  -s HARVESTING:@jetmet --conditions auto:run3_data --data  --filetype DQM --scenario pp --era Run3_2023 -n 1000 --filein file:step3_inDQM.root --fileout file:step4.root

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Not a backport

[cmsbuild]

cms-bot internal usage

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44336/39369

• This PR adds an extra 64KB to repository

[cmsbuild]

A new Pull Request was created by @rseidita for master.

It involves the following packages:

• DQMOffline/JetMET (dqm)

@tjavaid, @cmsbuild, @rvenditti, @syuvivida, @nothingface0, @antoniovagnerini can you please review it and eventually sign? Thanks. @seemasharmafnal, @ahinzmann, @gkasieczka, @jdolen, @mariadalfonso, @mmarionncern, @rappoccio, @rociovilar, @schoef, @jdamgov, @nhanvtran this is something you requested to watch as well. @sextonkennedy, @antoniovilela, @rappoccio you are the release manager for this.

cms-bot commands are listed here

[tjavaid]

please test

[cmsbuild]

-1

Failed Tests: RelVals RelVals-INPUT Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-35c537/37954/summary.html COMMIT: c931ea7bd3481c546ba7d5122abc217acf7e05a8 CMSSW: CMSSW_14_1_X_2024-03-06-2300/el8_amd64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44336/37954/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

• 140.56

----- Begin Fatal Exception 07-Mar-2024 15:54:46 CET-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 326479 lumi: 7 event: 1394020 stream: 0
   [1] Running path 'dqmoffline_step'
   [2] Calling method for module ChainedJetCorrectorProducer/'dqmAk4PFPuppiL1FastL2L3ResidualCorrector'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: reco::JetCorrector
Looking for module label: ak4PFPuppiL1FastjetCorrector
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

• 1000.0

----- Begin Fatal Exception 07-Mar-2024 15:59:11 CET-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 165121 lumi: 62 event: 23609118 stream: 0
   [1] Running path 'dqmoffline_3_step'
   [2] Calling method for module ChainedJetCorrectorProducer/'dqmAk4PFPuppiL1FastL2L3ResidualCorrector'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: reco::JetCorrector
Looking for module label: ak4PFPuppiL1FastjetCorrector
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

• 1001.0

----- Begin Fatal Exception 07-Mar-2024 16:00:13 CET-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 165121 lumi: 62 event: 23609118 stream: 0
   [1] Running path 'dqmoffline_2_step'
   [2] Calling method for module ChainedJetCorrectorProducer/'dqmAk4PFPuppiL1FastL2L3ResidualCorrector'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: reco::JetCorrector
Looking for module label: ak4PFPuppiL1FastjetCorrector
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

RelVals-INPUT

• 138.4138.4_PromptCollisions2021/step2_PromptCollisions2021.log
• 138.5138.5_ExpressCollisions2021/step2_ExpressCollisions2021.log
• 140.61140.61_RunHI2022/step2_RunHI2022.log

Expand to see more relval errors ...

• 140.57
• 140.56
• 141.901
• 141.902
• 1001.0
• 1000.0
• 1003.0
• 1001.4
• 1002.3
• 1002.4
• 1001.3
• 1001.2
• 1040.0

[rseidita]

@tjavaid I tried running the following workflows locally: 1001.4 1002.3 1002.4 1001.3 141.901 1001.2 1040.0 140.56 1000.0 1001.0 140.57 140.56 1003.0

Run successfully: 1001.4 1002.3 1002.4 1001.3 141.901 1001.2 1040.0

Fail because raw files are on tape:

• 140.56 RunHI2018 → On tape
• 140.57 RunHI2018Reduced → On tape
• 140.56 RunHI2018 → On tape

Fail because DAS query at step0 fails:

• 1000.0 RunMinBias2011A → DAS query in step0 fails
• 1001.0 RunMinBias2011A → DAS query in step0 fails
• 1003.0 RunMinBias2012A → DAS query in step0 fails

Assuming I'm testing the correct things, I don't see issues in the execution of the actual code. Please let me know if I'm looking in the wrong place

[AdrianoDee]

please test

[cmsbuild]

-1

Failed Tests: RelVals RelVals-INPUT Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-35c537/38093/summary.html COMMIT: c931ea7bd3481c546ba7d5122abc217acf7e05a8 CMSSW: CMSSW_14_1_X_2024-03-12-2300/el8_amd64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44336/38093/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

• 140.56

----- Begin Fatal Exception 13-Mar-2024 10:17:31 CET-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 326479 lumi: 7 event: 1394020 stream: 0
   [1] Running path 'dqmoffline_step'
   [2] Calling method for module ChainedJetCorrectorProducer/'dqmAk4PFPuppiL1FastL2L3ResidualCorrector'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: reco::JetCorrector
Looking for module label: ak4PFPuppiL1FastjetCorrector
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

• 1000.0

----- Begin Fatal Exception 13-Mar-2024 10:21:49 CET-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 165121 lumi: 62 event: 23609118 stream: 0
   [1] Running path 'dqmoffline_3_step'
   [2] Calling method for module ChainedJetCorrectorProducer/'dqmAk4PFPuppiL1FastL2L3ResidualCorrector'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: reco::JetCorrector
Looking for module label: ak4PFPuppiL1FastjetCorrector
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

• 1001.0

----- Begin Fatal Exception 13-Mar-2024 10:22:59 CET-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 165121 lumi: 62 event: 23609118 stream: 0
   [1] Running path 'dqmoffline_2_step'
   [2] Calling method for module ChainedJetCorrectorProducer/'dqmAk4PFPuppiL1FastL2L3ResidualCorrector'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: reco::JetCorrector
Looking for module label: ak4PFPuppiL1FastjetCorrector
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

RelVals-INPUT

• 138.4138.4_PromptCollisions2021/step2_PromptCollisions2021.log
• 138.5138.5_ExpressCollisions2021/step2_ExpressCollisions2021.log
• 140.61140.61_RunHI2022/step2_RunHI2022.log

Expand to see more relval errors ...

• 140.57
• 140.56
• 141.902
• 141.901
• 1000.0
• 1001.0
• 1003.0
• 1001.4
• 1002.3
• 1002.4
• 1001.3
• 1001.2
• 1040.0

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44336/39489

• This PR adds an extra 16KB to repository

[cmsbuild]

Pull request #44336 was updated. @cmsbuild, @rvenditti, @antoniovagnerini, @nothingface0, @tjavaid, @syuvivida can you please check and sign again.

[tjavaid]

please test

[cmsbuild]

-1

Failed Tests: RelVals RelVals-INPUT Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-35c537/38163/summary.html COMMIT: 1ebe746a84aa7254058310de88b7c75d5328ca54 CMSSW: CMSSW_14_1_X_2024-03-14-2300/el8_amd64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44336/38163/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

• 140.56

----- Begin Fatal Exception 15-Mar-2024 14:31:46 CET-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 326479 lumi: 7 event: 1394020 stream: 0
   [1] Running path 'dqmoffline_step'
   [2] Calling method for module PileupJetIdProducer/'pileupJetIdCalculatorPUPPIDQM'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for a container with elements of type: reco::Jet
Looking for module label: ak4PFJetsPuppi
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

RelVals-INPUT

• 140.61140.61_RunHI2022/step2_RunHI2022.log
• 140.57140.57_RunHI2018Reduced/step2_RunHI2018Reduced.log
• 140.56140.56_RunHI2018/step2_RunHI2018.log

[AdrianoDee]

please test (to inspect the failures given the latest tests are 1 month old)

[AdrianoDee]

This is just to make public what we were discussing “in private” : )

The issue there is that at the moment there’s no difference between JME DQM for HIon or pp, but this PR is introducing a discrepancy: you want to use PUPPI jets that are not defined in HIon conditions. So you need to change your modifications in order to be discarded in case of HIon run. So for example in

https://github.com/cms-sw/cmssw/blob/1ebe746a84aa7254058310de88b7c75d5328ca54/DQMOffline/JetMET/python/jetAnalyzer_cff.py#L7-L10

You need to change to something like

from Configuration.ProcessModifiers.pp_on_AA_cff import pp_on_AA

jetDQMAnalyzerSequence = cms.Sequence(jetDQMAnalyzerAk4CaloCleaned
                                      *jetDQMAnalyzerAk4PFUncleaned*jetDQMAnalyzerAk4PFCleaned
                                      *jetDQMAnalyzerAk4PFCHSCleaned
                                   ) #common pp and HI

_jetDQMAnalyzerSequenceWithPUPPI = cms.Sequence(jetDQMAnalyzerAk4CaloCleaned
                                      *jetDQMAnalyzerAk4PFUncleaned*jetDQMAnalyzerAk4PFCleaned
                                      *jetDQMAnalyzerAk4PFCHSCleaned*jetDQMAnalizerAk4PUPPICleaned
                                   ) #only for pp with PUPPI

(~pp_on_AA).toReplaceWith(jetDQMAnalyzerSequence, _jetDQMAnalyzerSequenceWithPUPPI) # if NOT HIon use PUPPI

[rseidita]

Thanks a lot @AdrianoDee, I will proceed like this

[cmsbuild]

-1

Failed Tests: RelVals RelVals-INPUT Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-35c537/38843/summary.html COMMIT: 1ebe746a84aa7254058310de88b7c75d5328ca54 CMSSW: CMSSW_14_1_X_2024-04-14-2300/el8_amd64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44336/38843/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

• 140.56

----- Begin Fatal Exception 15-Apr-2024 11:48:51 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 326479 lumi: 7 event: 1394020 stream: 0
   [1] Running path 'dqmoffline_step'
   [2] Calling method for module PileupJetIdProducer/'pileupJetIdCalculatorPUPPIDQM'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for a container with elements of type: reco::Jet
Looking for module label: ak4PFJetsPuppi
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

RelVals-INPUT

• 140.61140.61_RunHI2022/step2_RunHI2022.log
• 140.57140.57_RunHI2018Reduced/step2_RunHI2018Reduced.log
• 140.56140.56_RunHI2018/step2_RunHI2018.log

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44336/39959

• This PR adds an extra 20KB to repository

[cmsbuild]

Pull request #44336 was updated. @cmsbuild, @nothingface0, @antoniovagnerini, @rvenditti, @tjavaid, @syuvivida can you please check and sign again.

[AdrianoDee]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-35c537/38861/summary.html COMMIT: 866d59b3b005b2de7d861a3c727c9d4471f4c8b5 CMSSW: CMSSW_14_1_X_2024-04-15-2300/el8_amd64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44336/38861/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 83 lines to the logs
• Reco comparison results: 50 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3319475
• DQMHistoTests: Total failures: 7
• DQMHistoTests: Total nulls: 195
• DQMHistoTests: Total successes: 3319253
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 3605.085 KiB( 47 files compared)
• DQMHistoSizes: changed ( 1000.0,... ): 127.810 KiB JetMET/MET
• DQMHistoSizes: changed ( 1000.0 ): 43.835 KiB JetMET/Jet
• DQMHistoSizes: changed ( 10224.0,... ): 43.851 KiB JetMET/Jet
• DQMHistoSizes: changed ( 13234.0,... ): 0.016 KiB JetMET/Jet
• Checked 202 log files, 165 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[nothingface0]

@smuzaffar we noticed failures when histograms are uploaded to the Dev DQMGUI:

Using SSL private key /afs/cern.ch/user/c/cmsbuild/.globus/userkey.pem
Using SSL public key /afs/cern.ch/user/c/cmsbuild/.globus/usercert.pem
Uploading output:
visDQMUpload.py https://cmsweb.cern.ch/dqm/dev /pool/condor/dir_2749906/jenkins/workspace/compare-root-files-short-matrix/cms-bot/dqm-comparison/dqmComparisonOutput/pr/DQM_V0001_R000000001__RelVal_wf25_0_pr__CMSSW_14_1_X-PRcmssw_44336-62360__DQMIO.root
ERROR HTTP Error 403: Forbidden
Status code:  None
Message:      None
Detail:       None

Could it be an expired certificate?

Logs here

[tjavaid]

please test

• just to populate the bin-by-bin comparison plots, after certificate issues fix.

[cmsbuild]

-1

Failed Tests: RelVals RelVals-INPUT Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-35c537/38892/summary.html COMMIT: 866d59b3b005b2de7d861a3c727c9d4471f4c8b5 CMSSW: CMSSW_14_1_X_2024-04-16-2300/el8_amd64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44336/38892/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

• 136.8311

----- Begin Fatal Exception 17-Apr-2024 14:56:05 CEST-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named step2_PAT_DQM.py
Exception Message:
 unknown python problem occurred.
RuntimeError: An exception of category 'FileInPathError' occurred.
Exception Message:
edm::FileInPath unable to find file RecoBTag/Combined/data/UParTAK4/PUPPI/V00/UParTAK4.onnx anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44336/38892/CMSSW_14_1_X_2024-04-16-2300/poison:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44336/38892/CMSSW_14_1_X_2024-04-16-2300/src:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44336/38892/CMSSW_14_1_X_2024-04-16-2300/external/el8_amd64_gcc12/data:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/poison:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/external/el8_amd64_gcc12/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/matrix-results/136.8311_RunJetHT2017FreMINIAOD


At:
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Types.py(887): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Mixins.py(385): insertContentsInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Mixins.py(520): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Modules.py(161): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Config.py(1217): _insertManyInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Config.py(1490): fillProcessDesc
  <string>(2): <module>

----- End Fatal Exception -------------------------------------------------

• 136.88811

----- Begin Fatal Exception 17-Apr-2024 14:56:05 CEST-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named step2_PAT_DQM.py
Exception Message:
 unknown python problem occurred.
RuntimeError: An exception of category 'FileInPathError' occurred.
Exception Message:
edm::FileInPath unable to find file RecoBTag/Combined/data/UParTAK4/PUPPI/V00/UParTAK4.onnx anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44336/38892/CMSSW_14_1_X_2024-04-16-2300/poison:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44336/38892/CMSSW_14_1_X_2024-04-16-2300/src:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44336/38892/CMSSW_14_1_X_2024-04-16-2300/external/el8_amd64_gcc12/data:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/poison:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/external/el8_amd64_gcc12/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/matrix-results/136.88811_RunJetHT2018DreMINIAODUL


At:
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Types.py(887): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Mixins.py(385): insertContentsInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Mixins.py(520): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Modules.py(161): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Config.py(1217): _insertManyInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Config.py(1490): fillProcessDesc
  <string>(2): <module>

----- End Fatal Exception -------------------------------------------------

• 136.7611

----- Begin Fatal Exception 17-Apr-2024 14:56:06 CEST-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named step2_PAT_DQM.py
Exception Message:
 unknown python problem occurred.
RuntimeError: An exception of category 'FileInPathError' occurred.
Exception Message:
edm::FileInPath unable to find file RecoBTag/Combined/data/UParTAK4/PUPPI/V00/UParTAK4.onnx anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44336/38892/CMSSW_14_1_X_2024-04-16-2300/poison:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44336/38892/CMSSW_14_1_X_2024-04-16-2300/src:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44336/38892/CMSSW_14_1_X_2024-04-16-2300/external/el8_amd64_gcc12/data:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/poison:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src:/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/external/el8_amd64_gcc12/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/matrix-results/136.7611_RunJetHT2016EreMINIAOD


At:
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Types.py(887): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Mixins.py(385): insertContentsInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Mixins.py(520): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Modules.py(161): insertInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Config.py(1217): _insertManyInto
  /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02833/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_1_X_2024-04-16-2300/src/FWCore/ParameterSet/python/Config.py(1490): fillProcessDesc
  <string>(2): <module>

----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

• 4.53
• 135.4
• 140.023
• 139.001
• 136.731
• 136.793
• 136.874
• 140.043
• 140.063
• 9.0
• 25.0
• 1306.0
• 1330.0
• 141.042
• 141.044
• 141.046
• 13234.0
• 14034.0
• 14234.0
• 11634.0
• 25202.0
• 12434.0
• 12834.0
• 12834.7
• 12846.0
• 10224.0
• 24900.0
• 24896.0
• 13034.0
• 24834.0
• 23234.0
• 250202.181
• 25034.999
• 24834.911

RelVals-INPUT

• 4.64.6_MinimumBias2010A/step2_MinimumBias2010A.log
• 136.72411136.72411_RunJetHT2016B_reminiaodUL/step2_RunJetHT2016B_reminiaodUL.log
• 136.72412136.72412_RunJetHT2016B_reminiaodUL/step2_RunJetHT2016B_reminiaodUL.log

Expand to see more relval errors ...

• 136.7611
• 136.76111
• 136.7721
• 136.77211
• 136.8311
• 136.83111
• 136.9
• 136.88811
• 140.201
• 140.202
• 1325.5
• 1325.51
• 1325.5161
• 1325.516
• 1325.517
• 1325.518
• 134.0
• 144.6
• 1002.0
• 13234.0
• 14034.0
• 13434.0
• 14234.0
• 2500.0
• 2500.001
• 2500.002
• 2500.011
• 2500.01
• 2500.012
• 2500.326
• 2500.404

[AdrianoDee]

please test with https://github.com/cms-data/RecoBTag-Combined/pull/57

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-35c537/38896/summary.html COMMIT: 866d59b3b005b2de7d861a3c727c9d4471f4c8b5 CMSSW: CMSSW_14_1_X_2024-04-17-1100/el8_amd64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44336/38896/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @belforte cms-sw/cmsdist#9138
• @cms-sw cms-sw/cmsdist#9141

You can see more details here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-35c537/38896/git-recent-commits.json https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-35c537/38896/git-merge-result

Comparison Summary

Summary:

• You potentially added 7 lines to the logs
• Reco comparison results: 2987 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3319599
• DQMHistoTests: Total failures: 6039
• DQMHistoTests: Total nulls: 195
• DQMHistoTests: Total successes: 3313345
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 3605.085 KiB( 47 files compared)
• DQMHistoSizes: changed ( 1000.0,... ): 127.810 KiB JetMET/MET
• DQMHistoSizes: changed ( 1000.0 ): 43.835 KiB JetMET/Jet
• DQMHistoSizes: changed ( 10224.0,... ): 43.851 KiB JetMET/Jet
• DQMHistoSizes: changed ( 13234.0,... ): 0.016 KiB JetMET/Jet
• Checked 202 log files, 165 edm output root files, 48 DQM output files
• TriggerResults: found differences in 1 / 46 workflows

[tjavaid]

Hi @rseidita, @AdrianoDee , I am wondering whether changes to WF 11634.0 are expected with this update? (see for muons and L1TEMU)

[rseidita]

Hi @tjavaid, I can't open the links apparently, did you use the "Link me" feature of the DQM GUI?

[rseidita]

Regardless, this PR only touches files in the DQMOffline/JetMET package, so no it should not affect anything in other workspaces as far as I know

[tjavaid]

Hi @rseidita , sure. Please consider corresponding links for muons and L1TEMU workspaces.

[rseidita]

Thanks @tjavaid, the links work now. Looking at the logs, I see that the difference is in the HLT results for some events:

Found 10 matching events, out of which 3 have different HLT results

      Events    Accepted      Gained        Lost       Other  Trigger
          10           9          +1          -2           -  Flag_ecalLaserCorrFilter

Could this be due to a change in ECAL laser corrections? My changes are entirely restricted to the JetMET offline DQM sequence, so I don't see how they can have anything to do with trigger results and/or the muon/L1T DQM sequence. Are we sure that the tests were run on exactly matching conditions?