cmssw Modified data product delete early system

PR description:

now uses internal consumes rather then adding a mightGet parameter to a module
now specify which data products reference data in other data products

PR validation:

Code compiles. Running workflow 1.0 works.

Aug 05 '22 19:08 Dr15Jones

please test

Aug 05 '22 19:08 Dr15Jones

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-38983/31432

This PR adds an extra 148KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/TestObjects/src/classes_def.xml modified in PR(s): #38806
- File FWCore/Framework/interface/EventProcessor.h modified in PR(s): #38801
- File FWCore/Framework/src/EventProcessor.cc modified in PR(s): #38801
- File FWCore/ParameterSet/python/Config.py modified in PR(s): #37996
- File FWCore/ParameterSet/src/validateTopLevelParameterSets.cc modified in PR(s): #38801

Aug 05 '22 19:08 cmsbuild

A new Pull Request was created by @Dr15Jones (Chris Jones) for master.

It involves the following packages:

Configuration/StandardSequences (operations)
DataFormats/Common (core)
DataFormats/TestObjects (core)
FWCore/Framework (core)
FWCore/ParameterSet (core)
FWCore/ServiceRegistry (core)
RecoTracker/Configuration (reconstruction)

@perrotta, @smuzaffar, @rappoccio, @Dr15Jones, @makortel, @clacaputo, @jpata, @qliphy, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks. @fabiocos, @VourMa, @makortel, @felicepantaleo, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @Martin-Grunewald, @missirol, @ebrondol, @wddgit, @lecriste, @fwyzard, @gpetruc, @mmusich, @mtosi, @dgulhan, @slomeo this is something you requested to watch as well. @perrotta, @dpiparo, @qliphy, @rappoccio you are the release manager for this.

cms-bot commands are listed here

Aug 05 '22 19:08 cmsbuild

-1

Failed Tests: UnitTests RelVals RelVals-INPUT AddOn Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-98c24e/26664/summary.html COMMIT: ae4cb26318776f804b9fad5bf3c6c612e7e87127 CMSSW: CMSSW_12_5_X_2022-08-05-1100/el8_amd64_gcc10 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/38983/26664/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestFWCoreParameterSetDriver had ERRORS
---> test testMuonTrackRefitting had ERRORS
---> test TestDQMOnlineClient-beampixel_dqm_sourceclient had ERRORS
---> test TestDQMOnlineClient-beam_dqm_sourceclient had ERRORS
and more ...

RelVals

136.731

----- Begin Fatal Exception 06-Aug-2022 04:32:29 CEST-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/data/Run2016B/SinglePhoton/RAW/v2/000/274/199/00000/007498A6-8426-E611-8BA8-02163E013754.root
      [b] Input file root://cms-xrd-global.cern.ch//store/data/Run2016B/SinglePhoton/RAW/v2/000/274/199/00000/007498A6-8426-E611-8BA8-02163E013754.root could not be opened.
      [c] Fatal Root Error: @SUB=TStreamerInfo::BuildCheck

   The StreamerInfo of class edm::Wrapper<GlobalObjectMapRecord> read from file root://cms-xrd-global.cern.ch//store/data/Run2016B/SinglePhoton/RAW/v2/000/274/199/00000/007498A6-8426-E611-8BA8-02163E013754.root
   has the same version (=3) as the active class but a different checksum.
   You should update the version to ClassDef(edm::Wrapper<GlobalObjectMapRecord>,4).
   Do not try to write objects with the current class definition,
   the files will not be readable.


----- End Fatal Exception -------------------------------------------------

136.793

----- Begin Fatal Exception 06-Aug-2022 04:32:29 CEST-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/data/Run2017C/DoubleEG/RAW/v1/000/301/998/00000/0073E73C-B68C-E711-9BF8-02163E019B6C.root
      [b] Input file root://cms-xrd-global.cern.ch//store/data/Run2017C/DoubleEG/RAW/v1/000/301/998/00000/0073E73C-B68C-E711-9BF8-02163E019B6C.root could not be opened.
      [c] Fatal Root Error: @SUB=TStreamerInfo::BuildCheck

   The StreamerInfo of class edm::Wrapper<GlobalObjectMapRecord> read from file root://cms-xrd-global.cern.ch//store/data/Run2017C/DoubleEG/RAW/v1/000/301/998/00000/0073E73C-B68C-E711-9BF8-02163E019B6C.root
   has the same version (=3) as the active class but a different checksum.
   You should update the version to ClassDef(edm::Wrapper<GlobalObjectMapRecord>,4).
   Do not try to write objects with the current class definition,
   the files will not be readable.


----- End Fatal Exception -------------------------------------------------

136.874

----- Begin Fatal Exception 06-Aug-2022 04:32:33 CEST-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/data/Run2018C/EGamma/RAW/v1/000/319/450/00000/084198F2-E583-E811-AE82-FA163E0AD27F.root
      [b] Input file root://cms-xrd-global.cern.ch//store/data/Run2018C/EGamma/RAW/v1/000/319/450/00000/084198F2-E583-E811-AE82-FA163E0AD27F.root could not be opened.
      [c] Fatal Root Error: @SUB=TStreamerInfo::BuildCheck

   The StreamerInfo of class edm::Wrapper<GlobalObjectMapRecord> read from file root://cms-xrd-global.cern.ch//store/data/Run2018C/EGamma/RAW/v1/000/319/450/00000/084198F2-E583-E811-AE82-FA163E0AD27F.root
   has the same version (=3) as the active class but a different checksum.
   You should update the version to ClassDef(edm::Wrapper<GlobalObjectMapRecord>,4).
   Do not try to write objects with the current class definition,
   the files will not be readable.


----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

RelVals-INPUT

6.06.0_SingleMuPt1+SingleMuPt1INPUT+DIGI+RECO+HARVEST/step2_SingleMuPt1+SingleMuPt1INPUT+DIGI+RECO+HARVEST.log
8.08.0_BeamHalo+BeamHaloINPUT+DIGICOS+RECOCOS+ALCABH+HARVESTCOS/step2_BeamHalo+BeamHaloINPUT+DIGICOS+RECOCOS+ALCABH+HARVESTCOS.log
9.09.0_Higgs200ChargedTaus+Higgs200ChargedTausINPUT+DIGI+RECO+HARVEST/step2_Higgs200ChargedTaus+Higgs200ChargedTausINPUT+DIGI+RECO+HARVEST.log

Expand to see more relval errors ...

AddOn Tests

pat1

----- Begin Fatal Exception 06-Aug-2022 04:35:27 CEST-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://cms-xrd-global.cern.ch//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root'
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root
      [b] Input file root://cms-xrd-global.cern.ch//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root could not be opened.
      [c] XrdCl::File::Open(name='root://cms-xrd-global.cern.ch//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] No servers are available to read the file.
' (errno=3011, code=400). No additional data servers were found.
      [d] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/relval/CMSSW_9_2_2/RelValProdTTbar_13/AODSIM/91X_mcRun2_asymptotic_v3-v1/10000/EEB99F74-DA4D-E711-A41C-0025905A48F2.root?tried=&xrdcl.requuid=f509b585-75d2-4ebc-97d5-520639fd3f7d
      [e] Problematic data server: cms-xrd-global.cern.ch:1094
      [f] Disabled source: cms-xrd-global.cern.ch:1094
----- End Fatal Exception -------------------------------------------------

hlt_data_Fake1

----- Begin Fatal Exception 06-Aug-2022 04:35:30 CEST-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/data/Run2015D/MuonEG/RAW/v1/000/256/677/00000/80950A90-745D-E511-92FD-02163E011C5D.root
      [b] Input file root://cms-xrd-global.cern.ch//store/data/Run2015D/MuonEG/RAW/v1/000/256/677/00000/80950A90-745D-E511-92FD-02163E011C5D.root could not be opened.
      [c] Fatal Root Error: @SUB=TStreamerInfo::BuildCheck

   The StreamerInfo of class edm::Wrapper<L1GlobalTriggerObjectMapRecord> read from file root://cms-xrd-global.cern.ch//store/data/Run2015D/MuonEG/RAW/v1/000/256/677/00000/80950A90-745D-E511-92FD-02163E011C5D.root
   has the same version (=3) as the active class but a different checksum.
   You should update the version to ClassDef(edm::Wrapper<L1GlobalTriggerObjectMapRecord>,4).
   Do not try to write objects with the current class definition,
   the files will not be readable.


----- End Fatal Exception -------------------------------------------------

hlt_data_Fake1

----- Begin Fatal Exception 06-Aug-2022 04:35:43 CEST-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling File::sysopen()
Exception Message:
Failed to open the file 'RelVal_Raw_Fake1_DATA.root'
   Additional Info:
      [a] Input file file:RelVal_Raw_Fake1_DATA.root could not be opened.
      [b] open() failed with system error 'No such file or directory' (error code 2)
----- End Fatal Exception -------------------------------------------------

Expand to see more addon errors ...

Aug 06 '22 03:08 cmsbuild

please test

Aug 06 '22 19:08 Dr15Jones

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-38983/31438

This PR adds an extra 156KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File FWCore/Framework/interface/EventProcessor.h modified in PR(s): #38801
- File FWCore/Framework/src/EventProcessor.cc modified in PR(s): #38801
- File FWCore/ParameterSet/python/Config.py modified in PR(s): #37996
- File FWCore/ParameterSet/src/validateTopLevelParameterSets.cc modified in PR(s): #38801

Aug 06 '22 19:08 cmsbuild

Pull request #38983 was updated. @perrotta, @smuzaffar, @rappoccio, @Dr15Jones, @makortel, @clacaputo, @jpata, @qliphy, @fabiocos, @davidlange6 can you please check and sign again.

Aug 06 '22 19:08 cmsbuild

-1

Failed Tests: UnitTests Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-98c24e/26680/summary.html COMMIT: ff958cbfde7d54190a220f2d7caf7ff4f3c2d95a CMSSW: CMSSW_12_5_X_2022-08-06-1100/el8_amd64_gcc10 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/38983/26680/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestCTPPSDirectProtonSimulation had ERRORS

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3691612
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3691582
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.004 KiB( 50 files compared)
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 212 log files, 49 edm output root files, 51 DQM output files
TriggerResults: no differences found

Aug 07 '22 03:08 cmsbuild

I ran the test TestCTPPSDirectProtonSimulation 3 times in my local checkout and it ran just fine. I'll run the tests again to double check.

Aug 07 '22 13:08 Dr15Jones

please test

Aug 07 '22 13:08 Dr15Jones

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-98c24e/26686/summary.html COMMIT: ff958cbfde7d54190a220f2d7caf7ff4f3c2d95a CMSSW: CMSSW_12_5_X_2022-08-07-0000/el8_amd64_gcc10 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38983/26686/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3691612
DQMHistoTests: Total failures: 2
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3691588
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
Checked 212 log files, 49 edm output root files, 51 DQM output files
TriggerResults: no differences found

Aug 07 '22 20:08 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-38983/31537

This PR adds an extra 96KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File FWCore/Framework/interface/EventProcessor.h modified in PR(s): #38801
- File FWCore/Framework/src/EventProcessor.cc modified in PR(s): #38801
- File FWCore/ParameterSet/python/Config.py modified in PR(s): #37996
- File FWCore/ParameterSet/src/validateTopLevelParameterSets.cc modified in PR(s): #38801

Aug 12 '22 20:08 cmsbuild

Pull request #38983 was updated. @perrotta, @smuzaffar, @rappoccio, @Dr15Jones, @makortel, @clacaputo, @cmsbuild, @jpata, @qliphy, @mandrenguyen, @fabiocos, @davidlange6 can you please check and sign again.

Aug 12 '22 20:08 cmsbuild

please test

Aug 12 '22 20:08 Dr15Jones

-1

Failed Tests: UnitTests Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-98c24e/26791/summary.html COMMIT: 27d016a3c2777af0a7a33a21808031fb895dd82d CMSSW: CMSSW_12_5_X_2022-08-12-1100/el8_amd64_gcc10 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38983/26791/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test ExpressionEvaluatorUnitTest had ERRORS

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3692476
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3692446
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 50 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 212 log files, 49 edm output root files, 51 DQM output files
TriggerResults: no differences found

Aug 13 '22 06:08 cmsbuild

The unit test failure looks like a glitch

terminate called after throwing an instance of 'cms::Exception'
  what():  An exception of category 'ExpressionEvaluator' occurred.
Exception Message:
compilation/linking failed
c++ -H -Wall -shared -Winvalid-pch -DGNU_GCC -D_GNU_SOURCE -DEIGEN_DONT_PARALLELIZE -DTBB_USE_GLIBCXX_VERSION=100300 -DTBB_SUPPRESS_DEPRECATED_MESSAGES -DTBB_PREVIEW_RESUMABLE_TASKS=1 -DTBB_PREVIEW_TASK_GROUP_EXTENSIONS=1 -DBOOST_SPIRIT_THREADSAFE -DPHOENIX_THREADSAFE -DBOOST_MATH_DISABLE_STD_FPCLASSIFY -DBOOST_UUID_RANDOM_PROVIDER_FORCE_POSIX -DCMSSW_GIT_HASH='"CMSSW_12_5_X_2022-08-12-1100"' -DPROJECT_NAME='"CMSSW"' -DPROJECT_VERSION='"CMSSW_12_5_X_2022-08-12-1100"'     -isystem/cvmfs/cms-ib.cern.ch/nweek-02745/el8_amd64_gcc10/external/boost/1.78.0-12075919175e8d078539685f9234134a/include    -isystem/cvmfs/cms-ib.cern.ch/nweek-02745/el8_amd64_gcc10/lcg/root/6.24.07-a31cbfc28a0c92b3c007615905b5b9b2/include -isystem/cvmfs/cms-ib.cern.ch/nweek-02745/el8_amd64_gcc10/external/tbb/v2021.5.0-3cd580209e999b2fb4f8344347204353/include        -O2 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++1z -ftree-vectorize -Wstrict-overflow -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -msse3 -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wextra -Wpessimizing-move -Wclass-memaccess -Wno-cast-function-type -Wno-unused-but-set-parameter -Wno-ignored-qualifiers -Wno-deprecated-copy -Wno-unused-parameter -Wunused -Wparentheses -Wno-deprecated -Werror=return-type -Werror=missing-braces -Werror=unused-value -Werror=unused-label -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Wnon-virtual-dtor -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -Wno-error=unused-variable -DBOOST_DISABLE_ASSERTS -DUSE_CMS_DEPRECATED -fPIC -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_5_X_2022-08-12-1100/include/el8_amd64_gcc10/ -o /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_5_X_2022-08-12-1100/tmp/VI_4be09e4db634444cbc4b683a6d0aa30c.so /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_5_X_2022-08-12-1100/tmp/VI_4be09e4db634444cbc4b683a6d0aa30c.cc 2>&1
dlerror /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_5_X_2022-08-12-1100/tmp/VI_4be09e4db634444cbc4b683a6d0aa30c.so: cannot open shared object file: No such file or directory 

/bin/sh: line 1: 31121 Aborted                 timeout 3600 ExpressionEvaluatorUnitTest

Aug 15 '22 15:08 makortel

@cmsbuild, please test

Aug 15 '22 15:08 makortel

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-98c24e/26827/summary.html COMMIT: 27d016a3c2777af0a7a33a21808031fb895dd82d CMSSW: CMSSW_12_5_X_2022-08-15-1100/el8_amd64_gcc10 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/38983/26827/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3692476
DQMHistoTests: Total failures: 8
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3692446
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
Checked 212 log files, 49 edm output root files, 51 DQM output files
TriggerResults: no differences found

Aug 15 '22 22:08 cmsbuild

@cmsbuild please test

Aug 26 '22 12:08 jpata

-1

Failed Tests: UnitTests Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-98c24e/27121/summary.html COMMIT: 27d016a3c2777af0a7a33a21808031fb895dd82d CMSSW: CMSSW_12_5_X_2022-08-25-2300/el8_amd64_gcc10 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38983/27121/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestIOPoolInputNoParentDictionary had ERRORS

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3695708
DQMHistoTests: Total failures: 15
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3695670
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 50 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 212 log files, 49 edm output root files, 51 DQM output files
TriggerResults: no differences found

Aug 26 '22 20:08 cmsbuild

Hmh, the failure is

Read the file without TransientIntParentT<1> dictionary
26-Aug-2022 19:07:22 CEST  Initiating request to open file file:noparentdict_step1.root
26-Aug-2022 19:07:23 CEST  Successfully opened file file:noparentdict_step1.root
Begin processing the 1st record. Run 621, Event 1, LumiSection 6 on stream 0 at 26-Aug-2022 19:07:23.190 CEST
----- Begin Fatal Exception 26-Aug-2022 19:07:23 CEST-----------------------
An exception of category 'ValueMismatch' occurred while
   [0] Processing  Event run: 621 lumi: 6 event: 1 stream: 0
   [1] Running path 'p'
   [2] Calling method for module IntTestAnalyzer/'analysis'
Exception Message:
The value for "intProduct::" is -1966376447 but it was supposed to be 1
----- End Fatal Exception -------------------------------------------------

The failure comes from the "step 2" of the test, which only reads a product from an input file that was produced by the "step 1" of the test. The step 1 configuration does not do any early deletion.

Aug 26 '22 20:08 makortel

@cmsbuild, please test

Let's see if the failure persists

Aug 26 '22 20:08 makortel

-1

Failed Tests: UnitTests Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-98c24e/27133/summary.html COMMIT: 27d016a3c2777af0a7a33a21808031fb895dd82d CMSSW: CMSSW_12_5_X_2022-08-25-2300/el8_amd64_gcc10 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38983/27133/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestIOPoolInputNoParentDictionary had ERRORS

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3695708
DQMHistoTests: Total failures: 14
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3695672
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
Checked 212 log files, 49 edm output root files, 51 DQM output files
TriggerResults: no differences found

Aug 27 '22 05:08 cmsbuild

Failure is the same except for the actual value (The value for "intProduct::" is 65537 but it was supposed to be 1) so the edmtest::IntProcuct::value seems to be uninitialized when edmtest::IntTestAnalyzer accesses it.

Aug 29 '22 13:08 makortel

So running my tests locally (based on CMSSW_12_5_0_pre4) everything is fine. Wonder if it is an interaction with a more recently merged pull request.

Aug 29 '22 14:08 Dr15Jones

Wonder if it is an interaction with a more recently merged pull request.

Very likely since https://github.com/cms-sw/cmssw/pull/38806 (where I added that test) was include in pre5 but not in pre4.

Aug 29 '22 14:08 makortel

So I was able to reproduce the problem (though subsequent run returned a different value). However, when I dumped the content of the file it was the value stated

root [2] Events->Scan("edmtestIntProduct_intProduct__TESTPROD.obj.value")
************************
*    Row   * edmtestIn *
************************
*        0 *     32513 *
*        1 *     32513 *
*        2 *         1 *
*        3 *     32513 *
*        4 *     32513 *
*        5 *         1 *
*        6 *         1 *
*        7 *         1 *
*        8 *      -255 *
*        9 *         1 *
*       10 *         1 *
************************

Therefore the problem is in step1 of the test.

Aug 29 '22 22:08 Dr15Jones

@makortel I'm betting the failure is the Wrapper change. In your test, you check out the base, but that does NOT include this pull request. So the test is vulnerable to an ABI change (like happens here).

Aug 29 '22 22:08 Dr15Jones

Here is some further items. When I run the unit test via scram, I see the contents of the file are

root [1] Events->Print()
******************************************************************************
*Tree    :Events    :                                                        *
*Entries :       11 : Total =           11993 bytes  File  Size =       2727 *
*        :          : Tree compression factor =   2.71                       *
******************************************************************************
*Br    0 :EventAuxiliary : edm::EventAuxiliary                               *
*Entries :       11 : Total  Size=       1859 bytes  File Size  =        330 *
*Baskets :        1 : Basket Size=      16384 bytes  Compression=   4.04     *
*............................................................................*
*Br    1 :EventProductProvenance : vector<edm::StoredProductProvenance>      *
*Entries :       11 : Total  Size=       1096 bytes  File Size  =        181 *
*Baskets :        1 : Basket Size=      16384 bytes  Compression=   2.98     *
*............................................................................*
*Br    2 :EventSelections : vector<edm::Hash<1> >                            *
*Entries :       11 : Total  Size=       1050 bytes  File Size  =        183 *
*Baskets :        1 : Basket Size=      16384 bytes  Compression=   2.85     *
*............................................................................*
*Br    3 :BranchListIndexes : vector<unsigned short>                         *
*Entries :       11 : Total  Size=        807 bytes  File Size  =        147 *
*Baskets :        1 : Basket Size=      16384 bytes  Compression=   1.84     *
*............................................................................*
*Branch  :edmTriggerResults_TriggerResults__TESTPROD.                        *
*Entries :       11 : BranchElement (see below)                              *
*............................................................................*
*Br    4 :edmTriggerResults_TriggerResults__TESTPROD.obj : edm:              *
*         | :TriggerResults                                                  *
*Entries :       11 : Total  Size=       1601 bytes  File Size  =        230 *
*Baskets :        1 : Basket Size=      16384 bytes  Compression=   4.12     *
*............................................................................*
*Br    5 :edmTriggerResults_TriggerResults__TESTPROD.present : Bool_t        *
*Entries :       11 : Total  Size=        799 bytes  File Size  =        130 *
*Baskets :        1 : Basket Size=      16384 bytes  Compression=   1.00     *
*............................................................................*
*Branch  :edmtestIntProduct_intProduct__TESTPROD.                            *
*Entries :       11 : BranchElement (see below)                              *
*............................................................................*
*Br    6 :edmtestIntProduct_intProduct__TESTPROD.present : Bool_t            *
*Entries :       11 : Total  Size=        779 bytes  File Size  =        126 *
*Baskets :        1 : Basket Size=      16384 bytes  Compression=   1.00     *
*............................................................................*
*Br    7 :edmtestIntProduct_intProduct__TESTPROD.obj :                       *
*Entries :       11 : Total  Size=       1237 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      16384 bytes  Compression=   1.00     *
*............................................................................*
*Br    8 :edmtestIntProduct_intProduct__TESTPROD.obj.value : Int_t           *
*Entries :       11 : Total  Size=        822 bytes  File Size  =        161 *
*Baskets :        1 : Basket Size=      16384 bytes  Compression=   1.00     *
*............................................................................*

Notice the order difference of present and obj between edmTriggerResults and edmtestIntProduct. If I go into the temporary workspace setup by the unit test (via a new bash shel) and do cmsenv and then run the step1 test, I get the results of the file as

root [1] Events->Print()
******************************************************************************
*Tree    :Events    :                                                        *
*Entries :       11 : Total =           11993 bytes  File  Size =       2702 *
*        :          : Tree compression factor =   2.75                       *
******************************************************************************
*Br    0 :EventAuxiliary : edm::EventAuxiliary                               *
*Entries :       11 : Total  Size=       1859 bytes  File Size  =        325 *
*Baskets :        1 : Basket Size=      16384 bytes  Compression=   4.10     *
*............................................................................*
*Br    1 :EventProductProvenance : vector<edm::StoredProductProvenance>      *
*Entries :       11 : Total  Size=       1096 bytes  File Size  =        181 *
*Baskets :        1 : Basket Size=      16384 bytes  Compression=   2.98     *
*............................................................................*
*Br    2 :EventSelections : vector<edm::Hash<1> >                            *
*Entries :       11 : Total  Size=       1050 bytes  File Size  =        183 *
*Baskets :        1 : Basket Size=      16384 bytes  Compression=   2.85     *
*............................................................................*
*Br    3 :BranchListIndexes : vector<unsigned short>                         *
*Entries :       11 : Total  Size=        807 bytes  File Size  =        147 *
*Baskets :        1 : Basket Size=      16384 bytes  Compression=   1.84     *
*............................................................................*
*Branch  :edmTriggerResults_TriggerResults__TESTPROD.                        *
*Entries :       11 : BranchElement (see below)                              *
*............................................................................*
*Br    4 :edmTriggerResults_TriggerResults__TESTPROD.present : Bool_t        *
*Entries :       11 : Total  Size=        799 bytes  File Size  =        130 *
*Baskets :        1 : Basket Size=      16384 bytes  Compression=   1.00     *
*............................................................................*
*Br    5 :edmTriggerResults_TriggerResults__TESTPROD.obj : edm:              *
*         | :TriggerResults                                                  *
*Entries :       11 : Total  Size=       1601 bytes  File Size  =        230 *
*Baskets :        1 : Basket Size=      16384 bytes  Compression=   4.12     *
*............................................................................*
*Branch  :edmtestIntProduct_intProduct__TESTPROD.                            *
*Entries :       11 : BranchElement (see below)                              *
*............................................................................*
*Br    6 :edmtestIntProduct_intProduct__TESTPROD.present : Bool_t            *
*Entries :       11 : Total  Size=        779 bytes  File Size  =        126 *
*Baskets :        1 : Basket Size=      16384 bytes  Compression=   1.00     *
*............................................................................*
*Br    7 :edmtestIntProduct_intProduct__TESTPROD.obj :                       *
*Entries :       11 : Total  Size=       1237 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      16384 bytes  Compression=   1.00     *
*............................................................................*
*Br    8 :edmtestIntProduct_intProduct__TESTPROD.obj.value : Int_t           *
*Entries :       11 : Total  Size=        822 bytes  File Size  =        145 *
*Baskets :        1 : Basket Size=      16384 bytes  Compression=   1.11     *
*............................................................................*

With the order in the two data products being the same (and being the 'old' order).

Aug 29 '22 22:08 Dr15Jones

If I take the file I hand ran and use if for the step2 part of the unit test, the job runs fine.

Aug 29 '22 22:08 Dr15Jones

cmssw cmssw copied to clipboard

Modified data product delete early system

PR description:

PR validation:

Unit Tests

RelVals

RelVals-INPUT

AddOn Tests

Unit Tests

Comparison Summary

Comparison Summary

Unit Tests

Comparison Summary

Comparison Summary

Unit Tests

Comparison Summary

Unit Tests

Comparison Summary

cmssw
cmssw copied to clipboard