cmssw introducing BigPixel for phase-2 InnerTracker in T39 geometry

PR description:

This PR is a follow-up of the draft PR#42454 (closed in Jan 2024).

The PR introduces the so-called BigPixels (elongated pixels between two readout chips) for the phase-2 InnerTracker following the implementation proposed in:

https://indico.cern.ch/event/1382902/contributions/5816585/attachments/2805475/4895325/Phase-2_IT_BigPix_TrkDPG.pdf

Description. BigPixels are introduced in the file pixelStructureTopology.xml or by using the TrackerAdditionalParametersPerDet rcd for geometries loaded from DB (none yet). Therefore, the PR restores the changes of the infrastructure needed for using the TrackerAdditionalParametersPerDet rcd, introduced for the bricked pixels and then removed with PR#40443 but without reintroducing the functions related to bricked pixels. NOTE (similar to PR#34120): the record PTrackerAdditionalParametersPerDetRcd is consumed by the TrackerGeomBuilderFromGeometricDet, which is the same for Runs 2&3 and Phase 2. Therefore, this PR also modifies Runs 2&3, by including the new record.
Tracker geometry. The Tracker phase-2 test geometry in which BigPixels are introduced is T39, thus skipping versions T36-T37-T38 which are currently under development by @Raffaella07.
Implementation. Phase-2 pixels are now described by the class RectangularPhase2PixelTopology inheriting from PixelTopology. Code for Tracker local reco was adapted to work both the phase-1 pixels (RectangularPixelTopology) and with phase-2 pixels (RectangularPhase2PixelTopology). The function siPixelUtils::generic_position_formula(), used in PixelCPEGeneric, was modified in order to be used both for phase-1 and phase-2 BigPixels. A side-effect was to add to the abstract class PixelTopology the methods getPixelFractionInX() and getPixelFractionInY(). @fabiocos @martinamalberti: the implementation of these fcn in ProxyMTDTopology is just a placeholder to compile the code. Please let us know what the actual code should be.

PR validation:

compilation ok after git cms-checkdeps -a -A scram b code-checks, code-format scram b runtests runTheMatrix.py -l limited -i all --ibeos

No changes are expected on phase-1 (set of tracking validation plots on a single muon sample on phase-1 geometry) http://personalpages.to.infn.it/~migliore/BigPixels/SingleMuPt100-phase1/plots-20240323/

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

This PR is not a backport and no backport needed.

Specific w/f for test on T39: 30034.0

@gbardell @langufo @bartosik-hep @mmusich @sroychow @mdelcourt FYI

Mar 28 '24 23:03 emiglior

cms-bot internal usage

Mar 28 '24 23:03 cmsbuild

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44576/39731

This PR adds an extra 492KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): #33532, #43761, #43964
- File HLTrigger/Configuration/python/customizeHLTforCMSSW.py modified in PR(s): #43964

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44576/39731/code-format.patch e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44576/39731/code-format.patch | patch -p1 You can also run scram build code-format to apply code format directly

Mar 28 '24 23:03 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44576/39732

This PR adds an extra 480KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): #33532, #43761, #43964
- File HLTrigger/Configuration/python/customizeHLTforCMSSW.py modified in PR(s): #43964

Mar 29 '24 05:03 cmsbuild

A new Pull Request was created by @emiglior for master.

It involves the following packages:

Alignment/CommonAlignmentMonitor (alca)
Alignment/CommonAlignmentProducer (alca)
Alignment/LaserAlignment (alca)
Alignment/MillePedeAlignmentAlgorithm (alca)
Alignment/OfflineValidation (alca)
Alignment/SurveyAnalysis (alca)
Alignment/TrackerAlignment (alca)
CondFormats/GeometryObjects (alca, db)
CondFormats/SiPixelTransient (reconstruction, db)
CondTools/Geometry (db)
Configuration/Geometry (geometry, upgrade)
Configuration/PyReleaseValidation (pdmv, upgrade)
Configuration/StandardSequences (operations)
DQM/SiTrackerPhase2 (dqm)
Geometry/CMSCommonData (geometry, upgrade)
Geometry/CommonTopologies (geometry)
Geometry/MTDGeometryBuilder (geometry, upgrade)
Geometry/TrackerCommonData (geometry)
Geometry/TrackerGeometryBuilder (geometry)
Geometry/TrackerNumberingBuilder (geometry)
HLTrigger/Configuration (hlt)
L1Trigger/L1TTrackMatch (upgrade, l1)
L1Trigger/TrackFindingTracklet (l1)
L1Trigger/TrackTrigger (upgrade, l1)
RecoHI/HiTracking (reconstruction)
RecoLocalTracker/SiPixelRecHits (reconstruction)
RecoLocalTracker/SubCollectionProducers (reconstruction)
RecoLuminosity/LumiProducer (reconstruction)
RecoTracker/PixelLowPtUtilities (reconstruction)
SLHCUpgradeSimulations/Geometry (geometry, upgrade)

@sunilUIET, @rvenditti, @tjavaid, @AdrianoDee, @perrotta, @subirsarkar, @bsunanda, @civanch, @saumyaphor4252, @mmusich, @mandrenguyen, @consuegs, @Dr15Jones, @fabiocos, @aloeliger, @Martin-Grunewald, @davidlange6, @epalencia, @srimanob, @cmsbuild, @rappoccio, @mdhildreth, @nothingface0, @antoniovilela, @miquork, @antoniovagnerini, @francescobrivio, @jfernan2, @syuvivida, @makortel can you please review it and eventually sign? Thanks. @jazzitup, @ferencek, @vargasa, @mtosi, @yduhm, @ghugo83, @sroychow, @threus, @slomeo, @VourMa, @gbenelli, @alesaggio, @mroguljic, @venturia, @sviret, @tlampen, @GiacomoSguazzoni, @bsunanda, @seemasharmafnal, @gpetruc, @skinnari, @arossi83, @mmusich, @mandrenguyen, @VinInn, @felicepantaleo, @tocheng, @silviodonato, @erikbutz, @fabiocos, @JanChyczynski, @echabert, @tsusa, @sameasy, @pakhotin, @Martin-Grunewald, @yuanchao, @missirol, @dgulhan, @yetkinyilmaz, @jlidrych, @robervalwalsh, @tvami, @rovere, @kurtejung, @PonIlya, @JanFSchulte, @rsreds, @dkotlins, @yenjie, @adewit, @makortel this is something you requested to watch as well. @sextonkennedy, @rappoccio, @antoniovilela you are the release manager for this.

cms-bot commands are listed here

Mar 29 '24 05:03 cmsbuild

@cmsbuild, please test

Mar 29 '24 07:03 mmusich

In addition to the discussion above at https://github.com/cms-sw/cmssw/pull/44576#discussion_r1544282401 (existing HLT menus in release need to be patched in order to support the new record), the PR is also failing the unit tests:

Mar 29 '24 09:03 mmusich

-1

Failed Tests: UnitTests AddOn Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3f5fe0/38502/summary.html COMMIT: 12d11a20b4d5d9c8aace691bb0c5b93e4bd2c62d CMSSW: CMSSW_14_1_X_2024-03-28-2300/el8_amd64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44576/38502/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 2 errors in the following unit tests:

---> test test2021Geometry had ERRORS
---> test test2026Geometry had ERRORS

AddOn Tests

hlt_mc_GRun

----- Begin Fatal Exception 29-Mar-2024 10:09:46 CET-----------------------
An exception of category 'NoRecord' occurred while
   [0] Processing global begin Run run: 1
   [1] Prefetching for module SiPixelPhase1CompareRecHitsSoA/'hltSiPixelRecHitsSoACompareGPUvsCPU'
   [2] Calling method for EventSetup module TrackerDigiGeometryESModule/''
   [3] While getting dependent Record from Record TrackerDigiGeometryRecord
Exception Message:
No "PTrackerAdditionalParametersPerDetRcd" record found in the EventSetup.

 Please add an ESSource or ESProducer that delivers such a record.
----- End Fatal Exception -------------------------------------------------

hlt_mc_HIon

----- Begin Fatal Exception 29-Mar-2024 10:09:28 CET-----------------------
An exception of category 'NoRecord' occurred while
   [0] Processing global begin Run run: 1
   [1] Prefetching for module SiPixelHIonPhase1CompareRecHitsSoA/'hltSiPixelRecHitsSoACompareGPUvsCPUPPOnAA'
   [2] Calling method for EventSetup module TrackerDigiGeometryESModule/''
   [3] While getting dependent Record from Record TrackerDigiGeometryRecord
Exception Message:
No "PTrackerAdditionalParametersPerDetRcd" record found in the EventSetup.

 Please add an ESSource or ESProducer that delivers such a record.
----- End Fatal Exception -------------------------------------------------

hlt_mc_PRef

----- Begin Fatal Exception 29-Mar-2024 10:09:26 CET-----------------------
An exception of category 'NoRecord' occurred while
   [0] Processing global begin Run run: 1
   [1] Prefetching for module SiPixelPhase1CompareRecHitsSoA/'hltSiPixelRecHitsSoACompareGPUvsCPU'
   [2] Calling method for EventSetup module TrackerDigiGeometryESModule/''
   [3] While getting dependent Record from Record TrackerDigiGeometryRecord
Exception Message:
No "PTrackerAdditionalParametersPerDetRcd" record found in the EventSetup.

 Please add an ESSource or ESProducer that delivers such a record.
----- End Fatal Exception -------------------------------------------------

Expand to see more addon errors ...

Comparison Summary

Summary:

You potentially removed 10 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 3838 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3297469
DQMHistoTests: Total failures: 14181
DQMHistoTests: Total nulls: 163
DQMHistoTests: Total successes: 3283105
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 221968.83 KiB( 47 files compared)
DQMHistoSizes: changed ( 23234.0,... ): 36994.805 KiB TrackerPhase2ITDigi/DigiMonitor
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: found differences in 1 / 46 workflows

Mar 29 '24 13:03 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44576/39735

This PR adds an extra 492KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): #33532, #43761, #43964
- File HLTrigger/Configuration/python/customizeHLTforCMSSW.py modified in PR(s): #43964

Mar 29 '24 14:03 cmsbuild

Pull request #44576 was updated. @fabiocos, @aloeliger, @mandrenguyen, @antoniovilela, @miquork, @nothingface0, @cmsbuild, @tjavaid, @rvenditti, @mdhildreth, @srimanob, @perrotta, @davidlange6, @AdrianoDee, @makortel, @consuegs, @Dr15Jones, @epalencia, @jfernan2, @rappoccio, @sunilUIET, @bsunanda, @syuvivida, @antoniovagnerini, @civanch, @Martin-Grunewald, @saumyaphor4252, @francescobrivio, @mmusich, @subirsarkar can you please check and sign again.

Mar 29 '24 14:03 cmsbuild

@Martin-Grunewald @mmusich thanks for the feedback - please wait to restart testing - still fixin the error in the test2024 and test2026 units tests

Mar 29 '24 15:03 emiglior

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44576/39751

This PR adds an extra 28KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): #33532, #43761, #43964
- File HLTrigger/Configuration/python/customizeHLTforCMSSW.py modified in PR(s): #43964

Apr 02 '24 08:04 cmsbuild

Pull request #44576 was updated. @antoniovilela, @civanch, @tjavaid, @nothingface0, @mdhildreth, @cmsbuild, @mmusich, @subirsarkar, @perrotta, @rvenditti, @miquork, @Dr15Jones, @aloeliger, @sunilUIET, @rappoccio, @fabiocos, @saumyaphor4252, @francescobrivio, @jfernan2, @AdrianoDee, @Martin-Grunewald, @syuvivida, @makortel, @epalencia, @bsunanda, @antoniovagnerini, @consuegs, @srimanob, @davidlange6, @mandrenguyen can you please check and sign again.

Apr 02 '24 08:04 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44576/39752

This PR adds an extra 504KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): #33532, #43761, #43964
- File HLTrigger/Configuration/python/customizeHLTforCMSSW.py modified in PR(s): #43964

Apr 02 '24 08:04 cmsbuild

before testing again, would it be possible to squash the commits (in particular to get rid of the merges of PRs to the branch)?

Apr 02 '24 08:04 mmusich

Pull request #44576 was updated. @civanch, @srimanob, @epalencia, @jfernan2, @rappoccio, @sunilUIET, @miquork, @mdhildreth, @nothingface0, @syuvivida, @mandrenguyen, @Dr15Jones, @antoniovagnerini, @antoniovilela, @consuegs, @perrotta, @bsunanda, @cmsbuild, @aloeliger, @saumyaphor4252, @francescobrivio, @davidlange6, @rvenditti, @tjavaid, @mmusich, @Martin-Grunewald, @makortel, @AdrianoDee, @fabiocos, @subirsarkar can you please check and sign again.

Apr 02 '24 08:04 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44576/39754

This PR adds an extra 484KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): #33532, #43761, #43964
- File HLTrigger/Configuration/python/customizeHLTforCMSSW.py modified in PR(s): #43964

Apr 02 '24 11:04 cmsbuild

Pull request #44576 was updated. @makortel, @Martin-Grunewald, @epalencia, @mdhildreth, @fabiocos, @civanch, @antoniovagnerini, @mandrenguyen, @bsunanda, @Dr15Jones, @rvenditti, @consuegs, @syuvivida, @rappoccio, @subirsarkar, @jfernan2, @nothingface0, @cmsbuild, @antoniovilela, @mmusich, @francescobrivio, @srimanob, @AdrianoDee, @aloeliger, @sunilUIET, @davidlange6, @miquork, @saumyaphor4252, @perrotta, @tjavaid can you please check and sign again.

Apr 02 '24 11:04 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44576/39756

This PR adds an extra 496KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): #33532, #43761, #43964
- File HLTrigger/Configuration/python/customizeHLTforCMSSW.py modified in PR(s): #43964

Apr 02 '24 13:04 cmsbuild

Pull request #44576 was updated. @jfernan2, @Martin-Grunewald, @civanch, @cmsbuild, @makortel, @aloeliger, @mdhildreth, @francescobrivio, @subirsarkar, @saumyaphor4252, @sunilUIET, @perrotta, @mandrenguyen, @rappoccio, @rvenditti, @nothingface0, @antoniovagnerini, @bsunanda, @mmusich, @AdrianoDee, @tjavaid, @Dr15Jones, @consuegs, @davidlange6, @antoniovilela, @srimanob, @miquork, @fabiocos, @epalencia, @syuvivida can you please check and sign again.

Apr 02 '24 13:04 cmsbuild

@cmsbuild, please test

Apr 02 '24 13:04 mmusich

@cmsbuild, please abort

Apr 02 '24 13:04 mmusich

test parameters:

workflow_opts = -w upgrade
workflows = 30034.0

Apr 02 '24 13:04 mmusich

@cmsbuild, please test

Apr 02 '24 13:04 mmusich

-1

Failed Tests: UnitTests RelVals RelVals-INPUT AddOn Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3f5fe0/38545/summary.html COMMIT: 95cdac5bb691b63092640b4728746508f509b7cc CMSSW: CMSSW_14_1_X_2024-04-02-1100/el8_amd64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44576/38545/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 3 errors in the following unit tests:

---> test test2021Geometry had ERRORS
---> test test2026Geometry had ERRORS
---> test testPhase2PixelNtuple had ERRORS

RelVals

4.534.53_RunPhoton2012B/step2_RunPhoton2012B.log
136.731136.731_RunSinglePh2016B/step2_RunSinglePh2016B.log
136.793136.793_RunDoubleEG2017C/step2_RunDoubleEG2017C.log

Expand to see more relval errors ...

RelVals-INPUT

4.174.17_RunMinBias2011A/step2_RunMinBias2011A.log
4.234.23_ValSkim2011A/step2_ValSkim2011A.log
4.264.26_ZMuSkim2011A/step2_ZMuSkim2011A.log

Expand to see more relval errors ...

AddOn Tests

hlt_mc_Fake

----- Begin Fatal Exception 02-Apr-2024 17:01:16 CEST-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named /data/cmsbld/jenkins/workspace/ib-run-pr-addon/CMSSW_14_1_X_2024-04-02-1100/src/HLTrigger/Configuration/test/OnLine_HLT_Fake.py
Exception Message:
 unknown python problem occurred.
NameError: name 'customizeHLTfor44576' is not defined

At:
  /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44576/38545/CMSSW_14_1_X_2024-04-02-1100/src/HLTrigger/Configuration/python/customizeHLTforCMSSW.py(279): customizeHLTforCMSSW
  /data/cmsbld/jenkins/workspace/ib-run-pr-addon/CMSSW_14_1_X_2024-04-02-1100/src/HLTrigger/Configuration/test/OnLine_HLT_Fake.py(383): <module>

----- End Fatal Exception -------------------------------------------------

hlt_mc_Fake

[hlt_mc_Fake:3] cmsDriver.py RelVal -s HLT:Fake,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run1_mc_Fake --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --customise=HLTrigger/Configuration/CustomConfigs.HLTRECO --processName=HLTRECO --filein file:RelVal_Raw_Fake_MC.root --fileout file:RelVal_Raw_Fake_MC_HLT_RECO.root : FAILED - elapsed time: 21 sec (ended on Tue Apr  2 17:01:37 2024) - exit: 256

hlt_mc_Fake1

----- Begin Fatal Exception 02-Apr-2024 17:01:09 CEST-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named /data/cmsbld/jenkins/workspace/ib-run-pr-addon/CMSSW_14_1_X_2024-04-02-1100/src/HLTrigger/Configuration/test/OnLine_HLT_Fake1.py
Exception Message:
 unknown python problem occurred.
NameError: name 'customizeHLTfor44576' is not defined

At:
  /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44576/38545/CMSSW_14_1_X_2024-04-02-1100/src/HLTrigger/Configuration/python/customizeHLTforCMSSW.py(279): customizeHLTforCMSSW
  /data/cmsbld/jenkins/workspace/ib-run-pr-addon/CMSSW_14_1_X_2024-04-02-1100/src/HLTrigger/Configuration/test/OnLine_HLT_Fake1.py(400): <module>

----- End Fatal Exception -------------------------------------------------

Expand to see more addon errors ...

Apr 02 '24 16:04 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44576/39773

This PR adds an extra 28KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): #33532, #43761, #43964
- File HLTrigger/Configuration/python/customizeHLTforCMSSW.py modified in PR(s): #43964

Apr 02 '24 22:04 cmsbuild

Pull request #44576 was updated. @mdhildreth, @jfernan2, @antoniovagnerini, @saumyaphor4252, @miquork, @syuvivida, @srimanob, @fabiocos, @Martin-Grunewald, @rappoccio, @rvenditti, @subirsarkar, @nothingface0, @civanch, @davidlange6, @mmusich, @Dr15Jones, @antoniovilela, @tjavaid, @epalencia, @perrotta, @consuegs, @francescobrivio, @mandrenguyen, @sunilUIET, @bsunanda, @cmsbuild, @AdrianoDee, @makortel, @aloeliger can you please check and sign again.

Apr 02 '24 22:04 cmsbuild

Could somebody kindly start the tests?

Apr 03 '24 10:04 emiglior

Could somebody kindly start the tests?

was it tested privately?

Apr 03 '24 10:04 mmusich

please test

Apr 03 '24 10:04 civanch

Could somebody kindly start the tests?

was it tested privately?

at least for the issues related to the typo in customizeHLTfor44576() but some test failed because of problems in accessing the input dataset

Apr 03 '24 10:04 emiglior