cmssw CICADA-uGT emulator additions

PR description:

This PR adds the uGT half of the CICADA -> uGT emulator chain. It adds all the pieces needed to parse and compare the new CICADA condition added in uTM v 0.12.0. This is being opened in draft because no unpacker/packer combo is available yet, but will be added after I have had the chance to speak to experts about reading the CICADA record.

Depends on utm version 0.12.0

@elfontan You should likely take a look at my work on this, this is more working with uGT than I am used to.

@cms-sw/hlt-l2 Just in case you aren't automatically pinged on this, FYI. There is no unpacker/packer, but I will keep this in draft while I develop them. I will be speak to experts about it ASAP.

EDIT 06/03/24: Draft unpacker (not non-trivially tested) added.

PR validation:

All code compiles and has had code formatting applied.

In addition, I have tested the emulator on a new proto-type menu including a CICADA seed: https://cernbox.cern.ch/files/link/public/LeBbgndwFDjWoI5?tiles-size=1&items-per-page=100&view-mode=resource-table

The emulator runs all the way through. The following are debug statements (now removed) from various components of the global board, CICADA conditions, and trigger menu parser showing that the emulator receives the score, compares to the menu, and in this case, decides it is below threshold and doesn't pass the menu cut:

Begin processing the 2nd record. Run 368566, Event 2615907242, LumiSection 1195 on stream 0 at 27-Feb-2024 08:34:54.517 CST
GlobalBoard::receiveCaloObjectData receive cicada
Valid CICADA
2.73438
GlobalBoard::runGTL CondCICADA
CICADACondition::evaluationCondition begin
cicada score is: 2.73438
min threshold is: 42
passCondition? 0

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

This PR is not a backport however, is likely to be needed for data-taking so a backport is probable to CMSSW_14_0.

Feb 27 '24 15:02 aloeliger

cms-bot internal usage

Feb 27 '24 15:02 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44222/39227

This PR adds an extra 68KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File L1Trigger/L1TGlobal/plugins/L1TGlobalProducer.cc modified in PR(s): #44054
- File L1Trigger/L1TGlobal/plugins/TriggerMenuParser.cc modified in PR(s): #44054
- File L1Trigger/L1TGlobal/plugins/TriggerMenuParser.h modified in PR(s): #44054
- File L1Trigger/L1TGlobal/python/simGtStage2Digis_cfi.py modified in PR(s): #44054
- File L1Trigger/L1TGlobal/src/GlobalBoard.cc modified in PR(s): #44054

Feb 27 '24 15:02 cmsbuild

I'm marking this pull request ready after adding the unpacker commit. However, I should stress that the unpacker has not been tested in realistic conditions with actual CICADA input data present, and therefore has not been tested to be working. It does compile, and it does seem to properly fail-safe to a 0.0 score in the case of nothing to unpack/nothing unpacked.

Mar 06 '24 15:03 aloeliger

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44222/39361

This PR adds an extra 80KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File EventFilter/L1TRawToDigi/plugins/implementations_stage2/GTSetup.cc modified in PR(s): #44019
- File L1Trigger/L1TGlobal/plugins/L1TGlobalProducer.cc modified in PR(s): #44054
- File L1Trigger/L1TGlobal/plugins/TriggerMenuParser.cc modified in PR(s): #44054
- File L1Trigger/L1TGlobal/plugins/TriggerMenuParser.h modified in PR(s): #44054
- File L1Trigger/L1TGlobal/python/simGtStage2Digis_cfi.py modified in PR(s): #44054
- File L1Trigger/L1TGlobal/src/GlobalBoard.cc modified in PR(s): #44054

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44222/39361/code-format.patch e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44222/39361/code-format.patch | patch -p1 You can also run scram build code-format to apply code format directly

Mar 06 '24 15:03 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44222/39362

This PR adds an extra 80KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File EventFilter/L1TRawToDigi/plugins/implementations_stage2/GTSetup.cc modified in PR(s): #44019
- File L1Trigger/L1TGlobal/plugins/L1TGlobalProducer.cc modified in PR(s): #44054
- File L1Trigger/L1TGlobal/plugins/TriggerMenuParser.cc modified in PR(s): #44054
- File L1Trigger/L1TGlobal/plugins/TriggerMenuParser.h modified in PR(s): #44054
- File L1Trigger/L1TGlobal/python/simGtStage2Digis_cfi.py modified in PR(s): #44054
- File L1Trigger/L1TGlobal/src/GlobalBoard.cc modified in PR(s): #44054

Mar 06 '24 15:03 cmsbuild

A new Pull Request was created by @aloeliger for master.

It involves the following packages:

EventFilter/L1TRawToDigi (l1)
L1Trigger/L1TGlobal (l1)

@aloeliger, @epalencia, @cmsbuild can you please review it and eventually sign? Thanks. @Martin-Grunewald, @thomreis, @missirol, @dinyar, @eyigitba this is something you requested to watch as well. @sextonkennedy, @antoniovilela, @rappoccio you are the release manager for this.

cms-bot commands are listed here

Mar 06 '24 15:03 cmsbuild

assign hlt

I think HLT wants to sign off on changes to the uGT emulator?

Mar 06 '24 16:03 aloeliger

New categories assigned: hlt

@Martin-Grunewald,@mmusich you have been requested to review this Pull request/Issue and eventually sign? Thanks

Mar 06 '24 16:03 cmsbuild

It is not clear to me how/where the unpacker is called. Within existing (unpacking) code or a new module, which needs to be added to some python unpacking sequence? What about the packer?

Mar 06 '24 16:03 Martin-Grunewald

It is not clear to me how/where the unpacker is called. Within existing (unpacking) code or a new module, which needs to be added to some python unpacking sequence? What about the packer?

@Martin-Grunewald It's defined as part of the GT unpacking setup here: https://github.com/cms-sw/cmssw/blob/c0b5b266322de4b536b3e8bd7d41749da8b74764/EventFilter/L1TRawToDigi/plugins/implementations_stage2/GTSetup.cc#L131

I need to wait until I have test crate records before I can test this non-trivially however. (Seems like it wont unpack empty/unused records?).

Mar 06 '24 16:03 aloeliger

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44222/39370

This PR adds an extra 24KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File EventFilter/L1TRawToDigi/plugins/implementations_stage2/GTSetup.cc modified in PR(s): #44019
- File L1Trigger/L1TGlobal/plugins/L1TGlobalProducer.cc modified in PR(s): #44054
- File L1Trigger/L1TGlobal/plugins/TriggerMenuParser.cc modified in PR(s): #44054
- File L1Trigger/L1TGlobal/plugins/TriggerMenuParser.h modified in PR(s): #44054
- File L1Trigger/L1TGlobal/python/simGtStage2Digis_cfi.py modified in PR(s): #44054
- File L1Trigger/L1TGlobal/src/GlobalBoard.cc modified in PR(s): #44054

Mar 07 '24 10:03 cmsbuild

Pull request #44222 was updated. @epalencia, @Martin-Grunewald, @mmusich, @cmsbuild, @aloeliger can you please check and sign again.

Mar 07 '24 10:03 cmsbuild

@cmsbuild, please test

Mar 26 '24 21:03 mmusich

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-547e2e/38435/summary.html COMMIT: d7cb22a9e5325ddd4db1db0cf3ef8b0a89240af9 CMSSW: CMSSW_14_1_X_2024-03-26-1100/el8_amd64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44222/38435/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 59 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 51 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3297857
DQMHistoTests: Total failures: 9
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3297828
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: no differences found

Mar 27 '24 00:03 cmsbuild

code-checks

Mar 28 '24 16:03 Martin-Grunewald

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44222/39722

ERROR: Unable to merge PR.

See log https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44222/39722/cms-checkout-topic.log

Mar 28 '24 16:03 cmsbuild

CICADA is likely coming online for testing in the near future. I have rebased this PR to fix the conflicts.

uGT has also provided a more accurate input record example for what CICADA needs to unpack. I will be making the necessary changes ASAP and testing as soon as I am able.

Apr 12 '24 12:04 aloeliger

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44222/39931

This PR adds an extra 64KB to repository

Apr 12 '24 13:04 cmsbuild

Pull request #44222 was updated. @cmsbuild, @Martin-Grunewald, @epalencia, @aloeliger, @mmusich can you please check and sign again.

Apr 12 '24 13:04 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44222/39950

This PR adds an extra 88KB to repository

Apr 14 '24 14:04 cmsbuild

Pull request #44222 was updated. @aloeliger, @mmusich, @cmsbuild, @epalencia, @Martin-Grunewald can you please check and sign again.

Apr 14 '24 14:04 cmsbuild

Okay. After discussion with uGT experts, and a quick reference on what the input record for the calosummary/CICADA card is expected to look like (included below), I have updated the unpacker and CICADA emulator to take advantage of all BXs by changing emulator outputs and unpacker outputs to BX vectors (the uGT changes have been updated to use this BX vector & grab the only guaranteed available BX, the central one). The emulator will not, and really cannot create information for BXs outside the central one, but the unpacked CICADA BXs may be useful for other studies around CICADA (pre-firing comes to mind?) At this point, this unpacker has still not been thoroughly tested (I am searching for a way to verify it ASAP), but it should now theoretically be complete.

Reference on CICADA input records at uGT:

161e0200xxxxxxxx => "16" indicates begin of CICADA input data
2000000010000000 => AD I LSB (cicada frame 1, bx-2) & AD I MSB (cicada frame 0, bx-2)
4000000030000000 => AD D LSB (cicada frame 3, bx-2) & AD D MSB (cicada frame 2, bx-2)
0000000000000000 => free (cicada frame 5, bx-2) & free (cicada frame 4, bx-2)
6000000050000000 => AD I LSB (cicada frame 1, bx-1) & AD I MSB (cicada frame 0, bx-1)
8000000070000000 => AD D LSB (cicada frame 3, bx-1) & AD D MSB (cicada frame 2, bx-1)
0000000000000000 => free (cicada frame 5, bx-1) & free (cicada frame 4, bx-1)
a000000090000000 => AD I LSB (cicada frame 1, bx) & AD I MSB (cicada frame 0, bx)
c0000000b0000000 => AD D LSB (cicada frame 3, bx) & AD D MSB (cicada frame 2, bx)
0000000000000000 => free (cicada frame 5, bx) & free (cicada frame 4, bx)
e0000000d0000000 => AD I LSB (cicada frame 1, bx+1) & AD I MSB (cicada frame 0, bx+1)
f0000000f0000000 => AD D LSB (cicada frame 3, bx+1) & AD D MSB (cicada frame 2, bx+1)
0000000000000000 => free (cicada frame 5, bx+1) & free (cicada frame 4, bx+1)
8000000070000000 => AD I LSB (cicada frame 1, bx+2) & AD I MSB (cicada frame 0, bx+2)
1000000020000000 => AD D LSB (cicada frame 3, bx+2) & AD D MSB (cicada frame 2, bx+2)
0000000000000000 => free (cicada frame 5, bx+2) & free (cicada frame 4, bx+2)
xxxxxxxx181e0100 => "18" indicates begin of EXTERNAL CONDITION input data

"AD I": means Anomaly Detection Integer part (8 bits).
"AD D": means Anomaly Detection Decimal part (8 bits).
"MSB": means the 4 most significant bits of the 8 bits word (bits 31..28 in record), 
"LSB": means the 4 least significant bits of the 8 bits word (bits 63..60 in record).
(Bits 59..32 and 27..0 of frames 0..3 and all bits of frames 4..5 are always zero).

Apr 14 '24 18:04 aloeliger

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44222/39951

This PR adds an extra 60KB to repository

Apr 14 '24 18:04 cmsbuild

Pull request #44222 was updated. @cmsbuild, @mmusich, @Martin-Grunewald, @epalencia, @aloeliger can you please check and sign again.

Apr 14 '24 18:04 cmsbuild

please test

Apr 14 '24 18:04 aloeliger

-1

Failed Tests: RelVals RelVals-INPUT AddOn Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-547e2e/38837/summary.html COMMIT: 9c81c2f1d17eecd8ecf16e1764b5c12975222aa2 CMSSW: CMSSW_14_1_X_2024-04-14-0000/el8_amd64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44222/38837/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

139.001A fatal system signal has occurred: abort signal
140.023A fatal system signal has occurred: abort signal
136.731A fatal system signal has occurred: abort signal

Expand to see more relval errors ...

RelVals-INPUT

136.721136.721_RunHLTPhy2016B/step2_RunHLTPhy2016B.log
136.722136.722_RunDoubleEG2016B/step2_RunDoubleEG2016B.log
136.723136.723_RunDoubleMuon2016B/step2_RunDoubleMuon2016B.log

Expand to see more relval errors ...

AddOn Tests

hlt_mc_Fake2

A fatal system signal has occurred: abort signal

hlt_mc_Fake2

A fatal system signal has occurred: abort signal

hlt_mc_GRun

A fatal system signal has occurred: abort signal

Expand to see more addon errors ...

Apr 14 '24 21:04 cmsbuild

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44222/39952

This PR adds an extra 120KB to repository

Apr 14 '24 22:04 cmsbuild

Pull request #44222 was updated. @Martin-Grunewald, @epalencia, @aloeliger, @mmusich, @cmsbuild can you please check and sign again.

Apr 14 '24 22:04 cmsbuild

please test

Apr 14 '24 22:04 aloeliger

-1

Failed Tests: RelVals-INPUT Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-547e2e/38839/summary.html COMMIT: 2464dc6fe24eba32171954bac6a12837bb2a4706 CMSSW: CMSSW_14_1_X_2024-04-14-0000/el8_amd64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44222/38839/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

250202.17250202.17_TTbar13TeVPUpmx2017/step2_TTbar13TeVPUpmx2017.log

Comparison Summary

Summary:

You potentially removed 16 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 49 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3317870
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3317844
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: no differences found

Apr 15 '24 03:04 cmsbuild