Add Validation and DQM to Phase2 HLT WF

[AdrianoDee]

This is to remind us that the wf running the Phase2 HLT (39434.75) is not running any validation or DQM for hlt objects.

This will be necessary in the mid-long term and would be useful in the short term for developments to spot eventual changes in the menu behavior (see e.g. https://github.com/cms-sw/cmssw/pull/39323).

[cmsbuild]

A new Issue was created by @AdrianoDee .

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[AdrianoDee]

assign upgrade

[AdrianoDee]

assign hlt

[cmsbuild]

New categories assigned: upgrade,hlt

@AdrianoDee,@missirol,@srimanob,@Martin-Grunewald you have been requested to review this Pull request/Issue and eventually sign? Thanks

[srimanob]

I think the way is to run HLT at the ususal step (DIGI+L1+HLT). Then the validation will run as usual. No need to push for extra development.

[srimanob]

I make this https://github.com/cms-sw/cmssw/pull/39450 for the first attempt to converge on running HLT together with DIGI step.

[slava77]

What is the current status? is this topic being tracked in another issue or outside of github?

@beaucero @SohamBhattacharya

[srimanob]

Hi @slava77 I have not finished it yet, sorry. I can make it after May 22.

[slava77]

Hi @slava77 I have not finished it yet, sorry. I can make it after May 22.

I wanted to check if there was some progress Thanks.

[srimanob]

Hi @slava77 I've made a new PR to enable one-by-one HLT validation module in https://github.com/cms-sw/cmssw/pull/41898

[srimanob]

I think this PR can be closed as we have Phase-2 DQM in the release.

[srimanob]

+Upgrade

[mmusich]

https://github.com/cms-sw/cmssw/pull/42783 introduces some modification to run the Tracking DQM monitoring in Phase-2

[SohamBhattacharya]

Hi, I have naive question about this. I see two different approaches to adding HLT validation and DQM.

1. The one started by @srimanob here
2. One developed by @Sam-Harper and @finnlabe here (in cmssw)

If they're both trying to do the same (?) thing (i.e. validate HLT paths), is there a way to consolidate the two approaches to make it simpler?

[mmusich]

If they're both trying to do the same (?) thing (i.e. validate HLT paths), is there a way to consolidate the two approaches to make it simpler?

as far as I can tell , that's not the same thing. The issue here is more targeted to the offline validation of the objects used by the HLT (electrons, muons, tracks, jets, etc. etc.). Basically everything in DQMOffline/Trigger that is currently available for Run 3 (but not for phase-2).

[SohamBhattacharya]

If they're both trying to do the same (?) thing (i.e. validate HLT paths), is there a way to consolidate the two approaches to make it simpler?

as far as I can tell , that's not the same thing. The issue here is more targeted to the offline validation of the objects used by the HLT (electrons, muons, tracks, jets, etc. etc.). Basically everything in DQMOffline/Trigger that is currently available for Run 3 (but not for phase-2).

It seems to me that the updates to HLTriggerOffline in https://github.com/cms-sw/cmssw/pull/41898 are to do with validating the TriggerResults (validation for HLT_AK4PFPuppiJet520), which is similar to what's being done in 2 above. (please correct me if I'm wrong) While we absolutely need the validation of Phase-2 HLT objects in DQMOffline/Trigger, imo (from the pov of HLT-upgrade) we should first have the Phase-2 HLT path and filter validation. That would show us changes in not only the HLT objects (thus also effectively doing the job of DQMOffline/Trigger as far as the impact on the Phase-2 menu is considered), but also in the upstream quantities that the HLT filters depend on, like the L1 objects. For example, the egamma issue (fixed in https://github.com/cms-sw/cmssw/pull/42791) was missed because the filters were not monitored and changes at L1 were not propagated down to the filters used by the Phase-2 menu.

Hence I was wondering if there's a way to have a validation like this as part of the Phase-2 HLT WF.

[tagging @rovere as I'm not sure if he's in the thread]

[mmusich]

we should first have the Phase-2 HLT path and filter validation.

@SohamBhattacharya I guess you are free to twist the initial aim of this ticket, but in the initial message I read:

is not running any validation or DQM for hlt objects.

I think your proposal should go into a different issue!

[SohamBhattacharya]

we should first have the Phase-2 HLT path and filter validation.

@SohamBhattacharya I guess you are free to twist the initial aim of this ticket, but in the initial message I read:

is not running any validation or DQM for hlt objects.

I think your proposal should go into a different issue!

No problem at all with having this in a different issue. Only initiating the discussion here as the question of validation was rightfully raised in https://github.com/cms-sw/cmssw/pull/42791 -- so wanted to make it clear what kind of validation hlt-upgrade would like to focus on first. :)

[srimanob]

Hi @SohamBhattacharya

My PR (https://github.com/cms-sw/cmssw/pull/41898) is for Validation step (i.e. compare with MC) while @mmusich PR (https://github.com/cms-sw/cmssw/pull/42783) takes care for DQM (i.e. no need of GEN, MC. Something that you run on Tier-0 during data taking, and also regular relvals, both Data and MC).

The way that both PR work is the same, we create a sequence based on Run-3 and enable only modules which work for Phase-2. See:

• Validation: https://github.com/cms-sw/cmssw/blob/master/HLTriggerOffline/Common/python/HLTValidation_cff.py#L30-L91 (I chose to not create new sequence, but use the original one and exclude what do not work.)

• DQM: https://github.com/cms-sw/cmssw/blob/d9adfeaf74d34ee3c350e42f343f74d98d766d9e/DQMOffline/Trigger/python/DQMOffline_Trigger_cff.py#L193-L215 (Marco creates the specific Phase-2 sequence, and include only working module. Then use replace to replace the sequence.

If I understand @mmusich correctly, the suggestion is to try to enable this full sequence [1]. While on my part, POG, PAG should try to enable their trigger paths and re-enable [2]

[1]

offlineHLTSource4HLTMonitorPD = cms.Sequence(
    dqmInfoHLTMon *
    lumiMonitorHLTsequence *          # lumi
    sistripMonitorHLTsequence *       # strip
    sipixelMonitorHLTsequence *       # pixel
    BTVHLTOfflineSource *             # BTV
    bTagHLTTrackMonitoringSequence *  # BTV relative track efficiencies
    trackingMonitorHLT *              # tracking
    BTagAndProbeHLT *                 # BTag and Probe
    trackingMonitorHLTDisplacedJet*   # EXO : DisplacedJet Tracking 
    egmTrackingMonitorHLT *           # EGM tracking
    hltToOfflineTrackValidatorSequence *  # Relative Online to Offline performace
    vertexingMonitorHLT *             # vertexing
    particleNetMonitoringHLT          # HIG: monitoring of HLT PNET taggers (incl. comparisons to Offline PNET)
)

[2]

hltassociation = cms.Sequence(
    hltMultiTrackValidation
    +hltMultiPVValidation
    +egammaSelectors
    +ExoticaValidationProdSeq
    +hltMultiTrackValidationGsfTracks
    +hltMultiTrackValidationMuonTracks
    )

hltvalidationWithMC = cms.Sequence(
    HLTMuonVal
    +HLTTauVal
    +egammaValidationSequence
    +heavyFlavorValidationSequence
    +HLTJetMETValSeq
    +HLTSusyExoValSeq
    +HiggsValidationSequence
    +ExoticaValidationSequence
    +b2gHLTriggerValidation
    +SMPValidationSequence
    +hltbtagValidationSequence #too noisy for now
    +hltHCALdigisAnalyzer+hltHCALRecoAnalyzer+hltHCALNoiseRates # HCAL
)

[mmusich]

for the record https://github.com/cms-sw/cmssw/pull/43094 introduced tracking and vertexing HLT validation (w.r.t tracking particles) for the phase 2 setup.

[rovere]

@mmusich is it possible that some harvesting step/sequence is missing from #43094? I was not able to find the usual eff/fake/dup plots.

[mmusich]

I was not able to find the usual eff/fake/dup plots.

which workflow are you running?

[mmusich]

is it possible that some harvesting step/sequence is missing from https://github.com/cms-sw/cmssw/pull/43094?

yah, seems I forgot these:

https://github.com/cms-sw/cmssw/blob/7122236ff08088b246964f0e29179a90788f9000/HLTriggerOffline/Common/python/HLTValidationHarvest_cff.py#L45-L46

let me fix.

[mmusich]

let me fix.

see https://github.com/cms-sw/cmssw/pull/43330

[mmusich]

For HGCal (DQM):

• https://github.com/cms-sw/cmssw/pull/44684 (14_0_X)
• https://github.com/cms-sw/cmssw/pull/44558 (master)

[cmsbuild]

cms-bot internal usage

[mmusich]

For Muon:

• https://github.com/cms-sw/cmssw/pull/46860

[mmusich]

For intermediate tracking collections:

• https://github.com/cms-sw/cmssw/pull/46823/

[mmusich]

For the IT and OT, clusters, rechits and trackingrechits:

• https://github.com/cms-sw/cmssw/pull/47352

[mtosi]

based on the recent release validation 15_0_0_pre2_Phase2_D110 and available relval sample on the DQM GUI, I see the following directories

• EGM : outdated and it has only plots related to the GsfTracks (TrackingMonitor and MTV) --> to be deleted ? @DebabrataBhowmik @saumyaphor4252 @tihsu99
• HLTEgammaValidation : super outdated, with empty plots --> to be deleted / updated ? @DebabrataBhowmik @saumyaphor4252 @tihsu99
• HCAL : I think it still makes sense, do you agree @JHiltbrand ?
• HGCalHitCalibration : some plots are filled w/ 0 entries @rovere ?
• HLTJETMET : some plots are properly filled, other in the validation subdir are completely empty --> @theofil @setesami I think this would need to be reviewed
• HeavyFlavor, Higgs, Top, SusyExo : all empty (of course), I think it is premature to have PAG related validation, @rovere @VourMa what do you think ?
• Muon : I think it needs to be reviewed, because it seems it assumes a list of paths which is not in the menu @BlancoFS
• TauRelVal : completely outdated --> @brallmond I'm afraid it needs to be reviewed
• Tracking : it seems fine @lguzzi
• Vertexing : it seems fine @lguzzi
• TrackerPhase2ITCluster and TrackerPhase2OTCluster : they seems fine @bartonaz

@gomber @kelmorab