cmssw icon indicating copy to clipboard operation
cmssw copied to clipboard

Failure in NanoAOD v11 on MC samples.

Open bbilin opened this issue 1 year ago • 14 comments

We observe failures in nanoAOD production with the mismatch of genweight size. (CepGen + PY6 samples) (using 12_6_0_patch1)

cmsunified

The message as below:

Fatal Exception (Exit code: 8002)
An exception of category 'StdException' occurred while
[0] Processing Event run: 1 lumi: 809 event: 1 stream: 0
[1] Running path 'NANOEDMAODSIMoutput_step'
[2] Prefetching for module PoolOutputModule/'NANOEDMAODSIMoutput'
[3] Calling method for module GenWeightsTableProducer/'genWeightsTable'
Exception Message:
A std::exception was thrown.
vector::_M_range_check: __n (which is 27) >= this->size() (which is 24)

Fatal Exception (Exit code: 8002)
An exception of category 'StdException' occurred while
[0] Processing Event run: 1 lumi: 1 event: 11 stream: 3
[1] Running path 'NANOEDMAODSIMoutput_step'
[2] Prefetching for module PoolOutputModule/'NANOEDMAODSIMoutput'
[3] Calling method for module GenWeightsTableProducer/'genWeightsTable'
Exception Message:
A std::exception was thrown.
vector::_M_range_check: __n (which is 27) >= this->size() (which is 24)

The cmsDriver command as follows:

cmsDriver.py step1 --filein "dbs:/CEPDijets-GluGlu_M-250_survfact0_13p6TeV_superchic/Run3Summer22MiniAODv3-124X_mcRun3_2022_realistic_v12-v2/MINIAODSIM" --fileout file:PPS-Run3Summer22NanoAODv11-00005.root --mc --eventcontent NANOEDMAODSIM --datatier NANOAODSIM --conditions 126X_mcRun3_2022_realistic_v2 --step NANO --nThreads 4 --scenario pp --era Run3,run3_nanoAOD_124

And can be run on the following miniAOD set: /CEPDijets-GluGlu_M-250_survfact0_13p6TeV_superchic/Run3Summer22MiniAODv3-124X_mcRun3_2022_realistic_v12-v2/MINIAODSIM

PdmV

@kskovpen @sunilUIET @swertz @vlimant

bbilin avatar Mar 17 '23 13:03 bbilin

A new Issue was created by @bbilin Bugra Bilin.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

cmsbuild avatar Mar 17 '23 13:03 cmsbuild

assign xpog

makortel avatar Mar 17 '23 13:03 makortel

New categories assigned: xpog

@swertz,@vlimant you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild avatar Mar 17 '23 13:03 cmsbuild

assign generators in case they can help out

vlimant avatar Mar 21 '23 15:03 vlimant

New categories assigned: generators

@mkirsano,@menglu21,@alberto-sanchez,@SiewYan,@GurpreetSinghChahal,@Saptaparna you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild avatar Mar 21 '23 15:03 cmsbuild

This seems to happen because of the Pythia PS weights. The NanoAOD weights producer recognizes the weights as "alternative" ones here: https://github.com/cms-sw/cmssw/blob/master/PhysicsTools/NanoAOD/plugins/GenWeightsTableProducer.cc#L999 It then tries to read weight indices {27, 5, 26, 4}, see https://github.com/cms-sw/cmssw/blob/master/PhysicsTools/NanoAOD/plugins/GenWeightsTableProducer.cc#L186 But for some reason weight index 27 apparently doesn't exist for that sample, which only has 24 weights.

So, either fix the pythia fragment, or fix the weights producer for this special case.

Note that we really hoped to not have to touch that code anymore, as we've been eagerly expecting this to be integrated: https://github.com/cms-sw/cmssw/pull/32167

swertz avatar Mar 21 '23 15:03 swertz

Do we have any workflow in runTheMatrix testing this generator configuration in conjunction of nanoAOD?

makortel avatar Mar 21 '23 20:03 makortel

Do we have any workflow in runTheMatrix testing this generator configuration in conjunction of nanoAOD?

AFAIK not, anyway there are way too many different generator configurations in production to test them all with the matrix... This sort of things could be avoided by producing a few NanoGEN events when preparing a MC request and checking the weights.

swertz avatar Mar 21 '23 20:03 swertz

Do we have any workflow in runTheMatrix testing this generator configuration in conjunction of nanoAOD?

AFAIK not, anyway there are way too many different generator configurations in production to test them all with the matrix... This sort of things could be avoided by producing a few NanoGEN events when preparing a MC request and checking the weights.

I realize I was imprecise. What I was trying to mean with "generator configuration" was the "CepGen + PY6". I think it would be very beneficial to have at least one workflow for every generator we use in production.

makortel avatar Mar 21 '23 20:03 makortel

Let me add @SanghyunKo to this discussion.

Saptaparna avatar Mar 21 '23 20:03 Saptaparna

Hi, do we have any further instructions/wayout to fix this issue? We are facing the same issue in Summer23 with NanoV12 as well. https://cms-unified.web.cern.ch/cms-unified/showlog/?search=task_PPD-Run3Summer23pLHEGS-00001

sunilUIET avatar Oct 13 '23 10:10 sunilUIET

I think this would be in @cms-sw/generators-l2 hand.

vlimant avatar Oct 17 '23 07:10 vlimant

we will get to this soon, similarly to #43784

vlimant avatar Mar 15 '24 08:03 vlimant

cms-bot internal usage

cmsbuild avatar Mar 15 '24 08:03 cmsbuild

@sunilUIET : the request you pointed out was force-completed, but does not seem to be marked "done" ... has there been other cases of such failures ?

vlimant avatar Jul 29 '24 10:07 vlimant