TrackingTruthAccumulator: Something has gone wrong with the indexing

[vlimant]

I was looking at the recent premix production and there are jobs in "cool off" (meaning they will get resubmitted)

Fatal Exception (Exit Code: 8002)
An exception of category 'StdException' occurred while
   [0] Processing  Event run: 1 lumi: 1890 event: 1889249 stream: 1
   [1] Running path 'PREMIXoutput_step'
   [2] Prefetching for module PoolOutputModule/'PREMIXoutput'
   [3] Calling method for module MixingModule/'mix'
Exception Message:
A std::exception was thrown.
TrackingTruthAccumulator: Something has gone wrong with the indexing. Parent track index is 476.

one log is available under https://eoscmsweb.cern.ch/eos/cms/store/logs/prod/recent/PRODUCTION/pdmvserv_task_PPD-RunIIISummer24PrePremix-00004__v1_T_241021_203252_7210/PPD-RunIIISummer24PrePremix-00004_0/cmsgwms-submit12.fnal.gov-10291964-0-log.tar.gz

chances are that the resubmitted job will hit a different seed and pass. But in fine, this might be truncating the PU profile that is being effectively produced (due to selection bias).

https://cmsweb.cern.ch/reqmgr2/fetch?rid=pdmvserv_task_PPD-RunIIISummer24PrePremix-00004__v1_T_241021_203252_7210

    "CMSSWVersion": "CMSSW_14_0_18",
    "GlobalTag": "140X_mcRun3_2024_realistic_v26",

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @vlimant.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[davidlange6]

any notable differences in the GT since v20 used to make the gen-sim?

[makortel]

assign simulation

[cmsbuild]

New categories assigned: simulation

@civanch,@kpedro88,@mdhildreth you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

The exception is thrown from https://github.com/cms-sw/cmssw/blob/8340b52ad5b3272e98a300259ff0c67959520800/SimGeneral/TrackingAnalysis/plugins/TrackingTruthAccumulator.cc#L888-L911 (btw, the exception type should be changed to cms::Exception)

[vlimant]

@davidlange6 : https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/140X_mcRun3_2024_realistic_v20/140X_mcRun3_2024_realistic_v26 nothing sticks out that would/could explain this (I would not know what to look for though ...)

[davidlange6]

for sure the beam spot changing is interesting...(also from the point of view of usefulness of the resulting premixed sample)

[vlimant]

it's the reco beamspot, no the gen/sim one