[mg26x] cmsgrid_final.lhe can exist for madgraph gridpacks even if some steps fail at runtime

[bendavid]

Observed in mg26x branch with systematics weights step, but possibly affecting other additional steps as well (madspin, model reweighting, etc).

This could lead to silent failures of part of the chain which leaves an inhomogeneous set of events in the dataset.

Gridpack script must not produce cmsgrid_final.lhe in case ANY configured processing step fails, in order to guarantee that the CMSSW job fails in this case.

[kdlong]

Yes, this has already been observed and reported for some 2016 samples that used ME reweighting. How would you suggest proceeding? Is it enough to exit the job with a non-zero exit status if any step fails or does cmsgrid_final.lhe need to be deleted?

[bendavid]

The best is to make sure there is no file named cmsgrid_final.lhe until the very end. (and then also check the exit codes of each step and quit with a non-zero exit code if any step fails)