cms-sw
Non reproducibility in wf 136.793
It looks like that there is some non-reproducibility in (at least) workflow 136.793
It is witnessed by the following lines in the log, which appear quite often in different number and with different values in the bot tests when compared between baseline and baseline+PR
curv error not pos-def
[ 1.78225e+18 1.2382e+19 -4.3207e+19-1.49326e+20 1.63409e+20
1.2382e+19-7.05161e+30-3.66399e+30-1.25526e+31-9.31912e+31
-4.3207e+19-3.66399e+30 1.10223e+32 3.81575e+32-4.78835e+31
-1.49326e+20-1.25526e+31 3.81575e+32 1.32094e+33-1.64027e+32
1.63409e+20-9.31912e+31-4.78835e+31-1.64027e+32-1.23157e+33 ]
pos/mom/mf (-91.8426,13.6769,-182.754) (-7.26726,11.4853,-50.0727) (0.0342356,-0.00509825,3.75128)
A couple of examples in https://github.com/cms-sw/cmssw/pull/43025#issuecomment-1812578372 and https://github.com/cms-sw/cmssw/pull/43283 (e.g.)
New categories assigned: reconstruction
@jfernan2,@mandrenguyen you have been requested to review this Pull request/Issue and eventually sign? Thanks
A new Issue was created by @perrotta Andrea Perrotta.
@rappoccio, @antoniovilela, @sextonkennedy, @Dr15Jones, @smuzaffar, @makortel can you please review it and eventually sign/assign? Thanks.
cms-bot commands are listed here
From the log of tests of https://github.com/cms-sw/cmssw/pull/43025#issuecomment-1812578372 it seems the printout mentioned in the issue description comes from ConversionTrackCandidateProducer:conversionTrackCandidates
%MSG-w BasicTrajectoryState: ConversionTrackCandidateProducer:conversionTrackCandidates 15-Nov-2023 10:32:41 CET Run: 301998 Event: 9374081
curv error not pos-def
[ 1.78225e+18 1.2382e+19 -4.3207e+19-1.49326e+20 1.63409e+20
1.2382e+19-7.05161e+30-3.66399e+30-1.25526e+31-9.31912e+31
-4.3207e+19-3.66399e+30 1.10223e+32 3.81575e+32-4.78835e+31
-1.49326e+20-1.25526e+31 3.81575e+32 1.32094e+33-1.64027e+32
1.63409e+20-9.31912e+31-4.78835e+31-1.64027e+32-1.23157e+33 ]
pos/mom/mf (-91.8426,13.6769,-182.754) (-7.26726,11.4853,-50.0727) (0.0342356,-0.00509825,3.75128)
%MSG
With some little investigation made while in train, it seems to me that:
- this issue was not present in CMSSW_13_3_X_2023-11-02-1100 [1]
- this issue was present in CMSSW_13_3_X_2023-11-03-1100
- Observation: https://github.com/cms-sw/cmssw/pull/43145 was merged in CMSSW_13_3_X_2023-11-03-1100. Not necessarily the culprit, but perhaps worth investigating further...
[1] at least I'm not seeing a large number of lines added/removed from the log in a PR that was tested on that IB
I'm running valgrind memcheck on 136.793 and 11834.21 (I was already doing 11834.21 for #42700, I've updated to CMSSW_14_0_X_2023-11-16-1100 to cover this too)
Log differences in https://github.com/cms-sw/cmssw/pull/43310#issuecomment-1815799272 show similar non-reproducibility in workflow 136.874 as well. The full message in question (in PR's tests) is
%MSG-w BasicTrajectoryState: ConversionTrackCandidateProducer:conversionTrackCandidates 17-Nov-2023 01:41:50 CET Run: 319450 Event: 105991987
BasicTrajectoryState: attempt to access errors when none available accessing local error..
freestate pointer: parameters
x = 35.2409 -52.2478 81.7855
p = 1.74807e+08 1.21867e+08 -9.77032e+08
no error defined.
local error valid/values :0
[ -9.9999e+14 -2.3304e-05 0.00125994 0.02167531.15042e-310
-2.3304e-05 7.03905e-06-0.0005213461.15036e-3104.94066e-324
0.00125994-0.000521346 0.001443776.95253e-3101.15048e-310
0.02167531.15036e-3106.95253e-3101.15042e-3101.15042e-310
1.15042e-3104.94066e-3241.15048e-3101.15042e-3101.15042e-310 ]
%MSG
