Optimize ecal biasing factor for 8 GeV

[tvami]

I am updating ldmx-sw, here are the details.

What are the issues that this addresses?

Resolves https://github.com/LDMX-Software/ldmx-sw/issues/1535

Check List

• [x] I successfully compiled ldmx-sw with my developments
• [x] I ran my developments and the following shows that they are successful.

See https://github.com/LDMX-Software/ldmx-sw/issues/1535#issuecomment-2625960556

[tvami]

I'm gonna use this PR to test https://github.com/LDMX-Software/ldmx-sw/pull/1569

[tvami]

/run-validation

[github-actions[bot]]

The validation workflow is running here: https://github.com/LDMX-Software/ldmx-sw/actions/runs/13140056694.

[tvami]

I think this works now!

[tvami]

I'll test if it also works in a draft mode

[tvami]

/run-validation

[github-actions[bot]]

The validation workflow is running here: https://github.com/LDMX-Software/ldmx-sw/actions/runs/13140159042.

[tvami]

  Switched to a new branch 'iss1535-biasing-factor-optimization'
  branch 'iss1535-biasing-factor-optimization' set up to track 'origin/iss1535-biasing-factor-optimization'.

Yaay it does! Sorry everybody if you got more than than a dozen msg because of this!

[tvami]

I'm opening this for review, as we agreed today that this will not affect anything in the DR and is safe to merge

[tvami]

Here are the ecal shower plots: compare_230_550_bias_factor_ecal_shower.pdf

Looks pretty good

[tvami]

I meant this to be a sanity check... No warnings in 10k events! But the ECAL and PN (and HCAL & TS even) distributions change a lot ecal_pn_recon_validation_plots.pdf

[bryngemark]

@tvami LOL yes that z distribution is telling. All interactions in the first possible instance!

[tvami]

Exactly :D Would be good to have an earlier measure that helps to find the optimal number before having to look at the Z distribution

[tvami]

this is the Z vertex with a factor of a 1000:

[tvami]

Here is the rest of the KS-failed distributions: change_bias_factor_1000-ecal_pn_recon_validation_plots.pdf

[bryngemark]

With a factor 1000 it looks like we're getting about 10% more events in the first two bins than we used to (btw is the reference with 8 GeV/550, or 4 GeV/450?). I think this could still mean that we're getting a large enough xsc to trigger a warning but it is suppressed.

Did you get around to running with old_ecal? That should factor the low-Z materials out of the problem. If we can run without these materials, maybe finding the maximum warning-free biasing factor would be a good procedure to optimize without having to look at z.

[tvami]

(btw is the reference with 8 GeV/550, or 4 GeV/450?).

with ref to 8 GeV/550

Did you get around to running with old_ecal?

Not yet, but sounds good

[bryngemark]

I suggest we discuss this at the sw dev meeting today and type up what we learn from that and the email thread with natalia in this issue afterwards.

[tvami]

Let's test with the new gold

[tvami]

[tvami]

So a little bit more interactions in the non-W part of the ecal, but it seems the with a factor of a 1000 it's not too different.

Reminder about the Z dist (now with the clean gold, and normalized plots with error bars):

[tvami]

The rest of the plots that fail the KS (reminder the red is with 550 and the blue is with 1000) bias-opt-ecal_pn_recon_validation_plots.pdf

[tvami]

Results in https://github.com/LDMX-Software/ldmx-sw/actions/runs/13469761175

The z with using biasing factor of 5000

Material

Volume

So more is happening now in the non-W based elements, I guess this was expected

[tvami]

I'll check now how things look like for biasing factor of 1

[tvami]

I'll check now how things look like for biasing factor of 1

Ohh so with biasing factor of 1 the ecalPN takes longer than 6 hours to run, so github kills it...

[bryngemark]

@tvami did this move along lately? were you able to run some tests outside of the CI action?

[tvami]

@tvami did this move along lately? were you able to run some tests outside of the CI action?

Not really, after I'm done with the DR stuff I'll come back to this

[tvami]

PN_pn_gamma_int_z.pdf PN_pn_interaction_material.pdf PN_pn_vertex_volume.pdf

I conclude that we are good with the 550 as is.

Btw, the 1100 still doesn't trigger the warning, but it clearly biases the z distribution

[tvami]

@tomeichlersmith I agree with that sentiment, but "old_ecal" is in the software: https://github.com/LDMX-Software/ldmx-sw/blob/8f4cc8a0416855eb7647dd5ad79b615649a1972c/SimCore/src/SimCore/DetectorConstruction.cxx#L67-L68 etc it just didnt have the python config to use it, so I thought the consistent behavior is to add it. But I'm fine to remove it if you feel strongly about this.

[tomeichlersmith]

I would be happy to remove that old_ecal now. It was in reference to which volumes were being biased and I made the regretful decision to do the physics testing on trunk instead of on a separate branch (like you've done here). I don't think anyone should use old_ecal and thus, removing it is preferable.