cmssw icon indicating copy to clipboard operation
cmssw copied to clipboard

[Run3 PromptReco] GsfElectronProducer and GEDPhotonProducer duplicate TF sessions across streams

Open makortel opened this issue 1 year ago • 7 comments

I noticed in https://github.com/cms-sw/cmssw/issues/46040#issuecomment-2420384665 that the GsfElectronProducer and GEDPhotonProducer duplicate the Tensorflow sessions across streams, specifically here https://github.com/cms-sw/cmssw/blob/55fd97727507dd74a4020c9264f5f66118409151/RecoEgamma/EgammaPhotonProducers/src/GEDPhotonProducer.cc#L481 https://github.com/cms-sw/cmssw/blob/55fd97727507dd74a4020c9264f5f66118409151/RecoEgamma/EgammaElectronProducers/plugins/GsfElectronProducer.cc#L577

The tensorflow::Session has been thread safe for a while, so the Sessions could be moved to be part of the GlobalCaches. While the total cost of these is modest (0.95 MB / stream), it would still be something, and by quick look I think the code would become simpler.

makortel avatar Oct 23 '24 16:10 makortel

assign RecoEgamma/EgammaPhotonProducers, RecoEgamma/EgammaElectronProducers

makortel avatar Oct 23 '24 16:10 makortel

assign ml

makortel avatar Oct 23 '24 16:10 makortel

New categories assigned: reconstruction

@jfernan2,@mandrenguyen you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild avatar Oct 23 '24 16:10 cmsbuild

cms-bot internal usage

cmsbuild avatar Oct 23 '24 16:10 cmsbuild

A new Issue was created by @makortel.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

cmsbuild avatar Oct 23 '24 16:10 cmsbuild

type performance-improvements

makortel avatar Oct 23 '24 16:10 makortel

New categories assigned: ml

@valsdav,@y19y19 you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild avatar Oct 23 '24 16:10 cmsbuild

Hi @makortel, I wrote this code a while ago.. as the session handling is more indirect in this code, probably we forgot to migrate it when sessions became thread-safe. I will take care of this.

valsdav avatar Oct 23 '24 21:10 valsdav

Thanks!

makortel avatar Oct 23 '24 22:10 makortel

This was addressed in https://github.com/cms-sw/cmssw/pull/46655

makortel avatar Dec 09 '24 14:12 makortel