DSAlign icon indicating copy to clipboard operation
DSAlign copied to clipboard

Do not load model if it's already loaded

Open gibiansky opened this issue 4 years ago • 1 comments

Not only is this slow, this will break on GPUs because TensorFlow does not release its allocated memory. So if you try to run a catalog with multiple files on a GPU, the first file will succeed, and the second file will give you an OOM error.

gibiansky avatar Apr 10 '20 21:04 gibiansky

I'm not totally sure how this interacts with multiple processes though, or if this creates some sort of race condition. On first glance I think it's fine but I haven't thought about this deeply.

I made this PR because I hit this issue and the change I put in this PR fixed it so figured I'd put this up in case someone else hits this.

gibiansky avatar Apr 10 '20 21:04 gibiansky