inference module looks like it leaves HDF5 file handle open during computation

[danielsf]

Looking at this block of code

https://github.com/AllenInstitute/deepinterpolation/blob/master/deepinterpolation/inferrence_collection.py#L201-L247

It looks like, when running an inference job, the module opens the output HDF5 file and leaves that file open during the entire computation. Given that denoising jobs on videos can take in excess of 24 hours, I'm not sure this is a wise strategy. We (Allen Instittue informatics) have had experience with modules behaving poorly when file handles are left open for long periods of time.

Obviously, rapidly and repeatedly opening and closing the same file is not a great model, either.

I wonder if it is possible to gradually collect the data as it is being calculated and periodically re-open the HDF5 file to flush results from memory to disk.

I do not know of any problems that are being created by the current design. Seeing this code just gave me pause and I wanted to open this issue in case someone has time to look into it. I apologize if I am being a busybody.