databroker icon indicating copy to clipboard operation
databroker copied to clipboard

Published 20 hours ago • verified publisher icon bluesky

Out of memory HXN

Open gwbischof opened this issue 2 years ago • 1 comments

"There seems to be a memory leak when loading a batch of data with PyXRF using Data Broker V1 (no memory leak when using Data Broker V0). After loading 13 files, about 70 GB of memory is consumed. Loading a batch of runs in the range 92276-92335 using make_hdf in IPython it was observed that the amount of used memory increased from 42 GB to 113 GB (out of 126 GB) once run 92289 was reached. The memory is released once IPython is closed." - @dmgav

gwbischof avatar May 25 '22 15:05 gwbischof

This can be worked around by using databroker-intake and running, or using databroker v0

db._catalog._entries.cache_clear()

gwbischof avatar May 25 '22 15:05 gwbischof