mesmerize-core
mesmerize-core copied to clipboard
Cache size calculation does not take size of CNMF object into account
I just rewrote this function (to recurse for sequences and estimates fields). This changed the results in test_cache, and I realized the old implementation had a bug:
def _get_cache_size_bytes(self):
"""Returns in bytes"""
cache_size = 0
for i in range(len(self.cache.index)):
if isinstance(self.cache.iloc[i, 4], np.ndarray):
cache_size += self.cache.iloc[i, 4].data.nbytes
elif isinstance(self.cache.iloc[i, 4], (tuple, list)):
for lists in self.cache.iloc[i, 4]:
for array in lists:
cache_size += array.data.nbytes
elif isinstance(self.cache.iloc[i, 4], CNMF):
sizes = list()
# ^^^^^^^^^^^^^^ this list is never used
for attr in self.cache.iloc[i, 4].estimates.__dict__.values():
if isinstance(attr, np.ndarray):
sizes.append(attr.data.nbytes)
else:
sizes.append(getsizeof(attr))
else:
cache_size += sys.getsizeof(self.cache.iloc[i, 4])
return cache_size
So it's easy to fix but just noting that the cache size limit has not been working correctly for a while (and it seems like the test was written to preserve existing behavior rather than the correct behavior).