ArcticDB icon indicating copy to clipboard operation
ArcticDB copied to clipboard

Overlapping index check during update logic is incorrect

Open joe-iddon opened this issue 1 year ago • 2 comments

Describe the bug

When investigating an error I found that it would not occur if there was a "gap" between the two indexes during an update(...) call, but would if there is no gap, but not overlapping. This is suspicious since we would expect the same behaviour (no error in both cases). The error I'm investigating will be fixed soon (empty type fix), but this investigation reveals there are some logical errors in how we calculate if timeseries are overlapping.

Steps/Code to Reproduce

import arcticdb as adb
import pandas as pd
ac = adb.Arctic('lmdb:///tmp')
l2 = ac.get_library('test', create_if_missing=1)
l2._nvs._lib_cfg.lib_desc.version.write_options.dynamic_schema=True
l2.write('k', pd.DataFrame([None, None, None], columns=['a'], index=pd.to_datetime([1,2,3])))
l2.update('k', pd.DataFrame(['', '', ''], columns=['a'], index=pd.to_datetime([4,5,6])))
# Get the error which should only be the case for overlapping indexes
InternalException: E_ASSERTION_FAILURE Allocate data called with zero size

whereas

l2.write('k', pd.DataFrame(['', '', ''], columns=['a'], index=pd.to_datetime([1,2,3])))
l2.update('k', pd.DataFrame([None, None, None], columns=['a'], index=pd.to_datetime([5,6,7])))

does not show the error.

Expected Results

Consistent behaviour between the two cases: either an error in both cases, or in neither, depending on if the other bug has been fixed.

OS, Python Version and ArcticDB Version

Python: 3.8.12 | packaged by conda-forge | (default, Jan 30 2022, 23:42:07) [GCC 9.4.0] OS: Linux-3.10.0-1160.102.1.el7.x86_64-x86_64-with-glibc2.10 ArcticDB: 4.0.3

Backend storage used

LMDB

Additional Context

No response

joe-iddon avatar Feb 06 '24 17:02 joe-iddon