ArcticDB
ArcticDB copied to clipboard
Updating an array type is not working propery
Describe the bug
Assume a dense column of an array type. After update is run no sparse map will be written (as it's dense). The array type handler however assumes that if the sparse map is missing all values are None.
Note 1: The same behavior was observed for nullable booleans and fixed in #1227 Note 2: As of writing this issue array type is feature flagged as it occasionally crashes due to allocating python memory for a thread not holding the GIL. Do not enable arrays before fixing both issues.
Steps/Code to Reproduce
>>> lib.write("sym", pd.DataFrame({"col": [np.array([1,2,3]), np.array([1,2,3]), np.array([1,2,3]), np.array([1,2,3])]}, index=list(pd.date_range(start="1/1/2024", end="1/4/2024"))))
>>> lib.update("sym", pd.DataFrame({"col": [np.array([5,6,7]), np.array([8,9,10])}, index=list(pd.date_range(start="1/2/2024", end="1/3/2024"))))
>>> lib.read("sym")
col
None
[5,6,7]
[8,9,10]
None
Expected Results
col
[1,2,3]
[5,6,7]
[8,9,10]
[1,2,3]
OS, Python Version and ArcticDB Version
ArcticDB dev
Backend storage used
No response
Additional Context
No response