ArcticDB icon indicating copy to clipboard operation
ArcticDB copied to clipboard

Updating an array type is not working propery

Open vasil-pashov opened this issue 1 year ago • 0 comments

Describe the bug

Assume a dense column of an array type. After update is run no sparse map will be written (as it's dense). The array type handler however assumes that if the sparse map is missing all values are None.

Note 1: The same behavior was observed for nullable booleans and fixed in #1227 Note 2: As of writing this issue array type is feature flagged as it occasionally crashes due to allocating python memory for a thread not holding the GIL. Do not enable arrays before fixing both issues.

Steps/Code to Reproduce

>>> lib.write("sym", pd.DataFrame({"col": [np.array([1,2,3]), np.array([1,2,3]), np.array([1,2,3]), np.array([1,2,3])]}, index=list(pd.date_range(start="1/1/2024", end="1/4/2024"))))
>>> lib.update("sym", pd.DataFrame({"col": [np.array([5,6,7]), np.array([8,9,10])}, index=list(pd.date_range(start="1/2/2024", end="1/3/2024"))))
>>> lib.read("sym")

col
None
[5,6,7]
[8,9,10]
None

Expected Results

col
[1,2,3]
[5,6,7]
[8,9,10]
[1,2,3]

OS, Python Version and ArcticDB Version

ArcticDB dev

Backend storage used

No response

Additional Context

No response

vasil-pashov avatar Jan 29 '24 12:01 vasil-pashov