ArcticDB icon indicating copy to clipboard operation
ArcticDB copied to clipboard

Empty type decoding: move the segment buffer forward by the compressed data size for empty types

Open muhammadhamzasajjad opened this issue 11 months ago • 2 comments

Reference Issues/PRs

There is a compatibility issue with columns containing empty types written in 4.0.x or earlier. For version 4.0.x, the buffer was remaining behind the correct position which was causing an lz4 decoding error.

What does this implement or fix?

Any other comments?

Checklist

Checklist for code changes...
  • [ ] Have you updated the relevant docstrings, documentation and copyright notice?
  • [ ] Is this contribution tested against all ArcticDB's features?
  • [ ] Do all exceptions introduced raise appropriate error messages?
  • [ ] Are API changes highlighted in the PR description?
  • [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes?

muhammadhamzasajjad avatar Mar 26 '24 17:03 muhammadhamzasajjad

We should add a test that reproduces the original problem (ie writes with 4.0.x and updates with HEAD).

poodlewars avatar Mar 26 '24 19:03 poodlewars

We should add a test that reproduces the original problem (ie writes with 4.0.x and updates with HEAD).

Test should look something like this:

# version 4.0.3
df = pd.DataFrame({'c1': [None, None, None], 'c2': [1.2, None, 1.3]}, index=pd.date_range('1/1/2018', end='1/03/2018'))
lib.write("mysym", df)

# version latest master
df = pd.DataFrame({'c1': [None, None, None], 'c2': [1.1, None, 1.4]}, index=pd.date_range('1/1/2018', end='1/03/2018'))
lib.update("mysym", df)
...

arcticdb_ext.exceptions.InternalException: E_INVALID_ARGUMENT Error while decoding with lz4 at address 7eacd8d4efea with size 21. Code -4 // this error shows  failure in decoding something written with 4.0.3

G-D-Petrov avatar Mar 27 '24 10:03 G-D-Petrov

Superseded by another PR

willdealtry avatar Aug 19 '24 10:08 willdealtry