lance icon indicating copy to clipboard operation
lance copied to clipboard

feat: add a packed struct encoding to lance

Open raunaks13 opened this issue 1 year ago • 1 comments

Introduces a new PackedStruct encoding, should speed up random access for struct data, ref #2601

  • Can currently support non-nullable, primitive fixed-length types (including fixed size list)
  • Implemented as a physical type array encoder
  • The user can select whether they want to use this encoding by specifying the field "packed" as true or false in the metadata. The default will use the old StructFieldEncoder

raunaks13 avatar Jul 12 '24 18:07 raunaks13

Codecov Report

Attention: Patch coverage is 91.78744% with 34 lines in your changes missing coverage. Please review.

Project coverage is 79.25%. Comparing base (501a642) to head (825042a).

Files Patch % Lines
rust/lance-encoding/src/decoder.rs 82.02% 9 Missing and 7 partials :warning:
...e-encoding/src/encodings/physical/packed_struct.rs 96.61% 5 Missing and 2 partials :warning:
rust/lance-encoding-datafusion/src/zone.rs 0.00% 4 Missing :warning:
rust/lance-encoding/src/encoder.rs 93.75% 0 Missing and 3 partials :warning:
rust/lance-encoding-datafusion/src/lib.rs 0.00% 2 Missing :warning:
rust/lance-encoding/src/encodings/physical.rs 96.42% 1 Missing :warning:
rust/lance-encoding/src/encodings/utils.rs 92.85% 0 Missing and 1 partial :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2593      +/-   ##
==========================================
+ Coverage   79.14%   79.25%   +0.11%     
==========================================
  Files         218      219       +1     
  Lines       63617    63927     +310     
  Branches    63617    63927     +310     
==========================================
+ Hits        50351    50668     +317     
+ Misses      10325    10311      -14     
- Partials     2941     2948       +7     
Flag Coverage Δ
unittests 79.25% <91.78%> (+0.11%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov-commenter avatar Jul 18 '24 07:07 codecov-commenter