doris icon indicating copy to clipboard operation
doris copied to clipboard

[feature](parquet-reader) add delta bit pack decoder

Open wsjz opened this issue 2 years ago • 1 comments

Proposed changes

Support delta encoding, add delta bit pack decoder, add delta length byte array decoder, add delta byte array decoder.

We find some data type is read with delta encoding on AWS Glue, so it should be supported. The definition of delta encoding can refer to the delta encoding in parquet.

Problem summary

Describe your changes.

Checklist(Required)

  • [ ] Does it affect the original behavior
  • [ ] Has unit tests been added
  • [ ] Has document been added or modified
  • [ ] Does it need to update dependencies
  • [ ] Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

wsjz avatar Feb 24 '23 09:02 wsjz

run buildall

morningman avatar Mar 06 '23 00:03 morningman

LGTM

AshinGau avatar Mar 06 '23 12:03 AshinGau

wait for regression case

wsjz avatar Mar 06 '23 12:03 wsjz

run buildall

wsjz avatar Mar 10 '23 05:03 wsjz

run p0

wsjz avatar Mar 12 '23 09:03 wsjz