cudf icon indicating copy to clipboard operation
cudf copied to clipboard

Round trip FIXED_LEN_BYTE_ARRAY data properly in Parquet writer

Open etseidl opened this issue 2 years ago • 5 comments

Description

#13437 added the ability to consume FIXED_LEN_BYTE_ARRAY encoded data and represent it as lists of UINT8. When trying to write this data back to Parquet there are two problems. 1) the notion of fixed length is lost, and 2) the UINT8 data is written as a list of INT32 which can quadruple the storage required. This PR addresses both issues by adding fields to the input and output metadata to allow for preserving the form of the original data.

Checklist

  • [x] I am familiar with the Contributing Guidelines.
  • [x] New or existing tests cover these changes.
  • [x] The documentation is up to date with these changes.

etseidl avatar Apr 25 '24 18:04 etseidl

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

copy-pr-bot[bot] avatar Apr 25 '24 18:04 copy-pr-bot[bot]

CC @vuule

etseidl avatar Apr 25 '24 18:04 etseidl

/ok to test

vuule avatar Apr 25 '24 19:04 vuule

/ok to test

vuule avatar May 03 '24 19:05 vuule

/ok to test

vuule avatar May 08 '24 00:05 vuule

/merge

vuule avatar May 08 '24 03:05 vuule