delta icon indicating copy to clipboard operation
delta copied to clipboard

[Kernel][Writes] Add support for writing data file stats

Open raveeram-db opened this issue 1 year ago • 1 comments

Which Delta project/connector is this regarding?

  • [ ] Spark
  • [ ] Standalone
  • [ ] Flink
  • [x] Kernel
  • [ ] Other (fill in here)

Description

Serializes stats to the data file on writes.

How was this patch tested?

Unit tests.

Does this PR introduce any user-facing changes?

No

raveeram-db avatar Jul 08 '24 17:07 raveeram-db

Will fix the tests.

raveeram-db avatar Aug 05 '24 06:08 raveeram-db

Looks great! Left only a few minor comments / questions.

Seems like we need some tests for DEFAULT_DATA_SKIPPING_NUM_INDEXED_COLS and, in particular, need to test + document the expected behavior for how nested columns impact the total column count.

Thanks @scottsand-db , added a test and a clarifying comment for the same

raveeram-db avatar Mar 05 '25 23:03 raveeram-db