delta
delta copied to clipboard
[Kernel][Writes] Add support for writing data file stats
Which Delta project/connector is this regarding?
- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [x] Kernel
- [ ] Other (fill in here)
Description
Serializes stats to the data file on writes.
How was this patch tested?
Unit tests.
Does this PR introduce any user-facing changes?
No
Will fix the tests.
Looks great! Left only a few minor comments / questions.
Seems like we need some tests for
DEFAULT_DATA_SKIPPING_NUM_INDEXED_COLSand, in particular, need to test + document the expected behavior for how nested columns impact the total column count.
Thanks @scottsand-db , added a test and a clarifying comment for the same