deephaven-core icon indicating copy to clipboard operation
deephaven-core copied to clipboard

fix: Ensure ParquetInstruction's TableDefinition is set for ParquetTableLocation

Open devinrsmith opened this issue 1 year ago • 2 comments

Fixes an inconsistency between ParquetTools#readSingleFileTable and ParquetTools#readTable in regards to whether they set the TableDefinition in the ParquetInstructions after inference.

This enforces that the TableDefinition must be present when creating a ParquetTableLocation and IcebergTableParquetLocationKey (this is implicitly true after the change above, but this is making that requirement explicit).

This is done as part of an effort towards #6128.

devinrsmith avatar Sep 27 '24 21:09 devinrsmith

Note: checking w/ DHE on their usage, which seems to always pass ParquetInstructions.EMPTY - they might have this luxury because they control the writing and can assume 100% match between the Parquet files and their Schema.

devinrsmith avatar Sep 27 '24 22:09 devinrsmith

Note: the core of this PR has been merged as part of #6184; I do think it makes sense to continue pushing forward and try to be stricter.

devinrsmith avatar Oct 11 '24 15:10 devinrsmith