dremio-oss icon indicating copy to clipboard operation
dremio-oss copied to clipboard

fix: check for UseFileSplitsFromInputFormat annotation in hive plugins

Open zeroshade opened this issue 4 years ago • 1 comments

When using custom input formats with Hive that are parquet based the Hudi Project introduced the custom annotation "UseFileSplitsFromInputFormat" to tell hive plugins to explicitly use the file splits from the InputFormat rather than calling it from MapredParquetInputFormat. This enhances the hive2 and hive3 plugins in dremio to respect this annotation and call getSplits on the inputformat when found in order to ensure proper handling and filtering of splits/parquet files with custom input formats.

zeroshade avatar May 18 '21 16:05 zeroshade

LGTM

WilliamWhispell avatar May 25 '21 01:05 WilliamWhispell