incubator-gluten
incubator-gluten copied to clipboard
[CH] HiveText format file read diff from spark
Backend
CH (ClickHouse)
Bug description
HiveText format file read diff from spark
- when the hive text file first line has different number of fields from the table defines, then the left fields can not be read, and this is fixing at clickhouse pr : https://github.com/ClickHouse/ClickHouse/pull/62086
- Somethimes the HiveTextScanTransformer's output attributes order is mismatched with the table fields defines, which would make the field read order is reversed.
Spark version
Spark-3.3.x
Spark configurations
No response
System information
No response
Relevant logs
No response