incubator-gluten icon indicating copy to clipboard operation
incubator-gluten copied to clipboard

[CH] HiveText format file read diff from spark

Open KevinyhZou opened this issue 10 months ago • 0 comments

Backend

CH (ClickHouse)

Bug description

HiveText format file read diff from spark

  1. when the hive text file first line has different number of fields from the table defines, then the left fields can not be read, and this is fixing at clickhouse pr : https://github.com/ClickHouse/ClickHouse/pull/62086
  2. Somethimes the HiveTextScanTransformer's output attributes order is mismatched with the table fields defines, which would make the field read order is reversed.

Spark version

Spark-3.3.x

Spark configurations

No response

System information

No response

Relevant logs

No response

KevinyhZou avatar Mar 29 '24 11:03 KevinyhZou