hudi icon indicating copy to clipboard operation
hudi copied to clipboard

[SUPPORT] does hudi have plans to support default values

Open JoshuaZhuCN opened this issue 1 year ago • 2 comments

Our dataset requires that no field is allowed to have null values, but hudi currently does not support default values, resulting in new fields having null values when querying historical data. The only way currently is to update historical data.

Environment Description

  • Hudi version : 0.12.1

  • Spark version : 3.1.3

  • Hive version : 3.1.0

  • Hadoop version : 3.1.1

  • Storage (HDFS/S3/GCS..) : HDFS

  • Running on Docker? (yes/no) : no

JoshuaZhuCN avatar Dec 18 '23 08:12 JoshuaZhuCN

@JoshuaZhuCN Do you have any idea/design how can that be supported or have any thoughts in your mind? Adding @nsivabalan @yihua @xushiyan @danny0405 @codope

ad1happy2go avatar Dec 18 '23 16:12 ad1happy2go

In SQL, a tricky solution is when pasing the SQL, if a column defines a default value, the parser would translate and wrapper it as a function with a null fallback as the default.

danny0405 avatar Dec 19 '23 03:12 danny0405