hudi icon indicating copy to clipboard operation
hudi copied to clipboard

Published 20 hours ago • verified publisher icon apache

[HUDI-9359] add TIMESTAMP_MILLIS and TIME_MILLIS to InternalSchema

Open jonvex opened this issue 6 months ago • 1 comments

Change Logs

add new types TIMESTAMP_MILLIS and TIME_MILLIS to InternalSchema

add validation for row writer that we don't use timestamp millis

testing added the timestamp and time types to the schema so that TestAvroSchemaEvolutionUtils.testFixNullOrdering() will validate that the logical types are preserved.

Impact

Prevent information loss when converting avro schema to internal schema

Risk level (write none, low medium or high below)

low

Documentation Update

N/A

Contributor's checklist

  • [ ] Read through contributor's guide
  • [ ] Change Logs and Impact were stated clearly
  • [ ] Adequate tests were added if applicable
  • [ ] CI passed

jonvex avatar May 12 '25 19:05 jonvex

CI report:

  • 2424b580bd7b9748f8712db266846256a38c7604 UNKNOWN
  • d1fa7b7dfa483b5fe3e5e9e95c70cf926b2fd03f UNKNOWN
  • e312ae7e56c8793dba5f52e561b12ecac2eb46f6 UNKNOWN
  • 2485a2ab5d077739d1e911b4186cd57672dc0cb8 Azure: FAILURE
Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

hudi-bot avatar Jul 22 '25 18:07 hudi-bot

Given that the timestamp-millis type support is added by #13711 on master, we should get this fix into branch-0.x and disable col stats index on timestamp millis columns for 0.x releases.

yihua avatar Oct 07 '25 17:10 yihua