hudi icon indicating copy to clipboard operation
hudi copied to clipboard

[SUPPORT]--source-ordering-field should be deprecated to honor precombine field

Open bithw1 opened this issue 1 year ago • 3 comments

I am using Hudi 0.15, I find that ORDERING_FIELD has been deprecated with PRECOMBINE_FIELD_NAME,

  /** @deprecated Use {@link HoodieWriteConfig#PRECOMBINE_FIELD_NAME} and its methods instead */
  @Deprecated
  public static final ConfigProperty<String> ORDERING_FIELD = ConfigProperty
      .key(PAYLOAD_ORDERING_FIELD_PROP_KEY)
      .defaultValue("ts")
      .markAdvanced()
      .withDocumentation("Table column/field name to order records that have the same key, before "
          + "merging and writing to storage.");

but HoodieStreamer still use --source-ordering-field as parameter, I think HoodieStreamer should also deprecate --source-ordering-field and introduce a new parameter to honor precombine field to make concept consistent

    @Parameter(names = {"--source-ordering-field"}, description = "Field within source record to decide how"
        + " to break ties between records with same key in input data. Default: 'ts' holding unix timestamp of record")
    public String sourceOrderingField = "ts";

bithw1 avatar Sep 25 '24 09:09 bithw1

Agree, maybe you can fire a fix for it, we can firstly deprecate it.

danny0405 avatar Sep 26 '24 01:09 danny0405

@bithw1 Do you want to contribute on this?

ad1happy2go avatar Oct 03 '24 12:10 ad1happy2go

IIRC source ordering field config in hudi streamer is being used to set precombine field, so this is really just a naming mismatch/confusion (?)

kazdy avatar Oct 19 '24 21:10 kazdy