[HUDI-7240] Clean delete logic

[linliu-code]

Change Logs

1. When we create HoodieRecord for a delete, we store the necessary information into the metadata field.
2. When we need to merge delete records, we extract orderingVal from metadata field of HoodieRecord.
3. Removed HoodieRecordTestPayload.

Impact

Simplifies the logic for handling delete records.

Risk level (write none, low medium or high below)

Low.

Contributor's checklist

• [ ] Read through contributor's guide
• [ ] Change Logs and Impact were stated clearly
• [ ] Adequate tests were added if applicable
• [ ] CI passed

[linliu-code]

@yihua @codope @danny0405

[linliu-code]

Will clean the failures.

[hudi-bot]

CI report:

• 3d71d1c0e3220f0639b702d91539e1d070e93cca Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[danny0405]

Thanks for raising this fix, I think it is a good chance we fix the event time sequence comparison of delete records with payloads, I can see 2 mistaks in our code that uses processing time sequence for deletes:

1. OverwriteWithLatestAvroPayload#preCombine:

  public OverwriteWithLatestAvroPayload preCombine(OverwriteWithLatestAvroPayload oldValue) {
    if (oldValue.recordBytes.length == 0) {
      // use natural order for delete record
      return this;
    }
    ...
  }

2. DefaultHoodieRecordPayload#combineAndGetUpdateValue

  public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema, Properties properties) throws IOException {
    if (recordBytes.length == 0) {
      return Option.empty();
    }

    ...
  }

In any case, the orderingVal should be set up correctly and we should utilize it as much as possible.