[HUDI-8449] Fix deletion of record from FILES partition on empty files list

[vamsikarnika]

Change Logs

When entire file list becomes empty for a given partition(delete partition), we need to get rid of the entire record from FILES partition.

Impact

Metadata table will have no entry for a data table partition key in files partition when all files in that data table partition are deleted.

Risk level (write none, low medium or high below)

Low

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the ticket number here and follow the instruction to make changes to the website.

Contributor's checklist

• [ ] Read through contributor's guide
• [ ] Change Logs and Impact were stated clearly
• [ ] Adequate tests were added if applicable
• [ ] CI passed

[danny0405]

This explains why the metadata payload with isDelete as true does not write into the delete block: https://github.com/apache/hudi/pull/13411/commits/efce133ff21bb9116d1ad0330e404f150d0f7f41

[vamsikarnika]

This explains why the metadata payload with isDelete as true does not write into the delete block: efce133

@danny0405 I've modified this pr to create a delete log block for file Metadata when partition is deleted.

[hudi-bot]

CI report:

• 50820a5a8a07ee266ea2d7c4b491f0ac55c43aeb UNKNOWN
• be3d9e5811eb7c3bea97eea0fa9c21a203212e48 UNKNOWN
• a1a85b713c07566e69afcf5ad78e137081b49316 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build