[HUDI-7411] Meta sync should consider cleaner commit

[codope]

Change Logs

Cleaner could not delete partitions but meta sync fails to drop partition in that case. This could cause query using engines that depend on catalog to fail.

TODO: I have only tested locally. I am going to add a test.

Impact

Catalog will reflect correct partition metadata considering cleaner commits.

Risk level (write none, low medium or high below)

low

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the ticket number here and follow the instruction to make changes to the website.

Contributor's checklist

• [ ] Read through contributor's guide
• [ ] Change Logs and Impact were stated clearly
• [ ] Adequate tests were added if applicable
• [ ] CI passed

[codope]

@nsivabalan Added a test, please take a look.

[hudi-bot]

CI report:

• 2433556442faea975ed3c97a0abb0a8adf2610c8 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build