[HUDI-7867] Ensuring delete invalid all files during finalizing the writer

[beyond1920]

Change Logs

We should not skip failure which means fail to delete invalid files because users might get wrong result. The pr aims to fix the issue#11419.

Impact

Fix the bug when delete invalid files.

Risk level (write none, low medium or high below)

None

Documentation Update

None

Contributor's checklist

• [ ] Read through contributor's guide
• [ ] Change Logs and Impact were stated clearly
• [ ] Adequate tests were added if applicable
• [ ] CI passed

[beyond1920]

It's a draft yet, would add some test case soon.

[hudi-bot]

CI report:

• 713bcd6de66ffe31486fcc79cabfeff9750f940b Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build