Spark 3.5: Add max allowed failed commits to RewriteDataFiles when partial progress is enabled

[manuzhang]

When partial progress is enabled, rewrite_data_files Spark app can succeed without any "progress" (commits). This PR adds an option partial-progress.max-failed-commits and a RuntimeException will be thrown if the number of failed commits exceed it. The default value is partital-progress.max-commits to be consistent with current behavior.

[manuzhang]

@amogh-jahagirdar This is based on our discussion in #9400, but I'd like to go one step further. Throwing exception and fail can work with alert system easier than logging an error. WDYT?

[aokolnychyi]

Sorry for the delay. I didn't forget.

[aokolnychyi]

Looks like there are some related test failures:

TestRewriteDataFilesAction > testParallelPartialProgressWithMaxFailedCommits() FAILED
    org.opentest4j.AssertionFailedError: [We shouldn't have changed the data] 

[aokolnychyi]

I think we should continue to use assertEquals method (which is our custom assert method). It has proper value equality for Spark.

[aokolnychyi]

Thanks, @manuzhang! Thanks for reviewing, @nastra @RussellSpitzer!