[Feature] Conflicts during Compactions Better coping strategies,Instead of restarting

[yangtao0626]

Search before asking

• [X] I searched in the issues and found nothing similar.

Motivation

We have a flink task，only one commit user ,and set 'changelog-producer' = 'lookup', and other compact propreties are default value. flink checkpoint.interval =20s, and max-concurrent-checkpoints=3, restarts are triggered frequently throughout the day.

and the error message looks like the following: Caused by: java.lang.IllegalStateException: Trying to delete file xxxx.orc which is not previously added. Manifest might be corrupted.

Solution

We want to write a bucket in a single task, and we can adopt different coping strategies

1. Give up this compact, he did not have a great impact, but the small file merge is slow 2.Ignore the delete conflict in this compact and commit directly

Anything else?

I don't know if the community is interested in doing these things, or if there are any problems with them？

Are you willing to submit a PR?

• [ ] I'm willing to submit a PR!

[JingsongLi]

The key problem is that the state inside the WRITER has changed, and if you just IGNORE this conflict, then every subsequent COMMIT will conflict.