rewritedatafile: Cannot commit, found new position delete for replaced data

[chenwyi2]

i use spark to run rewritedatafile， and i have a flink job write this table every 5 min, but if the rewritefatafile spend more than 5 min, it will commit error with "ValidationException: Cannot commit, found new position delete for replaced data file: GenericDataFile{content=data, file_path=qbfs://online01/warehouse/prod_censor_datalake.db/document/dataxxx", so there is a good solution? so now i must keep rewritedatafile in 5 min, otherwise it will commit failed

[chenwyi2]

and i already use using use-starting-sequence-number variable, iceberg version is 0.13.1

[jingli430]

I am seeing the same issue. I am ok If there is new position-delete found and then we cannot commit here, but could we have a strategy to auto-retry?

[chenwyi2]

we use cdc mode, the data is always changing, even if we use strategy to auto-retry, rewritedatafile will always be failed

[humengyu2012]

My iceberg version is 1.0.0, but the same problem occurs.

[humengyu2012]

Is there any recent progress on this issue?

[anistal]

Same here, it is a big issue because we can not ingest and optimize at the same time. Is there any idea or workaround to solve it?