Danny Chan

I'm pretty sure it is a jar conflict, can you check the jar that involves the reported class?

You may need to tweak the `clean.retain_commits` option.

Is it beause they are being clustered continuously? And do you already skip reading the clustered files?

There are some logs that reports the reader progress in the monitor operator, you can check that to see if the reader lags too much from the producer.

If the job is not executing rollback repetitively, these files should be just a replacing of "COW" of files, for "COW", we create a new base file to replace the...

> clustered still. > And the downstream flink program read these files would met FileNOTEXTIES exception. Either clustering and compaction can be skipped in flink streaming read.

We did have the tests already in the repo for clustering and compaction skipping read, can you ensure the option takes effect and increase the numbers of retained commits before...

> clean.retain_commits was 1 That means each time a new version of file generated, the old one would be deleted, for "COW" table, there is very high possibility you would...

Are you enabling the clustering then? The clustering would rewrite all the partitions. > I think increasing the parameters of retention cleanup will probably generate more files The small files...

This is a replace commit, you can choose to skip it with option `read.skip_clustering` or `read.skip_insertoverride` enabled.