EJ Song

@andrei-ionescu I know this change all comes together, but could you have split this PR into small PRs by any chance? Since it's bit hard to get these in at...

@andrei-ionescu I have the same experience when I was working on Hybrid Scan last year - #123 (one big PR, but rejected 🙅) and I split the PR into several...

Yea as most of functionality is validated in this PR, we can do small PRs only for merging them partially. It's too long scrooooll and got distracted in the middle...

@andrei-ionescu Q1: Looks good Q2: Yep, but 1 ~ 2 PRs (at a time) would be good; just don't create all PRs at first.

Hi @dennyglee, what about Auto compaction and Optimize Write? I don't think the PRs are getting some attention for review / merge. Could you add it to the roadmap?

Hi @dennyglee any update?

@tdas Append operation throws ConcurrentAppendException. ``` io.delta.exceptions.ConcurrentAppendException: Files were added to partition [colC=1] by a concurrent update. Please try the operation again. Conflicting commit: {"timestamp":1662599111087,"operation":"OPTIMIZE", ... ``` https://github.com/delta-io/delta/blob/2041c3b7138b9614529540613195d4afd39fde71/core/src/main/scala/org/apache/spark/sql/delta/ConflictChecker.scala#L177-L188 This change...

@tdas I found that there's still can be a conflict if concurrent transaction reads a RemoveFile from Optimize. io.delta.exceptions.ConcurrentDeleteReadException: This transaction attempted to read one or more files that were...

@scottsand-db Seems the PR allows to switch isolation level to WriteSerializable. Not sure any other changes will be delivered, but with the current code, the issue still exists: https://github.com/delta-io/delta/blob/d2785aa93e8f2f5c70a3351ddc4995e2c943897b/core/src/main/scala/org/apache/spark/sql/delta/ConflictChecker.scala#L181-L188 it...

@tdas Ready-For-Review