materialize icon indicating copy to clipboard operation
materialize copied to clipboard

TAIL and SINK can miss writes due to conccurrent DDL

Open jkosh44 opened this issue 2 years ago • 0 comments

What version of Materialize are you using?

main

How did you install Materialize?

Built from source

What is the issue?

Group commit causes writes to be asynchronous. If a table is deleted while a write is pending in group commit, then we just respond to the client with an OK. See below for more details: https://github.com/MaterializeInc/materialize/blob/19c9736241017477be9d996d45d44e6a7b8a8e07/src/adapter/src/coord/appends.rs#L243-L250

The issue is that if someone is tailing a table when this happens, then they will never see the write even though we responded with an OK for the write.

DDL was recently updated to explicitly trigger a group commit. That means that before deleting a table, we drain all pending writes. Therefore, it's impossible for a table to be deleted before a the write is sent. So we can remove this check and instead replace it with an assert that the collection exists.

Relevant log output

No response

jkosh44 avatar Aug 08 '22 14:08 jkosh44