materialize
materialize copied to clipboard
TAIL and SINK can miss writes due to conccurrent DDL
What version of Materialize are you using?
main
How did you install Materialize?
Built from source
What is the issue?
Group commit causes writes to be asynchronous. If a table is deleted while a write is pending in group commit, then we just respond to the client with an OK. See below for more details: https://github.com/MaterializeInc/materialize/blob/19c9736241017477be9d996d45d44e6a7b8a8e07/src/adapter/src/coord/appends.rs#L243-L250
The issue is that if someone is tailing a table when this happens, then they will never see the write even though we responded with an OK for the write.
DDL was recently updated to explicitly trigger a group commit. That means that before deleting a table, we drain all pending writes. Therefore, it's impossible for a table to be deleted before a the write is sent. So we can remove this check and instead replace it with an assert that the collection exists.
Relevant log output
No response