realtime
realtime copied to clipboard
supabase
[WIP] Backlog of events
What kind of change does this PR introduce?
This PR fixes #125 by adding an additional module BacklogReplication that saves all DB updates to the disk.
Also, it suggests separating exports (Webhook/WebSocket) notifications to different processes and slows down the Websocket rate.
What is the current behavior?
- A large transaction can kill the
Realtimebecause all events are stored directly in a process state - Overflowing the message box through broadcast
- In the next code, two problems https://github.com/supabase/realtime/blob/c293c7680b84e19a068787d0d7f8e49ef8c275f8/server/lib/realtime/subscribers_notification.ex#L15-L16
The WebhookConnector.notify has to wait until the notify_subscribers finished, and if the notify_subscribers crashes, the WebhookConnector.notify never starts.
What is the new behavior?
Each exporter (Webhook/WebSocket) subscribes to a process group.
A BacklogReplication stores DB updates to disk and then sends the transactional to subscribed exporters. The system environment variable "REPLICATION" can change the replication module.
The Adapters.Changes.BacklogTransaction structure contains a link to a part of the file where stored updates. The changes decode in a time when need send to clients.
IMPORTANT. In the current implementation, the transaction for webhook is limited to 10 000 changes because that place can crash the application too.
Additional context
First of all, PR made for EC2 t4g.micro instance.
TODO:
- [x] Store DB changes to disc
- [ ] Decide how right handle large transactions for Webhook
- [ ] Add unit tests
@abc3 really excited for this PR! Let me know when it's ready and I'll start reviewing. Thanks!
@w3b6x9 of course, I've just stuck in routine tasks, but I will continue soon.
@w3b6x9, you can already start to review or testing. Interesting what you think about that. And I'll start writing tests.
