metaflow-service icon indicating copy to clipboard operation
metaflow-service copied to clipboard

Implement a messaging service for UI real-time updates, for read-replica support

Open saikonen opened this issue 2 years ago • 0 comments

Suggested improvement Introduce a separate service to behave as a pub/sub message queue for real-time updates related to the metadata service. Metadata service would publish to this message queue upon every successful insert and update of records on a fire-and-forget basis. ui_backend_service would then subscribe to this services relevant topics and receive near-realtime events related to the metadata service.

Prefer a separate service instead of tying the feature as part of the metadata service, to avoid impacting the metadata service performance with additional load that broadcasts might incur. No strong opinions whether this should be a custom built or an off the shelf solution.

Motivation Currently the ui_backend relies heavily on PostgreSQL features for broadcasting metadata related changes. One limitation of this approach is with the possible replication strategies that the ui_backend can employ. With event notifications being offloaded to a separate service, the ui_backend could easily run off a read-replica and require less setup when deploying.

This would align well with suggestions like #2 as well

saikonen avatar Nov 12 '21 20:11 saikonen