electric icon indicating copy to clipboard operation
electric copied to clipboard

Send individual changes to consumers

Open robacourt opened this issue 2 months ago • 0 comments

Part of #3413 . The shape log collector should send individual changes, transaction starts and commits to the shapes that they affect. The consumers can then buffers to form a filtered transaction from the changes and then process as normal. This will mean a consumers working memory is not expanded to the size of the full transaction when the shape only needs a subset of it. It also paves the way for the rest of #3413

Note: For shapes that depend on subqueries this will mean the Filter may have out of date materialised views at the point of sending the change to the consumer (as it does now) however this does not matter as:

  1. For move ins the missing data will be fetched from Postgres
  2. For move outs, the extra data should be filtered out by the consumer (as it does now)

Commit messages should be sent in dependency layers (as the transactions are done now) and not proceed to the next layer until a response from the commit has been received. Only commit messages will have responses, other changes are just cast to the consumer. The ShpeLogCollector will not proceed to the next transaction until all responses (or exit signals) to all commit messages sent have been received.

Transaction start and commit messages will need to be sent to all consumers who receive at least 1 change from the transaction.

Performance should not be compromised by this change, we may need to buffer messages to not bare the cost of the additional context switching from sending multiple messages rather than a whole transaction in 1 message.

robacourt avatar Nov 11 '25 09:11 robacourt