electric-sql
Send only rows for matching shapes instead of transactions
When a row in a transaction matches a shapes we're sending the entire transaction payload to the Shape Consumer. Shape Consumers will claim enough memory to hold the entire transaction which is not released until the process hibernates or terminates.
One hypothesis would be to have the shape matching algorithm sending only the rows, the advantage of this is that processes would possibly hold less memory, the downside is that we need to do a bit more processing sequentially. We want to test and evaluate this hypothesis.
There does look like there would be a benefit to this change as we do have large transactions where many of the changes are unused (bottom graph - MAX(unused_changes)) : https://ui.honeycomb.io/electric-sql-06/environments/production/datasets/electric-region/board-query/61CQ11i6Zqw/result/jCDCqQzUDjJ?vs=hideCompare&tab=overview
It will however, as mentioned above, mean more processing is done on the single ShapeLogCollector process. We currently don't have telemetry that measures that cost, but I can add some.
This change would not be necessary if we streamed transactions, which has been discussed. Streaming transactions would have the benefit of smaller memory footprint and concurrent processing.
That said we need not do the entire filter_changes function in ShapeLogCollector. We could just use the Filter to do the filtering and keep the change transformations (which are part of filter_changes) in the consumer.