electric
electric copied to clipboard
Reduce working memory of consumer processes not involved in subqueries
A shape consumer's heap currently expands to allow complete transaction to be held in memory, however there are scenarios where the complete transaction does not need to be held in memory:
- When the shape only needs a subset of the changes in the transaction
- When the shape does not depend on subqueries (it can instead send a change at a time to the storage)
These can be addressed in two parts:
- The shape log collector to send individual changes to the shapes that they affect. The consumers can then buffers to form a filtered transaction from the changes and then process as normal. This will address scenario 1 above, while also allowing for part 2 below.
- The consumer only buffers when the shape depends on subqueries. If the shape does not depend on subqueries, changes are written straight to file. This addresses scenario 2 above.