electric icon indicating copy to clipboard operation
electric copied to clipboard

Reduce working memory of consumer processes not involved in subqueries

Open robacourt opened this issue 2 months ago • 0 comments

A shape consumer's heap currently expands to allow complete transaction to be held in memory, however there are scenarios where the complete transaction does not need to be held in memory:

  1. When the shape only needs a subset of the changes in the transaction
  2. When the shape does not depend on subqueries (it can instead send a change at a time to the storage)

These can be addressed in two parts:

  1. The shape log collector to send individual changes to the shapes that they affect. The consumers can then buffers to form a filtered transaction from the changes and then process as normal. This will address scenario 1 above, while also allowing for part 2 below.
  2. The consumer only buffers when the shape depends on subqueries. If the shape does not depend on subqueries, changes are written straight to file. This addresses scenario 2 above.

robacourt avatar Nov 11 '25 08:11 robacourt