materialize icon indicating copy to clipboard operation
materialize copied to clipboard

[Epic] Engineering Differential for Memory and CPU reduction

Open lfest opened this issue 1 year ago • 0 comments

Product outcome

Differential (and Timely) are the engine/s that run our database capability/functionality. These underlying libraries are big sources of potential memory savings (e.g. already done by lgalloc) and of CPU-use reduction. This Epic encompasses multiple different implementation changes that will each tackle a different aspect of memory use and algorithmic (or other) improvement.

Work items

  1. Merge Batcher (est 2 weeks)
  2. The Builder -- enhance so that it works with containers. (set 2 weeks design and 2 implementation)
  3. Fixing Differential joins. (no est yet)
  4. Fixing linear joins (this is actually at the Materialize layer, not inside Differential, but fits with the theme). (no est yet)
  5. Enabling container use in dataflow edges in Materialize (no est yet)
### Tasks
- [ ] https://github.com/MaterializeInc/materialize/issues/26914
- [ ] https://github.com/MaterializeInc/materialize/issues/26915
- [ ] https://github.com/MaterializeInc/materialize/issues/26916
- [ ] https://github.com/MaterializeInc/materialize/issues/26917
- [ ] https://github.com/MaterializeInc/materialize/issues/26918

lfest avatar May 06 '24 09:05 lfest