materialize
materialize copied to clipboard
[Epic] Engineering Differential for Memory and CPU reduction
Product outcome
Differential (and Timely) are the engine/s that run our database capability/functionality. These underlying libraries are big sources of potential memory savings (e.g. already done by lgalloc) and of CPU-use reduction. This Epic encompasses multiple different implementation changes that will each tackle a different aspect of memory use and algorithmic (or other) improvement.
Work items
- Merge Batcher (est 2 weeks)
- The Builder -- enhance so that it works with containers. (set 2 weeks design and 2 implementation)
- Fixing Differential joins. (no est yet)
- Fixing linear joins (this is actually at the Materialize layer, not inside Differential, but fits with the theme). (no est yet)
- Enabling container use in dataflow edges in Materialize (no est yet)
### Tasks
- [ ] https://github.com/MaterializeInc/materialize/issues/26914
- [ ] https://github.com/MaterializeInc/materialize/issues/26915
- [ ] https://github.com/MaterializeInc/materialize/issues/26916
- [ ] https://github.com/MaterializeInc/materialize/issues/26917
- [ ] https://github.com/MaterializeInc/materialize/issues/26918