texera icon indicating copy to clipboard operation
texera copied to clipboard
verified publisher icon Texera

Enhance incremental computation support in Texera

Open zuozhiw opened this issue 2 years ago • 1 comments

This PR enhances incremental computation support in Texera, including:

  1. Added an incremental aggregation framework. Specifically:
  • PartialAggregateOpExec and FinalAggregateOpExec are updated to use incremental computation. They will perdoically emit partial results to downstream.
  • Aggregate and LineChart operators now use the new aggregation framework. Other aggregate-based visualizations are not using it as they are now implemented with Python UDFs and HTML visualizations.
  • WordCloud is not using the new framework, as WordCloud is a special top-k aggregation.
  1. Added a general incremental computation option for all operators. Specifically:
  • Added a new option supportRetractableInput to indicate whether an operator support retractions as input tuples.
  • Added a new incremental computation enforcer that rewrites the workflow based on incremental computation requirements. It propagate the incremental properties and adds a "consolidate" operator if necessary.
  1. Added a simple incremental join operator.

For detailed technical presentation on incremental computation, see this slide and descriptions in this PR

zuozhiw avatar Sep 26 '23 06:09 zuozhiw

will revisit after complier refactoring.

Yicong-Huang avatar Dec 19 '23 19:12 Yicong-Huang

@zuozhiw do you want to work with me to finish this PR?

Yicong-Huang avatar Jun 10 '25 23:06 Yicong-Huang