hydroflow icon indicating copy to clipboard operation
hydroflow copied to clipboard

Deprecate/refactor old stateful/replaying operators

Open MingweiSamuel opened this issue 1 year ago • 1 comments

  • [ ] replace join with state_join which can use keyed datastructures from state() op #969, bypass #564
  • [ ] remove persist() (or replace with state()), replace [lattice_]reduce/fold with state() versions #925
  • [ ] mutable state with state_mut() operator, with extra delete input #785
  • [ ] Deprecate message for others

MingweiSamuel avatar Feb 05 '24 17:02 MingweiSamuel

Replay on joins/bimorphisms is now working correctly by insisting on persisting the inputs (which is required semantically for bimorphism).

Remaining work plan is around references:

  1. Introduce an n-way cross operator to gather refs. Give it a meaningful name (not cross-join; maybe "tuple" or "gatherRefs" or something)
  2. Make sure/rewrite code so that output of the n-way cross is reflected in argument of its consumer.
  3. Check that replay of refs through the cross operator works as expected.

jhellerstein avatar May 13 '24 16:05 jhellerstein

We could do this now, but it would be better to gather more experience, and perhaps wait for more helpers in Hydroflow+ before we remove the more ergonomic existing "convenience" operators.

jhellerstein avatar May 20 '24 16:05 jhellerstein

Low priority of 0.9 release - hoping to drive from hydroflow+ use cases, and filter downward what works/is needed

And look at Shadaj's Flo paper

MingweiSamuel avatar Jul 16 '24 16:07 MingweiSamuel