hydroflow icon indicating copy to clipboard operation
hydroflow copied to clipboard

Flow propagation sort order is sub-optimal, naive implementation goes through multiple passes, sometimes emits duplicate operator diagnostics

Open MingweiSamuel opened this issue 2 years ago • 5 comments

Old name: Flow props order of propagation is broken, late-assigned props may conflict with already-visited operators

#882

Example: example_5_reachability Graph:

%%{init:{'theme':'base','themeVariables':{'clusterBkg':'#ddd','clusterBorder':'#888'}}}%%
flowchart TD
classDef pullClass fill:#8af,stroke:#000,text-align:left,white-space:pre
classDef pushClass fill:#ff8,stroke:#000,text-align:left,white-space:pre
linkStyle default stroke:#aaa,stroke-width:4px,color:red,font-size:1.5em;
subgraph sg_1v1 ["sg_1v1 stratum 0"]
    1v1[\"(1v1) <code>source_iter(vec![0])</code>"/]:::pullClass
    2v1[\"(2v1) <code>source_stream(edges_recv)</code>"/]:::pullClass
    7v1[\"(7v1) <code>union()</code>"/]:::pullClass
    3v1[\"(3v1) <code>map(|v| (v, ()))</code>"/]:::pullClass
    4v1[\"(4v1) <code>join()</code>"/]:::pullClass
    5v1[\"(5v1) <code>flat_map(|(src, ((), dst))| [src, dst])</code>"/]:::pullClass
    6v1[/"(6v1) <code>tee()</code>"\]:::pushClass
    8v1[/"(8v1) <code>unique()</code>"\]:::pushClass
    9v1[/"(9v1) <code>for_each(|x| println!(&quot;Reached: {}&quot;, x))</code>"\]:::pushClass
    10v1["(10v1) <code>handoff</code>"]:::otherClass
    10v1-->|cycle|7v1
    1v1-->|base|7v1
    2v1-->|1|4v1
    7v1-->3v1
    3v1-->|0|4v1
    4v1-->5v1
    5v1-->6v1
    6v1-->10v1
    6v1-->|print|8v1
    8v1-->9v1
    subgraph sg_1v1_var_my_join_tee ["var <tt>my_join_tee</tt>"]
        4v1
        5v1
        6v1
    end
    subgraph sg_1v1_var_origin ["var <tt>origin</tt>"]
        1v1
    end
    subgraph sg_1v1_var_reached_vertices ["var <tt>reached_vertices</tt>"]
        7v1
    end
    subgraph sg_1v1_var_stream_of_edges ["var <tt>stream_of_edges</tt>"]
        2v1
    end
end

Propegation order: [GraphNodeId(1v1), GraphNodeId(2v1), GraphNodeId(3v1), GraphNodeId(4v1), GraphNodeId(5v1), GraphNodeId(6v1), GraphNodeId(7v1), GraphNodeId(8v1), GraphNodeId(9v1), GraphNodeId(10v1)]

MingweiSamuel avatar Aug 30 '23 21:08 MingweiSamuel

If flow_props form a lattice, this will reach (the same) fixpoint in any order.

jhellerstein avatar Oct 16 '23 19:10 jhellerstein

We should eventually ensure the representation of the flowprops is a lattice

MingweiSamuel avatar Oct 16 '23 19:10 MingweiSamuel

Let's gather the diagnostics and dedup/output only the "highest" lattice values.

jhellerstein avatar Oct 16 '23 19:10 jhellerstein

Not sure what we're going to do with this lattice-flow-property code exactly

MingweiSamuel avatar Mar 26 '24 18:03 MingweiSamuel

We may end up implementing lattice property tracking a different way, such as in hf+

MingweiSamuel avatar Apr 26 '24 14:04 MingweiSamuel

Depreacted - planning to do more property/lattice traccking in hydroflow plus, as well as doing the optimizations there

MingweiSamuel avatar Aug 12 '24 16:08 MingweiSamuel