Joe Hellerstein
Joe Hellerstein
- Need a way to get fine-grained telemetry and control over its aggregation/publishing within a transducer - Need ways to collect/aggregate telemetry across transducers - write to S3/etc - a...
For "tick/tick" join, we'd like to minimize memory pressure by persisting only one side (hopefully the small side), and probing with the other. i.e. a traditional (asymmetric) Hash Join. We...
Lattice-based flows should be at least as easy to write as tuple-oriented flows. In fact, we want our users to default to lattice-based flows to encourage monotonicity.
See the test case `test_lattice_fold_loop` (from `hydroflow/tests/surface_lattice_fold.rs` and pasted below.) It generates the following warning: ``` warning: Input to `union()` will be downcast to `None` to match other inputs. -->...
Here's one version for `R \join S` on streaming inputs. This is essentially an implementation of the following Dedalus code: ``` R@t+1 :- R@t; R@t+1 :- dR; S@t+1 :- S@t;...
Some operators like `source_interval` have a parameter that we might like to change dynamically via dataflow.
Q: "how do I do a `for` loop in Hydroflow" A: it's built into the functional/comprehension model. Example, cite Rust. Q: "how do I do a nested `for` loop in...
Requires docs. Lead with CRDT for marketing, and in the spirit of VLDB "Keep CALM" paper, route people to Hydro as "fixing" the problems of CRDTs in a principled way....
```mermaid flowchart TD optimize["auto-optimize (rewrite) persist()/deltae() #347"] joinstate["split up join() state into persist(), etc. #347"] joinlattice["lattice types in join() (~#271)"] joinopt["auto-optimize join() state"] optimize --> joinopt joinstate --> joinlattice -->...
The online Hydroflow visualizer is awesome! You know what make it even more awesome? Virality! Make it possible to share URLs for your favorite programs with your colleagues, friends, etc....