streamz icon indicating copy to clipboard operation
streamz copied to clipboard

Gate node, super node or docs

Open CJ-Wright opened this issue 8 years ago • 3 comments

It may be helpful to have a Gate node, or super node or at least some usage docs/pattern docs.

A gate node would filter one stream by another stream. This could be done by

s1 = Stream()
s2 = Stream()

l = s1.zip_latest(s2).filter(lambda x: bool(x[1])).pluck(0).sink_to_list()

We have some options in terms of implementation:

  1. Make a node to do this as syntactical sugar
  2. Create some sort of system for "super nodes" where there are underlying nodes but we can treat it as a single big node.
  3. Just document usage patterns like this one (and the average square and others)

I don't know which one is better. Thoughts?

CJ-Wright avatar Oct 31 '17 18:10 CJ-Wright

Do you have a concrete need for this or is this hypothetical at the moment?

mrocklin avatar Oct 31 '17 19:10 mrocklin

Both? I have realized that I have this pattern in some of my graphs (and I'm about to use it a bunch more) so my need for gate is solid, but I am much more flexible on the implementation (especially as things get a bit difficult when we can choose among the various join operations).

CJ-Wright avatar Oct 31 '17 19:10 CJ-Wright

OK, is this something that you can play around with externally to see what patterns work and don't work before we add into streamz core? Presumably the register_api approach allows this sort of extensibility.

mrocklin avatar Nov 01 '17 21:11 mrocklin