streamz
streamz copied to clipboard
Gate node, super node or docs
It may be helpful to have a Gate node, or super node or at least some usage docs/pattern docs.
A gate node would filter one stream by another stream. This could be done by
s1 = Stream()
s2 = Stream()
l = s1.zip_latest(s2).filter(lambda x: bool(x[1])).pluck(0).sink_to_list()
We have some options in terms of implementation:
- Make a node to do this as syntactical sugar
- Create some sort of system for "super nodes" where there are underlying nodes but we can treat it as a single big node.
- Just document usage patterns like this one (and the average square and others)
I don't know which one is better. Thoughts?
Do you have a concrete need for this or is this hypothetical at the moment?
Both? I have realized that I have this pattern in some of my graphs (and I'm about to use it a bunch more) so my need for gate is solid, but I am much more flexible on the implementation (especially as things get a bit difficult when we can choose among the various join operations).
OK, is this something that you can play around with externally to see what patterns work and don't work before we add into streamz core? Presumably the register_api approach allows this sort of extensibility.