callysto icon indicating copy to clipboard operation
callysto copied to clipboard

Handle stream operations

Open vertexclique opened this issue 2 years ago • 4 comments

  • [ ] group_by
  • [ ] items: iterate over both key and value at the same time
  • [ ] events: access raw events
  • [ ] take: buffer up specific amount of values from a stream
  • [ ] enumerate: count values
  • [ ] through: forward through yet another topic/stream.
  • [ ] filter: filter values to omit from stream
  • [ ] echo: repeat to one or more topics.
  • [ ] and other ordinary iterators that can be inherited from Iterators/futures::Stream.

vertexclique avatar Mar 29 '22 18:03 vertexclique

For take operations will be nice to handle both amount of record and within timeout, to avoid to stale processing when the amount of values is not reached.

nicor88 avatar Apr 11 '22 18:04 nicor88

I didn't quite get it. What do you mean by stale processing when values aren't raced?

vertexclique avatar Apr 15 '22 11:04 vertexclique

@vertexclique check the implementation of take on faust. It supports an amount of records, in combination with a parameter called within.

Imaging to have a stream.take(100).

The problem with the above code is that it will block forever if there are 99 messages and the last hundredth message is never received. To solve this add a within timeout so that up to 100 values will be processed within 10 seconds.

cit.

So it will be nice to have such feature in callysto too.

nicor88 avatar Apr 26 '22 21:04 nicor88

ok makes sense.

vertexclique avatar Apr 27 '22 14:04 vertexclique