How to handle streaming input data out of memory?

[zsz00]

I have over 100 million streaming data sources, It needs to be streamed transducer.

The input data is out of memory, and is stream data such as from kafka. how can i do this: data |> trancducer ??

1. eachline(input_json) |> Map(prase_json) |> op_1 |> collect
2. kafka(topic) |> Map(prase_json) |> op_1 |> collect

have some problem:

1. Transducers not support parallelism for eachline
2. Transducers have not support kafka stream input