Song Jiaming

Also, I found the code in `RedisStreamReceiver` ``` while (!isStopped) { val response = jedis.xreadGroup( conf.groupName, conf.consumerName, conf.batchSize, conf.block, false, newMessId) if (response != null) { for (streamMessages

@fe2s I have written a simpler version, similar with spark-redis lib. And I found following things. the receiver code is like: ``` val response = jedis.xreadGroup(...) if (response != null)...

Hi @fe2s, to answer your questions. 1. write to redis stream: about 1000, so the producer efficiency is much higher than consumer. 2. `this_duration=200`, microbatch size of RDD could be...

> It might be that your laptop cannot handle the load and you need to scale with Spark cluster. What error do you get when trying to run with multiple...

@fe2s my code of building multiple consumer: ``` val images = ssc.union( Seq( ssc.createRedisXStream(Seq(ConsumerConfig("image_stream", "group1", "cli1"))), ssc.createRedisXStream(Seq(ConsumerConfig("image_stream", "group1", "cli2"))), ssc.createRedisXStream(Seq(ConsumerConfig("image_stream", "group1", "cli3"))) ) ) ``` By the way `spark-redis` structured...

Hi @fe2s, streaming should have at least 2 cores, 1 for receiver, others for workflow. Thus, I tried 2, 8, 28, 56 (server has 56 in total). All of them...

@fe2s Thank you for sharing link, I checked it, the scenario it specified is multiple InputDStream, I think Redis Stream is single InputDStream, if you subscribe to only one?

I think we should migrate it to BigDL? OpenVINO InferenceModel depends on it and it still has some usages?