Oleksii Diagiliev

@charsyam , sorry, no progress on this yet

Hi @Litchilitchy , As for structured streaming, please see [Level of Parallelism](https://github.com/RedisLabs/spark-redis/blob/master/doc/structured-streaming.md#level-of-parallelism). It explains some options to increase the number of partitions. As for spark streaming, do you refer to`createRedisXStream()`...

Hi @Litchilitchy , How many records per second to do you write to Redis stream? What is your batch interval (`this_duration` in your post above)? How many Spark executors do...

It might be that your laptop cannot handle the load and you need to scale with Spark cluster. What error do you get when trying to run with multiple consumers,...

Could you please share your code?

Hi @Litchilitchy , How many cores do you allocate for Spark? Please see this doc explaining that you should allocate enough cores when creating Spark receivers https://spark.apache.org/docs/latest/streaming-programming-guide.html#input-dstreams-and-receivers

Hi @dggc , Could you perform the update logic in Spark, i.e. read the dataframe from Redis to Spark, find out the rows that needs to be updated and then...

Yes, it's possible. You can specify only the id in the dataframe schema. https://github.com/RedisLabs/spark-redis/blob/master/doc/dataframe.md#reading-redis-hashes

With the read operation you specify the key prefix `.option("keys.pattern", "person:*")` so only the corresponding records are loaded into Spark.

Hi @Zhl-zero-bytes , can you try this? https://github.com/RedisLabs/spark-redis/blob/master/doc/dataframe.md#spark-sql