scio icon indicating copy to clipboard operation
scio copied to clipboard

More features in scio-redis

Open anish749 opened this issue 2 years ago • 0 comments

We are starting to use scio-redis, and right now have made some modifications to the RedisDoFn for our needs, and we wanted have this added to scio-redis.

  • Support for non-pipelined writes When working with Redis Cluster / setups with multiple redis nodes / proxies in front of redis, we ended up with writes happening without using MULTI / EXEC in pipelines (multi/exec are usually hard to support with proxy layers). This does have consequences that needs to be taken care of in the application desgin, however it is something that we went ahead with. It would be nice to make this configurable, with the default behavior being to use multi / exec and disabling via a flag / param to the IO.

  • Measuring Redis Client Side metrics We are using this with streaming and have two primary needs: alerting in case of exceptions / failures. Monitoring latency of writes. What we are currently doing is having a map step after applying the RedisDoFn where we update metrics to our internal scio-semantic metrics integration. What we would like in scio-redis is a way to capture the results of the redis operations / have a way to inject code to track latency / failure metrics, and then pass it on to other systems for alerting. A separate way to monitor the number of active connections being used to write to redis would be nice too, which is helpful in understanding load characteristics during dataflow auto scaling.

anish749 avatar Jan 21 '22 09:01 anish749