taskiq-redis
taskiq-redis copied to clipboard
taskiq-python
feat: Add get_progress and set_progress to redis result backend
Uses as standard suffix on the redis key (hardcoded as "__progress") to store progress results.
A few questions about this:
- Is a key with suffix
__progressa good default? Should the suffix be configurable? - When should the data expire? Using
redis.getdel()as with get_results is probably not appropriate as it's much more likely to be fetched repeatedly, but maybe it should have a shorter default expiry time? (2h?) - Should it be possible to configure progress expiry separately from results expiry?
I also note that there's a fair bit of duplication in RedisAsyncClusterResultBackend, RedisAsyncResultBackend, and RedisAsyncSentinelResultBackend. They could possibly be refactored to have a common base class with most of the implementation, and the subclass just defining a method that provides the redis connection object as a resource (i.e. with enter() and exit() even if it's not strictly necessary). That refactor is beyond the scope of this PR, but I'm happy to do it.
CC @Sobes76rus as you did the original set_/get_progress implementation
Hello, I think the __progress suffix is good enough and doesn't need configuration
Expiration should be customisable, and it would be even better if it is customisable for each task for results and for progress based on labels
OK so should I add 2 more arguments to the result backend(s) constructor?
progress_ex_time: Optional[int] = 600progress_px_time: Optional[int] = None
I'm happy to add that to this PR, if @s3rius gives a 👍.
The bit I'm least confident about is the 600 default - it's a little arbitrary, but I think some default is better than no expiry. My rationale is that if there have been no progress updates in the last 10 minutes, something has probably gone wrong.
FYI I believe I've fixed the build now, sorry about missing that in the first PR!
Hey, just giving this a nudge. I'm running my project on a fork so no great hurry, although it would be good to know if you're on board with the direction I took?
