feathub icon indicating copy to clipboard operation
feathub copied to clipboard

Support timestamp comparison when writing data to Redis

Open yunfengzhou-hub opened this issue 1 year ago • 0 comments

FlinkProcessor's RedisSink should support writing features with timestamp to Redis, where only features with larger keys preserve in Redis.

This function has not been supported so far because in a Redis cluster, the timestamp of a feature and the feature itself might be distributed onto different Redis nodes, and querying across nodes might bring a performance overhead. It is recently discovered that Redis cluster supports grouping keys with hashtags, and key-value pairs with the same hashtag would be held by the same cluster node. Thus this performance overhead can be walked around now.

yunfengzhou-hub avatar Jun 05 '23 10:06 yunfengzhou-hub