feathub icon indicating copy to clipboard operation
feathub copied to clipboard

Distinguish empty collection and none in Redis connector

Open yunfengzhou-hub opened this issue 1 year ago • 0 comments

When a Feathub job is writing a map-typed or list-typed feature into Redis with FlinkProcessor, if the feature is an empty map or empty list, there will not be a corresponding entry persisted in Redis, which is equivalent to the case when the feature is none. This causes the problem that Feathub jobs cannot distinguish between none values and empty collections when reading from RedisSources, potentially causing correctness problem.

To resolve this, we need to first check whether Redis does display the same behavior between when a key does not exist and when the key exists but corresponds to none or empty collections. If this problem cannot be resolved due to the limitations of Redis API, Feathub might need to implement special values to Redis to distinguish between these cases.

yunfengzhou-hub avatar Jul 03 '23 01:07 yunfengzhou-hub