Support GEOADD operation

[isaranchuk]

trafficstars

At the moment sink connector supports the following Redis operations: MSET and DEL. It's required to have GEOADD operation in order to add geospatial items (latitude, longitude, name) to the specified Redis key.

Are there any plans to introduce support of other Redis operations, e.g. GEOADD, etc?

Also at the moment, it's not possible to have multiple Redis keys per single Kafka record and there's no possibility to customize Redis key name. I'm wondering if these changes are on the roadmap?

[jcustenborder]

@isaranchuk I don't have anything planned at the moment. We could definitely add this. In this case the key is lat, long? This would take a refactor of the connector but that wouldn't be that bad. The reason this connector only supports keys and values of bytes is due to the schemaless nature of Redis. In redis we know that foo=bar is different than foo = bar and because Redis is just a big hashtable. Because of this I wanted the developer to take responsibility of the key and value formats.

[jcustenborder]

@isaranchuk I took a few minutes and read through the Geospatial best practices. Looking at the following example of the protocol we need 4 pieces of data:

GEOADD cars -115.17087 36.12306 my-car 

This would be:

Field	Value
set	cars
latitude	-115.17087
longitude	36.12306
key	my-car

How should we identify this? Should the set be pass with the data or be a global connector setting. Meaning you have a single connector instances that manages cars. Then you have a key of my-car with a value of the latitude and longitude? The other part I'm having trouble with is how would we handle a delete? How do I delete my-car when someone writes a null value to Kafka?

[isaranchuk]

@jcustenborder ideally we should have the support of key transformation based on incoming Kafka record. Les't say we want to group our cars by brand in Redis where brand is a part of data, e.g.

GEOADD cars:bmw -115.17087 36.12306 my-car

In this case cars is a global config and bmw is part of data that we received from Kafka. It looks like this can be achieved via transformations, in particular KeyToValue: https://docs.confluent.io/current/connect/transforms/valuetokey.html?_ga=2.250839558.1667589177.1578908119-559913963.1578581520#

In terms of delete, Redis geo commands actually piggy back on the sorted set datatype, so we can use ZREM for delete:

ZREM cars my-car

Also I'm wondering if single connector can handle multiple Redis operations at the same time? E.g. for the single car record we want to have:

GEOADD cars:bmw -115.17087 36.12306 my-car
SET cars:car-id my-car

[jcustenborder]

GEOADD cars:bmw -115.17087 36.12306 my-car

For sets we could do something like this. We could drive the set key based on the topic. topic: cars:bmw

You can use things like the regex router transformation to change the topic name. Then you could grab the data directly. Then you would pass the following data.

Key

Struct Field	Value
key	my-car

Value

Struct Field	Value
latitude	-115.17087
longitude	36.12306
key	my-car

Also I'm wondering if single connector can handle multiple Redis operations at the same time? E.g. for the single car record we want to have:

GEOADD cars:bmw -115.17087 36.12306 my-car
SET cars:car-id my-car

You would need to run more than one connector instance for that. As of right now the connector doesn't support sets. I added #24 to track that support for sets. Ideally we would just add support for both sets and geo at the same time.