The message formats for Source and sink do not correspond

[EquentR]

I want to synchronize Redis operations from source to kafka and hand over sink to write back to another Redis. But I found that the message format sent by source and the message format needed by sink are not unified, why this design? Is there any way to convert it?

[jaredpetersen]

It wasn't built with the idea of syncing Redis instances in mind. You're probably better off using native Redis replication features.

---

The main reason we don't make this easy today is that we want to keep the source connector more agnostic. We can get change events from Redis Pub/Sub (as seen in the demo), but that's not the only use case for Redis Pub/Sub. We allow the user to provide any channel/pattern for listening, which means that any payload coming out of PUBLISH can be picked up. So if you're using Redis as a message bus and want to get the messages into Kafka, that's totally fine and supported by this application.

You can still accomplish what you need by transforming the message. That could be done with a custom Single Message Transform (SMT) or something more complicated like a Kafka Streams application. In either case, you would parse the message from the CDC pattern/channel and convert it to another message schema.

This could be something that we try to support in this project, but I don't know if this is a common need given that Redis has native replication support.

[EquentR]

Because of some security requirements, our multiple Redis instances cannot use the replication feature that comes with Redis. And they are located in different security zones. Between them, they can only transfer data in one direction without replying due to the forward and reverse isolators, I just need one-way replication to the target Redis. Appreciate if this is possible!