scylla-cdc-rust icon indicating copy to clipboard operation
scylla-cdc-rust copied to clipboard

Add support for JSON based consumers

Open rasviitanen opened this issue 2 years ago • 0 comments

I have some use cases where I would like to send JSON in a CDC consumer. Right now, the consumer only accepts Cql, and extracting/processing the CqlRow can be a bit complicated, especially if you want to build some JSON response from it.

I think it would be beneficial if you didn't have to do any conversion client-side in this case. Would it be reasonable to add support for accepting JSON rows instead of Cql in the consumer implementation?

I've implemented a draft for what that could look like: https://github.com/scylladb/scylla-cdc-rust/compare/main...rasviitanen:scylla-cdc-rust:rasviitanen/json-consumer?expand=1.

In short:

  1. Add a trait ConsumerType and implement it for Json and Cql
  2. Add a marker type to the consumer trait and default it to Cql: Consumer<T = Cql> where T: ConsumerType
  3. Change the consume_cdc function to async fn consume_cdc(&mut self, data: T::Value<'_>) -> anyhow::Result<()>. T::Value for Cql will be the current CDCRow and for Json it's basically just a json text string.

If there's any interest from Scylla's side, I'd be happy to submit a PR for it. I'd understand if you wish to implement it some other way, or not have the feature available at all, but this felt decent enough to me!

Thank you!

rasviitanen avatar Aug 19 '23 22:08 rasviitanen