Consume message with json as payload, but with simple text as key.

[w3blogfr]

Hello,

I follow the getting started to produce and consume a json payload. https://strimzi.io/blog/2019/11/05/exposing-http-bridge/

The event payload is a json object, and the key is a simple string.

The consequence is when I poll message, I have this error. {"error_code":406,"message":"Failed to decode: Unrecognized token .... because strimzi kafka bridge try to convert my text to json.

If I change the format to "binary", and the accept to "application/vnd.kafka.binary.v2+json", it works, I can get the base64 string, and then convert to text, and then to json.

Is it not possible to consume the payload as json, but with accepting string as key? I should probably not bethe only one in this case, the key used in Kafka is simply a string and not a json object, a text key is more logic.

Thanks.

[scholzj]

The key in Kafka can be everything - including some structured data such as Json or ProtoBufs. In your case, one of the tricks you can use is to put the key in " which makes it valid JSON. I.e. key1 is invalid, but "Key1" would be a valid JSON key.

o we have any better solution @ppatierno @stanlyDoge ?

[ppatierno]

@w3blogfr if you are following the blog post, you are producing through the HTTP bridge right? In the example, the key is already a JSON string (which is what Jakub is suggesting). Or are you producing in a different way?

[w3blogfr]

Hello, Thanks, I knew already the tricks with double quote This solution suppose to change the producer just because one of the consumer is using strimzi kafka bridge. It's not really agnotic to the technology.

In our situation, producer could be node, or java. Events are mainly consumed by node and java and we have one application that don't have a native client, and we try to use for this application strimzi kafka bridge.

[ppatierno]

@w3blogfr I think I missed your reply to this issue ... I know it's two years late but now I see what you mean. I tried with a producer sending (via bridge) a JSON having

"key": "key-1",
"value": "value-1",

A consumer (via bridge) will get this the right way still as:

"key": "key-1",
"value": "value-1",

But a consumer, not using the bridge, for example the kafka console consumer in my case, is getting them as "key-1" and "value-1".

So the quotes are part of the key and value while they should not. In the JSON from the producer they are there just for representing a string. A non bridge-based Kafka consumer should get just key-1 and value-1 with no quotes.

The right approach to have quotes even when not received by a bridge-based consumer, should be that the producer (via bridge) has to escape quotes this way:

"key": "\"key-1\"",
"value": "\"value-1\"",

Is that what you were referring to?

[ppatierno]

After discussing about this on the #737 PR, it's actually impossible to distinguish between a raw string and a malformed JSON when it comes to decoding messages. For this reason, we think that the current json embedded format is working fine as expected but we opened #739 to track the idea about having a new string embedded format for supporting the raw strings use case easily.