kcat
kcat copied to clipboard
edenhill
AVRO Record types without fields can't be consumed
When consuming a topic that contains an AVRO type that has no fields, the following error is produced:
% ERROR: Failed to format message in event-rule-engine-eventTriggerStateStore-changelog [2] at offset 18734304: Avro/Schema-registry message deserialization: Record type must have at least one field : terminating
The AVRO specification does not indicate that such types are not allowed. You can create them using the official Java AVRO library, and the Confluent Java AVRO serdes can also serialize/deserialize them without issues.
Is it possible to remove this check: https://github.com/confluentinc/avro-c-packaging/blob/master/src/schema.c#L922 so kafkacat can also consume this type of record?
Can you clarify the purpose of a record without fields? Perhaps you'd be better off just sending a boolean primitive instead if you simply want some message?
does not indicate that such types are not allowed
How so? The docs say "fields: a JSON array, listing fields (required)"
Can you clarify the purpose of a record without fields? Perhaps you'd be better off just sending a boolean primitive instead if you simply want some message?
For forward compatibility I find it easier to have a record without fields and then add (optional) fields to it later than to replace a boolean with a record.
How so? The docs say "fields: a JSON array, listing fields (required)"
Yes the "fields" array must exist, but it is allowed to be empty. At least that is my interpretation, and the offical AVRO library will also accept and generate classes for such a schema.
I'd find it useful too, as we have empty record types as well. Other avro-aware software works well with it, so, in case one finds the spec ambiguous, it's allowed de-facto.
It looks like a fix for this was merged into avro-c a while ago (apache/avro#159) and is in the release-1.11.0 tag. https://github.com/confluentinc/avro-c-packaging doesn't have the same fix but I'm not really sure what the purpose of that repo is as it just looks like a copy and paste of an old version of the apache/avro c implementation?
Would we be able to just update the version being used here? https://github.com/edenhill/kcat/blob/7a61200247732ff057724cb7ed56411b6a5797e4/bootstrap.sh#L147
We generally build this using the bootstrap script but not sure if this would fix the problem for everyone (haven't tested for ourselves yet either) or if there is somewhere else that would need to be updated?
