kafka-rest
kafka-rest copied to clipboard
confluentinc
BUG: Error serializing Avro schema containing timestamp logical types
Background
As per Avro specification 1.11.0, logical-types Timestamp (millisecond precision) should be supported. Until kafka-rest version 6.2.14 it's working fine. I found that starting v7, this is broken.
How to reproduce To demonstrate the issue, use the AVRO schema below:
{
"type": "record",
"name": "Key",
"namespace": "com.example",
"doc": "Avro schema for Kafka Key",
"fields": [
{
"name": "firstName",
"type": "string",
"doc": "First name of the user"
},
{
"name": "lastName",
"type": "string",
"doc": "last name of the user"
},
{
"name": "timestamp",
"type": {
"type": "long",
"logicalType": "timestamp-millis"
},
"doc": "Timestamp in number of milliseconds from the unix epoch"
}
]
}
Sample Request:
curl --location 'https://kafka-rest.example.com/topics/test-topic' \
--header 'Accept: application/vnd.kafka.v2+json, application/vnd.kafka+json, application/json' \
--header 'Content-Type: application/vnd.kafka.avro.v2+json' \
--data '{
"value_schema_id": 43,
"records": [
{
"value": {
"firstName": "Foo",
"lastName": "Bar",
"timestamp": 1709703520
}
}
]
}'
Error:
{
"error_code": 40801,
"message": "Error serializing Avro message"
}
Having the same issue upgrading from 7.4.0 to 7.5.3. I suspect this has to do with https://github.com/confluentinc/schema-registry/pull/2805.
I think setting 'avro.use.logical.type.converters=true' for the AvroSerializer somewhere might fix it but I'm not sure how to set it as an environment variable so it's taken into account...
Having the same issue upgrading from 7.4.0 to 7.5.3. I suspect this has to do with confluentinc/schema-registry#2805.
I think setting 'avro.use.logical.type.converters=true' for the AvroSerializer somewhere might fix it but I'm not sure how to set it as an environment variable so it's taken into account...
I think we can't set 'avro.use.logical.type.converters=true' without this: https://github.com/confluentinc/kafka-rest/commit/176d4b544d7a0c02c103d47c1e781dee2cdaa6a5 But it's only coming in v7.7...
I'm encountering what I believe to be the same issue when trying to serialise UUIDs.
However, it looks to me as if older versions of kafka-rest are compatible with newer versions of the other tools from the confluent-stack. So pinning kafka-rest to an older version and upgrading the rest of the stack appears to work, at least for my use cases.
It now works for me on 7.7.0 with this env variable (using docker compose):
KAFKA_REST_PRODUCER_AVRO_USE_LOGICAL_TYPE_CONVERTERS: true (derived from the producer.avro.use.logical.type.converters property)
