logstash-codec-avro Confluent Schema Registry as a schema uri

We keep our Avro schemas in Confluent's Schema Registry. It would be great if we could point the schema_uri to the registry's API, but the registry returns the Avro schema as a nested JSON object with the key schema:

HTTP/1.1 200 OK
Content-Type: application/vnd.schemaregistry.v1+json

{
  "schema": "{\"type\": \"string\"}"
}

Thoughts on what would be the best way to support this?

Jan 26 '16 01:01 rbolkey

I agree, this is needed for us as well. It looks like there's a new codec that lets you point to the Schema Registry in the logstash Input (https://github.com/revpoint/logstash-codec-avro_schema_registry). But we need a way in the logstash Output to send to Kafka using avro with a schema_uri pointing to our confluent schema registry. Is this possible, or is anyone working on this already?

May 09 '16 21:05 malonej7

@suyograo shouldn't this be implemented in revpoint's repository? https://github.com/revpoint/logstash-codec-avro_schema_registry/blob/master/lib/logstash/codecs/avro_schema_registry.rb#L90

May 09 '16 22:05 talevy

+1 Confluent platform seems to becoming popular and revpoint repository is unfortunately outdated. Do you consider to create your own avro codec with schema registry support?

There are two problems:

Schema Registry API returns the Avro schema in schema object. You expect that the Avro schema is not inside of nested object.
Every message inside of Confluent Platform should have following structure: < magic byte > < schema id (4 bytes) > < Avro blob > Currently the codec is not prefixing the Avro blob with magic byte nor schema id.

It would be really great to have this codec available. Unfortunately I know nothing about Ruby :/