Kafka REST proxy mandates field with null values for optional fields of AVRO schema

[keyankay]

I have plenty of optional fields in my AVRO schema, and i do not want to send value as null for all the fields that are optional (there are about 20 optional fields and 5 mandatory fields) via JSON

The Kafka REST proxy is asking for the attribute to be specified even though default has been specified to null.

Ex: "schema": " { "namespace" : "sample", "type" : "record", "name" : "samp", "fields": [ {"name" : "header", "type": { "type" : "record", "name" : "sub_header", "fields" : [ {"name" : "version", "type" : "string", "default" : "NONE"}, {"name" : "id", "type" : "string", "default" : null}, {"name" : "messageID", "type" : "string", "default" : "NONE"} ] } } ] }"}

Also i tried using union {"name" : "id", "type" : ["null","string"], "default" : null} . It did not help still

Now when i want to populate the JSON data: i want to avoid sending the field "id" in the curl command, but it is failing

curl -X POST_schema_id": 93, "records": [{"value": {"sub_header" : {"version" : "images/img.png","messageID" : "123" }}}]}' "http://localhost:8082/topics/avrotest2"

{"error_code":42203,"message":"Conversion of JSON to Avro failed: Failed to convert JSON to Avro: Expected field name not found: id"}

However the below works: curl -X POST_schema_id": 93, "records": [{"value": {"sub_header" : {"version" : "images/img.png","id":null,"messageID" : "123" }}}]}' "http://localhost:8082/topics/avrotest2"

I see that it has been fixed in AVRO for Java script : mtth/avsc#118

[yatharthranjan]

Is there a particular reason this has not been attended to yet? Seeing as the major USP of avro is schema evolution, this is causing a great obstacle to that. We are facing similar problem -- https://github.com/RADAR-base/RADAR-Schemas/issues/153

Would be good to know if there is a planned fix for this in the near future @ewencp @mageshn .

[ijesh]

Running into a similar issue myself. How are people getting around this issue? are there any workarounds? Doesn't seem like this issue is going to get looked at anytime soon. The issue seems to have been introduced here per this comment. Open ticket AVRO-2447

[reneveyj]

Any workaround? This is really needed.

[RaviVaranasi]

+1 on this. Seems odd to define default values only to state to client that it is required

[Mnantier]

I'm also currently running into this issue. Has anyone figured out a solution for this ?