Add support for Protobuf and JSON Schema on KafkaIO + Confluent Schema Registry

[damccorm]

trafficstars

Confluent Schema Registry (CSR) 6 introduces support for other Schema types like Protobuf and JSON Schema. We should adapt KafkaIO to obtain the type of Schema and configure correctly its Deserializer and Coder.

Protobuf support https://github.com/confluentinc/schema-registry/pull/1285 JSON Schema support https://github.com/confluentinc/schema-registry/pull/1289

 

Imported from Jira BEAM-9330. Original Jira may contain additional context. Reported by: iemejia.

[zachariahyoung]

@damccorm What is the status of this request?

[damccorm]

I'm not aware of any work being done on this, but I'm also not very intimately involved with the issue (I just migrated it from jira). @iemejia any chance you're aware of anything to do with this?

[iemejia]

I had a prototype of this looong time ago, but I never finished. It should be pretty straigthforward to implement, since the API of Confluent is closer, and now that reminds me that it was that probably the reason why I did not finish it because I realized that then we would need to care about backwards compatibility and I did not have more time to polish it.

[robertwb]

@Polber This'd be really useful for YAML too, where the type and schema could be inferred rather than needing to be fully spelled out.